One of the principal gatekeepers to housing-finance markets is stepping up scrutiny of nonbank mortgage lenders, concerned that some may not have the financial heft needed to overcome stressed conditions.

The increased oversight by the Government National Mortgage Association, or Ginnie Mae, comes as nonbank lenders play an ever-bigger role in making mortgages to Americans and as housing markets are cooling. Many of these companies flourished after the financial crisis as banks stepped back from the mortgage market but haven’t...