Aryanna Hering didn’t have pay stubs or tax forms to document her income when she shopped around for a mortgage last year—a problem that made it tough for her to get a loan.

But the nursing student who works part time providing home care for children and the elderly eventually hit pay dirt: For a roughly $610,000 home loan, a mortgage company let her verify her earnings with 12 months of bank statements and letters from clients.

Ms. Hering’s case highlights how a flavor of mortgage once panned for its role in the housing meltdown a decade ago is making a comeback. These loans, aimed at buyers with unusual circumstances such as those who can’t provide the standard proofs of income, are growing rapidly even as rising interest rates and higher home prices crimp demand for mortgages.

Lenders issued $34 billion of these unconventional mortgages in the first three quarters of 2018, a 24% increase from the same period a year earlier, according to Inside Mortgage Finance, an industry research group. While that makes up less than 3% of the $1.3 trillion of mortgage originations over that period, the growth is notable because it came as traditional home loans declined. Those originations fell 1.2% over the same period and were on track for a second down year in 2018.

Get Pre-Approved Find a lender who can offer competitive mortgage rates and help you with pre-approval. Enter the ZIP code where you plan to buy a home GO Please enter a valid ZIP code Please enter a ZIP code

During the financial crisis, many unconventional loans soured after borrowers misstated their incomes and lenders didn’t ask for documentation, earning them the nickname “liar loans.” Today, industry executives say the new unconventional mortgage, now referred to as “nonqualified” in industry jargon, has changed drastically from its crisis-era predecessor and is far safer.

Even so, some regulators, consumer advocates and others worry that the growth in this type of mortgage and rising competition to make such loans could lead to renewed risks for the housing market.

Ms. Hering cooks in the kitchen of her newly purchased home. Jessica Pons for The Wall Street Journal

“It’s a slippery slope,” said Mat Ishbia, the president and CEO of United Wholesale Mortgage, a large nonbank lender that doesn’t issue these loans.

So far, specialty mortgage companies have dominated in making such unconventional loans. But traditional lenders, which are doing less conventional business as interest rates rise, are turning to borrowers with harder-to-document creditworthiness as a new source of revenue and are helping to drive the growth. Nearly half of lenders who participated in a recent survey said they have plans to get into this business, according to Inside Mortgage Finance.

These mortgages don’t meet the criteria to be backed by Fannie Mae or Freddie Mac. Some are large relative to a borrower’s income, are interest-only or span longer than 30 years. In other cases, borrowers have blemishes in their credit histories. Many can’t document their income using pay stubs because they are self-employed or retired.

Tom Jessop, the loan consultant at New American Funding who handled Ms. Hering’s loan, said he has seen demand for unconventional loans double over the past 18 months and they currently makes up more than one-third of his business. “I think it’s just catering to an audience that’s been neglected for years,” Mr. Jessop said. “Now they have an opportunity to get financing finally.”

At the same time, Wall Street investors who buy home loans are scooping up unconventional mortgages that have been packaged into bonds, edging back into a corner of the market that is riskier but provides higher returns. There were $12.3 billion of such residential-mortgage-backed securities sold in 2018, nearly quadruple from a year earlier, according to credit-rating firm DBRS Inc.

DBRS says these deals have shown very little in the way of losses and nearly all underlying loans have remained current, though borrowers could become more stressed if the economy turns down.

Credit-rating firms say there are key differences between old loans and new: The new loans comply with postcrisis “ability-to-repay” rules that require mortgage lenders to make a determination that borrowers can pay down their debts. Underwriting standards and due diligence are stronger these days, they say.

Still, some are worried about the risks for borrowers.

“While they might not be as toxic as some of the loans precrisis, you still have a host of affordability concerns,” said Scott Astrada, director of federal advocacy at the Center for Responsible Lending, a consumer-advocacy group.

Though the new loans have been issued judiciously for most of the postcrisis period, “It’s definitely starting to swing,” said Guy Cecala, chief executive of Inside Mortgage Finance. “As more companies enter the space you’re going to see more competition, and with more competition, you’re going to see loosening of underwriting” standards.

Ms. Hering holds a photograph of her grandfather, who owned the house. Jessica Pons for The Wall Street Journal

In many unconventional loans for which it is difficult to document income, borrowers use bank statements like Ms. Hering did to show they are making money. There are other ways for borrowers to prove they can repay: One type of unconventional mortgage, called an asset-depletion or asset-dissipation loan, generally entails dividing a borrower’s total assets by the term of the loan to derive a stand-in figure for monthly income.

Regulators have highlighted risks in these. “Some banks have initiated this practice without the appropriate risk governance controls,” the Office of the Comptroller of the Currency said in a December report. The bank overseer has noticed a range of methods for doing asset-depletion calculations and wants to make sure the level of oversight is commensurate with the level of risk in these loans, according to people familiar with the matter.

Ms. Hering, who is 30 years old, received a loan at a rate of just over 6% for the first five years; it adjusts after that. She used the money to fully buy out her grandfather’s house in San Clemente, Calif. She jointly inherited it with other relatives and said she needed a loan to pay them for their shares of the property.

“It was scary because if it didn’t work out then I would have had to give up the house, but I was determined to keep it,” said Ms. Hering, who is renting out rooms in the house to help pay the mortgage.