American banks’ shares are performing twice as poorly as the market. So much for diversification.

As banks continued to rise throughout 2019, even as interest rates tumbled, the talk was of their success in diversifying their businesses. Falling interest rates certainly hurt, but banks seemed to always have something to offset it, ranging from a boost to trading desks, mortgage-refinancing volumes, buoyant credit cards or a shift from cash into securities.

One way to measure this confidence was to see banks’ forward price-to-earnings ratio advance relative to the S&P 500 in the latter part of 2019. They grew from 0.57 times the multiple of the S&P 500 overall in August to 0.67 times by December, according to FactSet. “Rates schmates,” the market seemed to say.

But now that faith has evaporated with the coronavirus-inspired yield collapse. S&P 500 banks’ relative forward multiple has collapsed, falling to its lowest level relative to the broader index since 2000. In price terms, S&P 500 banks are down 22.9% in the past month; the index is down 12.4%. Despite banks’ higher capital levels and efforts to shift into steadier businesses, the lesson appears to be that markets still judge them to be much more at risk than other companies in an economic downturn.

These risks should have been more on investors’ minds last year. A quick inventory of banks’ challenges in a low-rate and slow-growth scenario reveals little upside. Yes, trading desks can benefit from volatility when rates change. Yet in a truly wild market, sometimes called “bad volatility,” banks more often lose out. Take December 2018, the last correction: It was a horrible quarter for banks’ trading desks.

Stronger banks like JPMorgan Chase may have the resources to take advantage of others’ struggles to ride out the storm and grab market share. Photo: Bess Adler/Bloomberg News

Lower rates will spark more people to refinance mortgages or buy homes, generating mortgage-banking fees. Banks also tend to hedge out mortgage-rate risk in various ways. But associated fees may be offset by rising prepayment speeds and the resulting impact on the accounting value of mortgage-servicing rights and mortgage-backed securities.

The coronavirus offers some unique additional challenges. Cash might flee to banks in the form of deposits, keeping funding cheap. That is good when low rates are helping spark loan growth. But if companies and consumers also stop borrowing, or are only borrowing to stave off liquidity problems, banks face the prospect of either adding low-quality assets or pushing yields further down by buying bonds instead of lending.

A slowdown in business and travel spending may hit credit-card revenue for banks hard; many have key card partnerships with airlines and hotels. Cross-border business payments may also be in for a rough time as companies halt activity. Revenue from financing shipping and trade already looks vulnerable.

None of this is to say that banks are necessarily doing a bad job at being banks. The risk of bank failure remains remote, and stronger banks like JPMorgan Chase JPM -1.14% may have the resources to take advantage of others’ struggles to ride out the storm and grab market share to boost future returns.

But banks are still banks. Despite the many things that have changed about the industry, the fact remains that, over a full cycle, bad times can be especially bad for them.

Write to Telis Demos at telis.demos@wsj.com