Is Wall Street finally getting ready to throw in the towel on President Trump? Perhaps.

Stocks fell sharply Wednesday following reports of a memo from former FBI Director James Comey that says Trump asked him to stop the investigation of former national security adviser Michael Flynn.

The Dow ended 373 points lower Wednesday -- about a 1.8% decline. It's the biggest drop since September.

A key measure of market volatility also spiked. The common "I word" in the market lexicon -- inflation -- is now being replaced by whispers of impeachment.

"With a growing chorus of Democrats accusing Trump of obstructing justice and even calling for his impeachment, the Trump administration could come to an early season finale," wrote Lukman Otunuga, a research analyst with currency broker FXTM.

Gold prices have popped over the past week too. The yellow metal often does well when investors are nervous. It's a classic flight to safety bet. The US dollar has also sunk recently against the euro and other currencies. And virtual currency Bitcoin has soared.

Related: The super wealthy aren't selling stocks because of Trump

Investors had been betting that Trump would cut back on regulations against Wall Street firms put into place by President Obama. Those hopes may now be fading.

CNNMoney's Fear and Greed Index, which looks at seven gauges of market sentiment, slipped to Neutral mode Wednesday. It had been showing signs of Greed previously.

The VIX (VIX), a measure of market volatility that is one of the components of the Fear & Greed Index, spiked nearly 40% Wednesday.

It's a fairly big selloff, but nowhere close to some of the bigger market drops that stocks experienced in the wake of the 2008 financial crisis.

It also comes just two days after both the S&P 500 and Nasdaq closed at record highs. Corporate earnings have been strong. Consumers remain confident as well.

The Dow is still up about 4% so far in 2017 while the S&P 500 is up 5%.

And the Nasdaq has soared 12% -- largely due to strong earnings from five tech giants in the index that just so happen to be the most valuable companies in America -- Apple (AAPL), Google owner Alphabet (GOOGL), Microsoft (MSFT), Amazon (AMZN) and Facebook (FB).

Still, there are some indications of investor unease about the situation in Washington. Banks were among the biggest losers Wednesday.

Shares of JPMorgan Chase (JPM) fell 4%. Goldman Sachs (GS) was down 5%. And Morgan Stanley (MS) and Bank of America (BAC) each nosedived close to 6%.

Related: Dollar slumps as Trump's latest crisis spooks investors

The market seems to be betting that, at a bare minimum, Trump's economic plans -- tax reform, undoing Obama-era regulations on banks and healthcare and stimulus though a big boost to infrastructure spending -- are dead in the water for some time.

"Trump's agenda already had stalled; now there's the likelihood that issues like tax reform will get crowded out by this political crisis," wrote Greg Valliere, chief global strategist with Horizon Investments, in a report.

Valliere thinks that talk of impeachment is still premature, but that "this a political earthquake, and the aftershocks will persist for the rest of Trump's presidency."

Focus on that last part of the quote. "The rest of Trump's presidency."

One reason that the markets may not be in full blown panic mode about the latest crisis du jour in Washington is because they may be already anticipating the possibility that Trump will not serve a full term in the White House.

And even if Trump were to quit, be impeached or be removed from office via the 25th Amendment of the Constitution, Vice President Mike Pence might be able to have a better chance of getting some of Trump's proposals through Congress.

"People are still looking ahead to the hopes of tax reform as opposed to whatever the new scandal of the moment on 1600 Pennsylvania Ave is," said John Norris, managing director with Oakworth Capital Bank.

"And even if there is impeachment, some would argue that you have a smoother political operator with Pence," Norris added.

Related: Wall Street Journal urges Trump to stop the 'pointless melodrama'

Investors just have to sit and wait to see how the Trump soap opera unfolds. For now, the market still doesn't see a need to freak out.

"The question is whether these are US political rain showers like the ones drenching London's commuters, or real storms with lasting damage. Only time will tell," said Kit Juckes, global strategist at Societe Generale.

But if the chaos in Washington intensifies and puts the chances of any meaningful laws being passed that could help boost the economy in jeopardy -- even if Pence were to soon become the 46th president -- then Wall Street might finally begin to worry.