What a difference a week makes.

The Dow hit 20,000 for the first time ever on January 25 -- and many investors attributed this milestone to continued optimism about the new presidency of Donald Trump.

But now that Trump has started to focus more on protectionist policies -- most notably the ban on immigrants from seven predominantly Muslim nations -- and less on stimulus and tax reform, a sense of unease is starting to creep in on Wall Street.

The Dow fell 107 points Tuesday, its second consecutive triple-digit point drop and third straight daily loss. The S&P 500 has fallen for the past four days.

It's important to point out though that this is hardly a panic. The Dow has only fallen about 270 points, or 1.4%, from last week's record highs. The S&P 500 and tech-heavy Nasdaq both hit new all-time highs last week too.

Related: Trump's travel ban: Companies and executives speak out

But the unbridled bullishness that Wall Street had for Trump could continue to fade if his administration stays on a path of economic isolationism.

Several leading CEOs have already urged Trump to soften his stance on the immigration ban. And Wall Street strategists are starting to worry that Trump's pro-growth stimulus plans are taking a back seat to anti-foreigner rhetoric.

"There are a number of questions on Donald Trump when it comes to handling issues of trade deals, immigration, protectionism and developing a sound foreign relationship," said Naeem Aslam, chief market analyst for Think Markets UK, in a report Tuesday.

Aslam noted that "optimism around infrastructure and fiscal spending" had helped boost the markets after the election. But traders are now "increasingly worried that there is a possibility of trade wars."

Trump's tough talk with big American businesses about keeping costs down -- be it the price of military planes or prescription drugs -- seems to be scaring some investors as well.

Related: Trump isolationism is bad for business

To that end, Wall Street's so-called fear gauge, a measure of volatility known as the VIX (VWA), shot up 7% Tuesday following a more than 12% spike Monday. The level of the VIX is still extremely low by historical standards.

But the recent increase has helped push CNNMoney's Fear & Greed Index, a gauge of market sentiment that includes the VIX and six other indicators, closer to Fear territory. The index is now in Neutral mode. It was showing signs of Greed just a month ago.

What could get stocks moving higher again? More concrete proposals about tax reform and financial reform and less talk about border controls.

Katie Nixon, chief investment officer with Northern Trust Wealth Management, wrote Monday that while "Trump's pro-growth agenda has the ability to add momentum to the economy," she is concerned that "immigration and trade proposals may offset some of the benefits to tax reform."

Nixon added that "a border tax, significant limitations/constraints on trade and limits on immigration" could also lead to a significant rise in inflation that could threaten the market rally.

The recent spike in gold, which also tends to rise during times of market turmoil, might be another sign that investors are starting to question just how good President Trump will be for stocks.

Related: Will Trump kill the Trump rally?

Tom Stringfellow, president and chief investment officer with Frost Investment Advisors, said in a report that investors should not ignore the rise in gold prices, as well as the recent weakening of the dollar and spike in long-term bond yields.

Stringfellow said that Trump's quick decision to get out of the Trans-Pacific Partnership and talk about tearing up the NAFTA agreement should unnerve the market even more than it has already.

"While President Trump has clearly telegraphed his intentions to back away from existing trade agreements, many incorrectly assumed these actions were to be explored further down the road. Last week proved otherwise," he wrote.

Still, other market experts aren't too concerned about Trump's tough talk.

"Despite mounting rhetoric, the lesson we can draw from recent and past events is that the U.S. market continues to be optimistic about the prospective impact of proposed economic policies, such as tax cuts, deregulation, and greater infrastructure and defense spending," wrote Patrick Chovanec, chief strategist at Silvercrest Asset Management, in a report Tuesday.

In other words, all of this is just part of Trump's penchant for negotiating in public, be it through media interviews, press conferences or Twitter.

Many of Trump's predecessors in the White House have been far more diplomatic. So this new way of governing may take some getting used to on the part of Wall Street. President Trump may continue to tweet first and worry about the consequences later.