Stocks that would benefit from a federal infrastructure spending program fell more than the rest of the market and hit their lows of the day after President Donald Trump abruptly ended a meeting on infrastructure spending with Democratic leaders.

Stocks such as Granite Construction, Nucor, Martin Marietta Materials, Simpson Manufacturing and Fluor fell roughly in tandem, after Trump said he wouldn't make an infrastructure deal while Democrats investigate him. Trump walked out of the meeting at the White House attended by House Speaker Nancy Pelosi, D-Calif., and Senate Majority Leader Chuck Schumer, D-N.Y.

"I walked into the room and I told Sen. Schumer and Speaker Pelosi, 'I want to do infrastructure' … but we can't do it under these circumstances," Trump told members of the press in the Rose Garden.

Trump appeared to be reacting to comments Pelosi made earlier in the day when she said, "We believe the president of the United States is engaged in a cover-up" by blocking White House aides from giving testimony and responding to document requests in ongoing congressional investigations.

A White House official told CNBC the meeting between Trump and the Democrats lasted just about seven minutes.

Individual infrastructure stocks fell more than ETFs, including IFRA, the iShares U.S. Infrastructure ETF, down 0.7%, and IGF, iShares Global Infrastructure ETF, which was flat.

Art Cashin, director of floor operations at UBS, said the infrastructure program would be important for the market, but overall, stocks reacted little.

Stocks were already weaker, under pressure from trade tensions. The Dow Jones Industrial Average fell 96 points. The S&P 500 slipped 0.3%, with the tech sector sliding 0.2%. The Nasdaq Composite declined by 0.4%.

"They should care about that," said Cashin.

Cashin said there are no dates for China talks, and the market is dealing with that uncertainty.

"You don't have anything else going on. The hope was for an infrastructure deal," he said.