House Speaker Paul Ryan was like a one-person wrecking crew to the stock market, CNBC's Jim Cramer said Thursday.

U.S. stocks closed lower on Wednesday after Ryan told an audience in Washington that tax reform will take longer to accomplish than repealing and replacing Obamacare.

"The House has a (tax reform) plan but the Senate doesn't quite have one yet. They're working on one. The White House hasn't nailed it down," Ryan said.

Like repealing Obamacare, reducing tax rates was a central promise by President Donald Trump during his presidential campaign. The possibility of tax reform was one of the main drivers in the stock market's postelection rally.

"[Ryan's] not sensitive. He's like a Hillary Clinton. They're just not sensitive to the stock market," Cramer said on "Squawk Box."

"I'm not saying he shorted the market because that would be facetious. But he's a one man wrecking crew," Cramer added.

"What matters is Ryan saying look, 'Trump has no game,'" Cramer said.

Watch: Market dips on comments from Fed and Speaker Paul Ryan