Wall Street's biggest gripe with President Obama is more about what he's said than what he's done.

"There's been so much finger pointing. He's made it seem bad to be successful and to be millionaires and billionaires," said Karl Wellner, CEO of Papamarkou Wellner Asset Management, a fund with $3 billion under management.

Outside of Obama calling bankers "fat cats," most Wall Street professionals point to few specifics and basically say it's more of a vibe they get from the president.

Wellner, who emigrated from Sweden in the early 1980s, says his biggest concern is that the president appears to want to clamp down on the accumulation of wealth. It's a sentiment echoed by many other top hedge fund managers and executives at financial firms.

"There's the idea that if you're successful you didn't do that on your own," said Sander Gerber, CEO of hedge fund Hudson Bay Capital Management. "Romney believes that individuals' drive and willingness to work hard is what makes them successful." And that's a theme that resonates on Wall Street.

And it's also a big reason why Wall Street has given the majority of its contributions to the Republican party during this election cycle.

In fact, Republican candidate Mitt Romney has received more than three times what Obama has generated from Wall Street professionals, according to the Center for Responsive Politics. That's a sharp turnaround from 2008, when Obama generated nearly double the Wall Street contributions of his then rival John McCain.

Beyond the rhetoric, several hedge fund managers also say that while health care and financial regulation were in need of reform, the president tackled them in a manner that put the government too deeply in the middle of both industries.

Congress passed the Dodd-Frank Act on financial reform in 2010, but many of the rules still need to be written.

Romney has criticized the law and said he would take steps to change it, and even attempt to dismantle it. Yet, many on Wall Street are simply betting that Romney would appoint regulators who would make financial reform less onerous.

"Obama would give carte blanche to heavy-handed regulators and just say go for it," said Wellner.

Related: Wall Street overwhelmingly backs Romney

Wall Street's ire comes as many still criticize the president for not pushing back on Wall Street enough and see Dodd-Frank as largely toothless.

"There's the idea that he wasn't willing to hold Wall Street over a barrel when he had them over a barrel," said Jeff Connaughton, former chief of staff to Democratic Sen. Ted Kaufman and the author of "The Payoff: Why Wall Street Always Wins."

Meanwhile, Obama's rival has made it clear that he is in favor of regulating banks. Romney's main objection to Dodd-Frank is that it gives big banks the implicit guarantee of further bailouts. Wall Street executives seem to believe Romney would treat the industry in a more fair and balanced manner.

"Romney looks at things more analytically," said Wellner. "He's more of a consensus builder. The president is dogmatic, it's his way or no way."

Related: Private equity's choice: Obama or Romney

While Romney's experience at private equity firm Bain Capital has caused the broader American public to question whether he has their best interests at heart, many on Wall Street see Romney's business background as a big plus.

"Romney is a consultant and looks at what is efficient, so he'll be reasonable," said Wellner. "He understands how business works so he'll put the rules and guidelines in place that make business work right."