Washington (CNN) -- Despite President Obama's accusation Wednesday that Republicans are holding middle class income tax cuts "hostage" by tying them to an extension of tax cuts for wealthier Americans, the reality is several Democratic senators also oppose allowing President Bush's tax cuts for higher earners to expire.

Democrats Ben Nelson of Nebraska, Evan Bayh of Indiana, Kent Conrad of North Dakota, and Independent Joe Lieberman of Connecticut have each publicly expressed concern about the impact of raising taxes, even on the well-to-do, during an economic downturn.

"The general rule of thumb would be you don't want to do tax changes, tax increases...until the recovery is on more solid ground," Conrad said recently, summarizing their view.

Each has said the tax cuts should be extended at least temporarily.

Senate Democratic leaders know it will be virtually impossible to pass a controversial tax bill without the support of all their members but intend to press ahead with the debate later this month. They believe the political and fiscal arguments are on their side and that allowing the tax cuts for the wealthy to expire is "the consensus position of the caucus," even if a few Democrats oppose doing so, according to a top Senate Democratic leadership aide.

"There is bipartisan consensus that we should not raise taxes," retorted a senior senate Republican leadership aide, seizing on the division in the Democratic ranks.

The GOP aide didn't rule out a compromise that involved passing a short-term extension, a proposal put forth by House Majority Leader John Boehner. "If the only way to pass a bill that didn't raise taxes would to pass a temporary extension, we'd be open to talking about it," said the aide.

But the Democratic aide doubted Republicans would seriously work to help Democrats pass a bill this close to the November election.

"I don't believe the Republicans are interested in compromise," the aide said.