NEW YORK (CNNMoney.com) -- Despite recent calls for him to resign coming from both the right and the left, Treasury Secretary Tim Geithner isn't likely to lose his job.

The bigger question is how much the recent criticism limits his ability to do his job.

And that's an important question. Geithner will be in the lead in many issues about to break into the forefront for the Obama administration.

He'll be at the center of the debate over what to do with $300 billion left in the Troubled Asset Relief Program, and whether TARP should be extended beyond Dec. 31.

He'll be involved in debates over what additional stimulus should be passed to try to help the still-battered job market.

He's also the administration's point man in efforts to reform the regulation of Wall Street and banking.

But Geithner heads into these battles under fire for continued job losses and an unemployment rate of 10.2%, much worse than the administration forecast earlier this year when it was rounding up votes for the stimulus package.

He also received criticism in a recent report on the bailout of AIG, which he oversaw during his time as president of the Federal Reserve Bank of New York in late 2008. The report by Neil Barofsky, special inspector general overseeing TARP, reported that the Fed overpaid Wall Street firms for contracts they had with the troubled insurer.

And there was the embarrassment that a report on jobs created by the stimulus bill showed some being in Congressional districts that don't actually exist.

While those two reports caused anger in Congress, the real problem for Geithner is the continued job losses, said Jaret Seiberg, an analyst for Concept Capital Washington Research Group.

"You always need someone to blame, and Geithner is a big juicy target right now," he said. "When you have 10.2% unemployment and an economy struggling to get into recovery, it doesn't matter who is in that job, they're always going to be under attack and in a position of weakness."

Still, the attacks on Geithner on Capitol Hill have been unusually public and bipartistisan.

Rep. Kevin Brady, R-Texas, used a recent hearing of the Joint Economic Committee to confront Geithner directly, telling him, "The public has lost all confidence in your ability to do the job, and it is reflecting on your President. Conservatives agree, liberal Democrats agree, it really is time for a fresh start."

That attack followed one by liberal Rep. Peter DeFazio, D-Ore., who said in a recent cable news interview that both Geithner and White House economic advisor Larry Summers were too close to Wall Street to be trusted with turning around the economy. He said when those names are mentioned inside the Democratic caucus, "there are boos and cries of derision. I think there's a growing consensus in the caucus, we need a new economic team that cares more about jobs, Main Street and the American people than it does about Wall Street and huge bonuses."

The Treasury did not respond to a request for comment on the attacks on Geithner. But Geithner himself responded to Brady at the JEC hearing, arguing that the stimulus program and other administration efforts have trimmed the huge job losses that were taking place when Obama took office and that the bailout of insurer AIG and other Wall Street rescues were necessary to halt a broader collapse in the economy.

"If you look at any measure of consumer or investor confidence today, if you look at any measure of the strength and stability of the American economy, if you look at any measure of confidence in the financial system, it is substantially stronger today than when the President took office," he said.

Battles ahead

Still, some Washington observers believe that level of public criticism for the Treasury Secretary will leave the Obama administration at a disadvantage in the legislative battles that lay ahead involving Treasury. At least some expect a change will be made sooner rather than later.

"I don't think it's only coming from a narrow base on the far right and far left. Otherwise it would never have been done publicly," said Richard Bove, banking analyst with Rochdale Securities.

Bove said that beyond any problems Geithner has on Capitol Hill, the Obama administration also has to worry about his credibility with foreign governments and in global financial markets. The steady decline in the value of the dollar is at least partly a result of that lack of credibility, Bove said. He predicts a change sometime in the next few months.

"If Obama feels Geithner can't sell the program, if he doesn't have credibility in either Congress or internationally, the President has to replace him," he said.

But most experts disagree with Bove, saying they doubt that Geithner is in trouble within the administration, and that a change anytime before the midterm elections next November is unlikely.

"I don't sense it's even remotely reached a crisis point for him," said Greg Valliere, a political economist with Soleil Securities. He said the calls for Geithner to resign have been limited to "a handful of loudmouths" on the political fringe.

Brian Gardner, chief political analyst for Keefe, Bruyette & Woods, an investment bank that specializes in financial firms, agrees that Geithner isn't in any imminent danger of being forced out of his job.

Despite some like Bove who have suggested that the administration turn to JPMorgan Chase CEO Jamie Dimon to be Treasury Secretary, Gardner said any replacement who would have the necessary credibility with financial markets would spark just as much backlash for ties to Wall Street.

"Despite Geithner having a couple of not-great days, it's not like anyone can step in and fill that void," he said.

But given the problems in the economy and popular anger at bailing out Wall Street, Gardner and Seiberg both wonder how effective Geithner can be in the coming legislative battles.

"We're in a very populist era that is more pronounced today than we have seen throughout most of our lifetimes," Gardner said. "Anything seen as helping elites or Wall Street is not going to be unpopular. No matter who the Treasury Secretary is, with Democrats in Congress getting nervous, the administration itself is not in a strong position."