But the vote of confidence—which was expected—is only the first step and doesn't mean Greece will ultimately go along with the austerity plan, or even avoid default. Thousands of Greeks protested outside the parliament building as the vote was being held.

"Getting stuff through parliament and getting it implemented are two very different things," David Lea, European analyst at Control Risks told CNBC.com. "If they can get past the confidence vote, it doesn't mean they can pass the austerity plan, and if they pass the austerity plan it doesn't mean they can implement it."

Both US stocks and European markets rallied on hopes that the confidence vote would pass and clear the path for the austerity package mandated by the International Monetary Fund and the European Union.

The EU and IMF are currently withholding a 12 billion euro portion of their 110 billion euro loan to Greece as they wait for concrete movement on the package of cuts and privatizations that Papandreou's government have been struggling to pass.

Analysts said that if Greece is unable to get the next portion of the loan, default is likely and could cause a domino effect in other areas of the euro zone.

Parliament is now due to vote at the end of June on the austerity package, and the outcome is far from certain.

"I don't see this fiscal plan taking effect whether it gets passed or not," said Lea. "Does that push Greece inexorably towards default? I think it probably does."

Lea added that those in Greece pushing for a default, "thinking that it's going to be a massively better situation to austerity—might have a nasty surprise on their hands."

Several fund managers said the vote of confidence, and the legal passage of the austerity plan, will only be temporary measures, and that most in the market are accepting that Greece will default. What matters now is how that process is managed.

"For the past few months all of the Greek news has been leading towards this type of event," Ryan Hughes, a portfolio manager at Skandia Investment Management told CNBC.com.

Ahead of the vote, acting IMF head John Lipsky said in a speech in Berlin that the Greek program could work and prevent a default—all that mattered was political will.

"The underlying Greek fiscal system is broken, but if it's broken, that implies it can be fixed. It's not that hard to figure out how to fix it. It's a matter of political will," he said.