Reform of US healthcare, a centrepiece of the Obama presidency, was in danger today of being derailed after the Republicans produced one of the biggest political upsets of recent US history by winning Ted Kennedy's Senate seat in Massachusetts.

The victory against the odds came on the eve of the anniversary today of Barack Obama's inauguration, a heady time for the Democrats when they entertained hopes of a prolonged period of dominance.

The win robs the Democrats of their filibuster-proof 60-40 majority in the Senate and throws into doubt the future of Obama's health reform plan. Obama and the Democratic leadership will have to decide whether to take another look at the bill.

Scott Brown, a truck-driving National Guardsman who was virtually unknown even in Massachusetts a few weeks ago, beat Martha Coakley, the state attorney general who had expected to inherit the seat, by 52% to 47%.

Brown, in his victory speech, implicitly referred to a decisive moment in the campaign when the moderator in a debate asked him how, given his views on healthcare reforms, he could sit in "Ted Kennedy's seat" – a description repeatedly used by Coakley in the campaign. Echoing his response to the moderator, Brown said in his victory speech: "This Senate seat belongs to no one person, to no political party … This is the people's seat."

Coakley, in her concession speech, said: "I am heartbroken at the result, as I know you are, and I know we will get up together tomorrow and continue this fight, even with this result tonight."

It is a huge psychological blow to the Democrats: the seat of John F Kennedy and then Edward Kennedy until his death last year is now in Republican hands. If a seat regarded by Democrats as one of the safest in the country can fall, then scores of Democrats standing in the congressional mid-term elections in November will see themselves as also vulnerable.

"It is a shock, a total shock," said David Hadas, 37, one of more than 1,000 Coakley supporters at a Boston hotel for what they hoped would be a celebration.

"It is only a year ago everyone was very upset with the Republican party and we swept Obama into office."

Voters, citing reasons for the shift to the Republicans, repeatedly expressed hostility towards the healthcare bill but also a belief that Obama represented too much government interference, was too leftwing and was spending too much.

The Democrats have several Plan Bs for the health bill, none of which they regard as satisfactory. One was to vote on the bill before Brown takes up his seat but Jim Webb, a Democratic senator, appeared to block that last night by saying the election had been both about healthcare and the integrity of the government process.

More than 2,000 Republicans turned up at another Boston hotel last night to noisily celebrate a rare victory after heavy defeats in the 2006 congressional elections and again in 2008 for the White House.

"I pray this will be the start of a bloodless revolution, the start of the campaign against the Obama agenda, in which the silent majority are heard," David Knight, 43, a Republican from neighbouring Rhode Island, said. "We hope this is the end of the health bill but they could still ram it through."

Michael Nicolazzo, 26, who was also at the party, was a Democrat until two years ago but felt Obama was too leftwing. "This was a referendum on Obama. For the bluest of all states to elect a Republican, it really sends a message that people do not want extravagant spending." He, too, hoped the health bill would be killed.

Brown received a lot of backing from Republicans who had travelled to join him from all round the US, and also from grassroots groups that have grown up in opposition to Obama's agenda, particularly on health, such as the Tea Party.

At the Boston hotel where the Democrats held their wake, some began crying as the first results came through. Most headed home early.

"We are all in shock," said Addrienne Walker, 40, who was still carrying a Coakley poster. "We have not had a Republican in that seat since 1952." She hoped Obama would not back off on health reform but admitted that the November elections would be tough. "It is not looking good. Obama is going to have a fight on his hands."

Even before polling closed, the Democrats were engaged in a blame game. The White House and the national leadership hinted that Coakley had been responsible because she had fought too low-key a campaign.

Obama issued a relatively terse statement, thanking Coakley for her hard work.

Local Democrats blamed the national leadership, saying they had been too slow in recognising the danger and providing the necessary campaign cash and staff. They accused the national leadership of having pushed for the negative ads in the final days and claimed this had alienated independents.

A third candidate, Joe Kennedy, representing the Libertarians, took only 1%. He is no relation to the late senator.

• This article was amended on 21 January 2010. The original said that Martha Coakley had used the phrase "Ted Kennedy's seat" in a political debate. This has been corrected.