An opinion poll for The Independent gives the Conservatives a 21-point lead over Labour, which should now "seriously consider" dumping Gordon Brown, according to a close ally of Tony Blair.

Labour backbenchers called on cabinet ministers to tell Mr Brown to quit after the party came a humiliating fifth in the Henley by-election, behind the Greens and the BNP. "Brown's people said it couldn't get any worse – now it has," said a senior Labour MP.

The Labour jitters will be increased by the latest ComRes poll for The Independent, which would give David Cameron an overall majority of 212 if repeated at a general election.

Download the new Independent Premium app Sharing the full story, not just the headlines

It is the Tories' highest rating and the biggest lead for any party since ComRes began polling for this newspaper in 2006. The Tories are on 46 per cent (up two points on last month), Labour on 25 per cent (down five points), the Liberal Democrats on 18 per cent (up two points) and other parties on 9 per cent (no change).

The ComRes findings suggest that Mr Brown may be dragging Labour's support down. When voters were asked whether they thought of themselves as being a natural supporter of Labour, the Tories or another party, more people (27 per cent) opted for Labour than the Tories (26 per cent), with the Liberal Democrats on 11 per cent and "don't knows" on 14 per cent. That may encourage Labour critics of Mr Brown to believe the party could recover under a new leader. "We can't just sleepwalk to defeat," one former minister said. "A lot of people are saying after Henley that Gordon is a loser and has got to go."

Lord Levy, Labour's chief fundraiser under Mr Blair, said it was up to party members to decide whether Mr Brown should stand down. He told the BBC's Newsnight programme: "I certainly, seeing the polls, would have to say that this is something that needs to be very seriously considered."

"It's a bad day when the BNP can beat Labour. It sticks in my craw," said Ronnie Campbell, the Labour MP for Blyth Valley. "I don't like to back losers. He [Mr Brown] is a loser at the moment." David Drew, the Labour MP for Stroud, said: "There's no point in doing the job when you are not enjoying it ... He's got to look at himself. He's got to make that decision."

Mr Brown, on a visit to Manchester on the first anniversary of him becoming Prime Minister, said: "By-elections come and by-elections go. Of course we have to listen to what people say. But my main job is to improve our public services, to get the economy moving forward, to make sure that in the health service and education people have the best services that they want and I am going to continue to do that. And I think people know that we are going through difficult times in the economy. It's my job to steer us through these difficult times."

ComRes says the Tories are ahead of Labour among every age and social group, and in every region except Scotland. Some 13 per cent of people who identify themselves as Labour voters intend to vote Tory, as do 14 per cent of Liberal Democrat identifiers. Only 4 per cent of Liberal Democrat identifiers say they will vote Labour, while 7 per cent of Labour identifiers intend to back the Liberal Democrats. Some 74 per cent of Tory supporters are "absolutely certain" to vote, compared with 58 per cent of Labour supporters and 50 per cent of Liberal Democrats.

ComRes telephoned 1,007 adults in Britain between 25 and 26 June. Data were weighted by past vote recall. ComRes is a member of the British Polling Council and abides by its rules. Full tables at www.comres.co.uk