The Liberal Democrats are on course to sharply increase their number of MPs, largely at Labour's expense, a Guardian/ICM poll suggests today.

The unique ICM poll of voters in seats within the Lib Dems' grasp suggests the party's vote is climbing more strongly in Labour-held marginal seats than in Conservative ones.

The poll, carried out on Tuesday night, before the prime minister's criticism of a voter in Lib Dem-held Rochdale as "bigoted", shows that Labour and the Conservatives may lose seats to the Lib Dems. Nick Clegg can hope to increase his number of MPs to at least 80; 17 more than the party won in 2005 and the largest at any election since 1923.

In the 42 seats in which the party came second in 2005 and which it could take on a swing of 6% or less, Lib Dem support is now 39%. That is four points ahead of the Conservatives on 35% and 21 points ahead of Labour, on 18%.

Since most key Lib Dem targets are Tory-held, and have often had a low Labour vote, that 18% share is less dramatic than it appears. Compared with 2005 results – estimated on new boundaries – Labour's vote has dropped five points from 23%, in Lib Dem target marginals. Meanwhile, the Conservatives have dropped only one point, from a 2005 score of 36%.

The Lib Dem vote has risen by four points to 39%. That represents an average 2.5% swing from Conservative to Liberal Democrat. That is within the margin of error and lower than the swing suggested by national polls. But it is enough, in theory, to deprive Oliver Letwin, the author of the Conservative manifesto, of his Dorset West seat.

On today's results the Lib Dems could hope to win some seats, such as St Albans, from third place. ICM did not poll voters in seats where the Lib Dems came third in 2005.

A Lib Dem advance on this scale would badly undermine David Cameron's hopes of getting a majority. But there are reasons for Tory optimism in the scale of the Lib Dem advance against Labour.

On the average swing suggested by today's poll, five Labour seats would elect Liberal Democrats. They include Islington South and Finsbury, Edinburgh South and Watford.

The Lib Dems also look certain to pick up Oxford East, held by the former Labour cabinet minister Andrew Smith – although it is already nominally Lib Dem held after boundary changes and so has not been polled for today's figures.

Crucially, there are strong signs in the results that the Lib Dems will also win many other Labour seats. The sample size in Labour-Lib Dem marginals is small and needs to be treated with extreme caution. But there is evidence that the swing away from Labour in Lab-Lib Dem marginals may be as much as 8%, while there is no discernible swing from the Conservatives in Con-Lib Dem battles.

If that is right, then the Lib Dems could expect to pick up all 16 seats where they came second to Labour in 2005 and could gain on a swing of 6% or less. They would also expect to win several seats, such as Colne Valley, from third place, leapfrogging the Conservatives. They may also be able to squeeze the Labour vote in Con-Lib marginals to pick up seats even in places where the Tory vote is firm.

• ICM Research interviewed a random sample of 1020 adults aged 18+ by telephone on 27 April 2010. Interviews were conducted in the 42 top Liberal Democrat target seats according to UKPollingReport.co.uk, where the Liberal Democrats need a swing of between 0% and 6% to win from current holder. ICM is a member of the British Polling Council and abides by its rules.