Senior Tory cabinet ministers have raised the prospect of pulling Britain out of the European convention on human rights, despite a pledge by David Cameron that the party will not lurch to the right in the wake of its drubbing in the Eastleigh byelection.

Chris Grayling, the justice secretary, claimed that a future Conservative majority government would scrap the Human Rights Act, which enshrines the ECHR in domestic law. Theresa May, the home secretary, has reportedly been working on the plans for the Tories' next manifesto.

In the Eastleigh byelection on Thursday the Tories came third behind the Liberal Democrats and the UK Independence party, prompting soul-searching among MPs and ministers.

The prime minister promised to stick to the centre ground of politics, writing in the Sunday Telegraph: "The battle for Britain's future will not be won in lurching to the right, nor by some cynical attempt to calculate the middle distance between your political opponents and then planting yourself somewhere between them.

"That is lowest common denominator politics – and it gets you nowhere. The right thing to do is to address the things people care about; to fix yourself firmly in what Keith Joseph (Margaret Thatcher's ideological mentor) called the 'common ground' of politics. And that's what we have done."

May is drawing up a manifesto promise to withdraw from the convention if Cameron wins an overall majority, according to the Mail on Sunday. She would like to withdraw from the convention now but cannot because of objections from the Tories' coalition partners, the Lib Dems, it was claimed.

Grayling told the Sunday Telegraph he could not conceive of a situation where a Tory majority administration would not repeal the Human Rights Act, which was introduced by Labour.

"We cannot go on with a situation where people who are a threat to our national security, or who come to Britain and commit serious crimes, are able to cite their human rights when they are clearly wholly unconcerned for the human rights of others," he said. "We need a dramatically curtailed role for the European court of human rights in the UK."

The ministers' reported claims were met with disquiet within the Tory party. The foreign secretary, William Hague, dismissed the reports as "speculation" about a manifesto that was still some time from completion.

Mark Field, MP for the City of London, said that if May or Grayling wished to introduce reforms they should bring forward a bill despite Lib Dem objections or risk further cynicism from the public.

"People are not fools," he told Dermot Murnaghan on Sky News, "I think even the headlines today … about getting out of the European court of human rights … we are in government. Either do something now and call the Liberal Democrats' bluff on this, or stay quiet. It's that sort of cynicism, it's just politicians saying words and not doing anything."

Conservative MPs have long been frustrated by controversial rulings by the Strasbourg-based court, including blocking the deportation of the radical cleric Abu Qatada and ruling that prisoners must be allowed to vote.

Labour accused Cameron of caving in to the Tory right. "It's clear David Cameron's response to his disastrous result in Eastleigh is a big lurch to the right," a spokesman said. "He is a weak prime minister who is caving in to the demands of the rightwing MPs in his party."

Sir Menzies Campbell, the former Lib Dem leader and an advocate of human rights, told Sky that ministerial comments over human rights appeared to contradict each other. "If it is a lurch to the right then it's a very uneasy lurch to the right. They don't seem to agree with each other. There does seem to be some confusion between ministers and the PM," he said.