The European court of human rights has lost its legitimacy in Britain, Chris Grayling, the justice secretary, has said.

Grayling, a senior Tory minister, said he is finalising proposals to curtail the court's power after 2015 and make sure the UK supreme court's judgments are final.

The court has proved controversial because of judgments on issues such as prisoner voting, and overruling the wishes of parliament. It was recently criticised by Lord Judge, the former lord chief justice, for getting too powerful.

However, any move to detach the UK from the European court or scrap the Human Rights Act would have to be part of a Conservative manifesto rather than government policy, as they would be blocked by the Liberal Democrats in the coalition.

"I think that what we've got to is a situation where the European court of human rights has lost its legitimacy in the UK by doing things that frankly, the people of this country and their elected representatives do not want," Grayling told BBC Radio 4's Today programme.

Grayling said the Tories would set out a "clear pathway" for change by publishing a draft bill and leaving the European convention on human rights would be one possible option.

"We're looking at a number of different options," he said. "There are four principles that have to underpin what we do. We have to curtail the role of the court in the UK. We have to replace the Human Rights Act, which as Lord Judge rightly says is one of the key reasons why the European court of human rights seems to have such sway in the UK. We've got to ensure there's a balance of rights and responsibilities in our laws, and I think above all, we've got to make our supreme court supreme."

He said legal advice suggests that the UK could remain part of the Council of Europe while withdrawing from the jurisdiction of its court. However, he said the Strasbourg court should first rethink its purpose when a country such as Britain reconsiders membership.

"The key issue here is that we were told recently by one of the senior officials of the court who came to parliament that Britain was best in class when it came to human rights," he said.

"If the country that is best in class is saying: 'Hang on, this has gone too far, enough is enough', then actually it is for those in Strasbourg to ask themselves the question: 'Why has that come to pass?'."

Grayling said every word of the European convention on human rights is sensible but the way that it is interpreted is problematic.