Prisoners "damn well shouldn't" be given the right to vote, David Cameron said as he called for the powers of European court of human rights to be restricted.

The prime minister reiterated on Friday his opposition to a legal attempt for suffrage, saying the final decision must lie with the UK parliament and not Europe.

The ECHR – which has ruled Britain's blanket ban on votes for convicted inmates is a breach of their human rights – has announced it is reopening 2,281 compensation claims by UK prisoners.

Quizzed by workers about the move during a tour of the Tetley tea factory near Stockton-on-Tees, Cameron said: "If parliament decides that prisoners should not get the vote then I think they damn well shouldn't. It should be a national decision taken in our parliament."

He said the court's powers must be restricted, telling workers "we need to clip its wings".

Cameron also said tougher controls on freedom of movement within the EU will be needed in the future and suggested restrictions could be placed on new member states until they reached a similar level of wealth as the UK.

"The reason for that is if you look at migration between Britain and Germany or France and Germany, countries of pretty even GDP, the movements are pretty much balanced.

"Its only when you have a real imbalance when you have a poor country and a much wealthier country that you get these vast movements."

From January, Bulgarians and Romanians will gain the same rights to work in the UK as other EU citizens.

Cameron said: "I know an influx of non-skilled workers is a major cause of concern. We belong to the European Union where there are rules saying that if you apply for a job in another country you can go and take that job. That enables British people to go and work in Germany, Spain or elsewhere and it enables European nationals to come and work here.

"When a new country joins the European Union they should not have automatic access to our market.

"Poland and the other eastern European countries joined in 2004 and they were given instant access to British jobs even though Poland and those countries are much poorer than us. As a result the numbers that came were far bigger than anyone expected, 1.5 million came, it was one of the biggest movements in population we've seen in the last few decades.

"That was under the last government and it was a very bad decision and we must not make that mistake again."