Britain cannot impose a blanket visa requirement on family members originally from outside Europe but who have valid EU residence rights, the European court of justice has ruled.

The decision is another setback in the government’s campaign to control immigration from the European Union.

Concluding that the EU’s freedom of movement rules trumped British claims that visas were needed to combat abuse of the EU residence card system, the judges in Luxembourg said the Colombian wife of Sean McCarthy, a dual British and Irish national living in Spain, did not need a UK visa or family permit to visit Britain.

The high court referred the case to the ECJ after McCarthy contested UK insistence on a family permit or visa, valid for six months, for his wife, Helena, every time they visit Britain. The couple have two children, both British nationals. The ECJ decided that Helena McCarthy’s Spanish residence card entitled her to travel to Britain without first obtaining a UK visa in Spain.

“The UK is disappointed with the judgment in this case,” said a government spokesman. “As the case is still to return to the UK’s high court for a final judgment, it would be inappropriate to comment further.”

The government argues that because it views the system of residence permits in some EU countries as suspect and open to abuse, it is entitled to impose a blanket entry requirement. The EJC dismissed this view, ruling that where suspicion existed, individual cases could be investigated and visa requirements imposed, but not as a general catch-all system.

“The [UK] legislation at issue requires an entry permit to be obtained prior to entry into UK territory, even where the authorities do not consider that the family member of an EU citizen may be involved in an abuse of rights or fraud. Family members who possess a valid residence card are thus prevented absolutely and automatically from entering the territory of the member states without a visa,” the court of justice said.

“The fact that a member state is faced with a high number of cases of abuse of rights or fraud cannot justify the adoption of a measure founded on considerations of general prevention, to the exclusion of any specific assessment of the conduct of the person concerned himself.

“Such measures would mean, as in the present case, that the mere fact of belonging to a particular group of persons would allow the member states to … disregard the very substance of the primary and individual right of EU citizens to move and reside freely within the territory of the member states.”

Immigration, EU freedom of movement and whether Britain should remain in Europe are increasingly vexed issue in the runup to next year’s general election. The Luxembourg ruling looks likely to cause trouble for David Cameron since it could open doors to thousands of non-EU nationals. It will also disgruntle Eurosceptic Tory backbenchers and fuel the Ukip anti-EU and anti-immigration campaign.

“Britain will be forced to recognise residence permits issued by any EU member state, even though the system of permits is wide open to abuse and fraud,” said the Ukip MEP Steven Woolfe.

“This ruling extends the so-called ‘right to free movement’ to millions of people from anywhere in the world who don’t have citizenship of any country of the EU. This is yet more proof that Britain can never take back control of its borders as long as it remains in the European Union.”

Timothy Kirkhope, a Conservative MEP, said: “We need a visa system controlled by the UK and not the EU. Of course the UK should have an immigration system which is fair, and does not disadvantage the right of British citizens to be with their family.

“We are disappointed as we believe that the UK’s visa system is both fair and lawful. Britain will always be best placed to decide and deal with its own immigration needs … not a judge in Luxembourg.”