The ability of UK police to use "arbitrary" counter-terror stop and search powers against peace protesters and photographers lay in tatters today after a landmark ruling by the European court of human rights.

The Strasbourg court ruled it was unlawful for police to use the powers, under section 44 of the Terrorism Act 2000, to stop and search people without needing any grounds for suspicion.

The widely-drawn ruling said that not only the use of the counter-terror powers, but also the way they were authorised, were "neither sufficiently circumscribed, nor subject to adequate legal safeguards against abuse".

The use of these powers has grown fourfold, from 33,177 times in 2004 to more than 117,200 in 2008.

The Metropolitan police have used them most, but 11 other forces in England and Wales also make routine use of them.

A political furore ensued when it was disclosed that the whole of Greater London had been secretly designated for stop and search without suspicion since 2001.

The case ruled on today was brought by Kevin Gillan and Pennie Quinton, who were stopped by police while their way to a demonstration outside the annual arms fair at the Excel centre, in London's Docklands, in September 2003.

Quinton, a journalist, was ordered to stop filming the protest despite showing her press card, while Gillan, who was riding his bicycle, was only allowed to go on his way after 20 minutes. They were awarded €33,850 (£30,400) in costs and expenses.

The European judges said the power to search a person's clothing and belongings in public included an element of humiliation and embarrassment which was a clear interference with the right to privacy.

But they also criticised the way in which the police use of stop and search powers was authorised. There is no requirement that the powers be considered "necessary" – only "expedient".

The use of the powers is subject to confirmation by the home secretary within 48 hours, renewable after 28 days, but the European court said there was no real check on authorisations by parliament or the courts.

This was demonstrated by the continuous renewal, every 28 days, of the use of the powers in London since their first introduction, the judges added.

They said they were further concerned that the decision to stop and search somebody was "based exclusively on the 'hunch' or 'professional intuition' of the police officer".

The sole condition was that the search had to look for articles used in connection with terrorism – but this covered many things commonly carried by people in the street. Police did not even have to have grounds for suspecting such articles were present.

The judges said that, because of this, there was a "clear risk of arbitrariness in granting such broad discretion" to a police officer.

Their concerns were compounded by the fact that black and Asian people were four times more likely to be stopped under section 44 and there was a risk that the power could be misused against demonstrators.

"The absence of any obligation on the part of the officer to show a reasonable suspicion made it almost impossible to prove that the power had been improperly exercised," the judges said in describing the lack of judicial checks.

As such, section 44 was not in "accordance with the law" and amounted to a violation of article eight – the right to respect for private and family life.

Quinton told the Guardian she was delighted with the judgment, saying: "There has to be a balance between private life and security.

"The court has shown that section 44 is an invasion of people's right to liberty and privacy. Hopefully the government will have to put a fairer law in place to protect us."

Gillan hailed the ruling as "fantastic news after a long struggle" and added: "I look to the government for a strong response."

Corinna Ferguson, the legal officer for Liberty and acting for the two, said: "Liberty has consistently warned the government about the dangers of stop and search without suspicion and actively campaigned for the tightening up of the infamous section 44 power.

"The public, police and court of human rights all share our concerns for privacy, protest, race equality and community solidarity that come with this sloppy law.

"In the coming weeks parliamentarians must finally sort out this mess."

The shadow home secretary, Chris Grayling, said: "We have long said that anti-terror laws should not be used as a way of conducting normal day-to-day policing.

"The government needs to make sure that the police have the appropriate powers to deal with crime and antisocial behaviour."

The full judgment can be read here.