The government suffered a big defeat in the House of Lords on Wednesday evening over its planned new injunctions to tackle antisocial behaviour.

Peers voted by 306 to 178, majority 128, against the plans amid fears that noisy children, carol singers and nudists could fall victim to the new injunctions.

The government wants to replace antisocial behaviour orders (asbos) with a new type of measure, an ipna (injunction to prevent nuisance and annoyance).

But peers, led by crossbencher Lord Dear, a former chief constable of West Midlands, and supported by several prominent Tories, said the injunctions cast the net too wide and put at risk "fundamental freedoms" and free speech.

Peers backed Dear's amendment to the antisocial behaviour, crime and policing bill that would replace the phrase "nuisance and annoyance" in the legislation with "harassment, alarm or distress" – the words used for asbos. He forced a vote on the issue despite the Home Office minister, Lord Taylor of Holbeach, promising talks on the issue and possible concessions if he withdrew his amendment.

Dear said ipnas could be used for anyone over the age of 10, only required proof "on the balance of probability", could last for an indefinite period of time and result in a prison term if breached. The crossbench peer said a power which had worked for dealing with housing issues would be inappropriate to be used in the "public environment".

He warned: "It risks it being used for those who seek to protest peacefully, noisy children in the street, street preachers, canvassers, carol singers, trick-or-treaters, church bell ringers, clay pigeon shooters, nudists. This is a crowded island that we live in and we must exercise a degree surely of tolerance and forbearance.

"Because I shall continue to be privately annoyed at those who jump the bus queue, those who stand smoking in large groups outside their office, drinkers who block the footpath outside a pub on a summer's evening, those who put their feet on the seats on public transport, those who protest noisily outside parliament or my local bank, but none of that surely should risk an injunctive procedure on the grounds of nuisance and annoyance."

Lady Mallalieu, a QC and Labour peer, said: "My main concern is the extent to which lowering the threshold to behaviour capable of causing nuisance or annoyance to any person has the potential to undermine our fundamental freedoms and, in particular, the way in which the proposed law might be used to curb protest and freedom of expression."

Peers were told that, under coalition plans, the police, British Transport Police, the Environment Agency, local authorities, Transport for London, the health secretary and housing providers would be among the bodies able to apply to ipnas.

Lord Mackay of Clashfern, a Conservative former lord chancellor, also opposed the government's current wording, telling peers: "One of the fundamental freedoms is the freedom of speech and it is surely clear that in exercising that one may annoy other people, one or more.

"If I have an opinion which I know some people, many people, will disagree with, I'm surely entitled to come out with it. Do I have to reasonably consider whether it will cause annoyance to somebody else? And if so, what should be the consequence? Am I to muzzle my point of view in order to placate people who may be annoyed?"

Labour's Lady Howells of St Davids said she was worried about the effect on race relations.

"This clause could have as damaging an effect as the sus laws, which black people have struggled to have repealed," she said. "It will have serious effects on the black communities and the divisions will be further and further stretched as we know under the sus laws."

Lord Faulks, a Conservative who is soon to become a justice minister, defended the legislation, questioning whether the fears expressed by his fellow peers were realistic. He said the applications for an ipna could only be made by an agency such as a local authority, adding: "That is in fact a defence against inappropriate use. It means that somebody who is the victim of antisocial behaviour has to go through the filter of a hard-working agency in order to establish there is sufficient basis to seek antisocial behaviour, or ipna in this case… the use of an agency provides an important filter."

Faulks said the fact a judge also had to decide on the injunction provided another safeguard. "I simply cannot see a judge ordering an injunction for any of the sort of trivial matters which have been referred to in the course of the argument," he added.

Debate on the bill was later adjourned until Tuesday.