Government proposals to expand secret courts suffered a series of crushing defeats in the House of Lords on Wednesday night, significantly narrowing the scope of the justice and security bill.

By margins of more than a hundred votes, peers opposed to the bill limited the government's power to control the deployment of secret intelligence in civil courts and gave judges greater independence.

During a lengthy debate in a packed Lords, the government came under ferocious assault from many of the country's most senior retired judges and politicians. The bill has, unusually, been introduced into the Lords first. The defeats will send a powerful message to the House of Commons and the government. In a sign of tension within the coalition, the Liberal Democrat leader and deputy prime minister, Nick Clegg, has indicated he believes the bill requires improvement.

The votes were a victory for those who wish to narrow the scope of the bill. But those who prefer outright rejection of any expansion of secret courts received less support in the opening debate.

The successful opposition amendments removed any secretary of state's exclusive right to apply for a secret hearing and gave judges more discretion to decide whether hearings should be held behind closed doors.

In the first three votes, the government lost by majorities of 100, 105 and 87. The government also caved in on two minor amendments to avoid more defeats.

Prominent among those demanding that the bill be amended was the recently retired president of the supreme court, Lord Phillips. Unaltered, he warned, the bill would not be accepted by the European court of human rights as consistent with the right to a fair trial.

Human rights groups and prominent lawyers have condemned the central section of the justice and security bill, which expands the use of "closed material proceedings" (CMPs), or secret courts.

Opponents feared the powers would enable ministers, rather than judges, to control the way evidence is withheld or presented, depriving litigants of a fair trial.

The government argues that expanding CMPs into civil courts will enable judges to try a greater range of national security cases and prevent claims being settled out of court.

There were signs of co-operation between Labour and Liberal Democrats to rewrite central sections of the controversial legislation. During the debate, Lord Beecham, the peer leading for Labour in the upper chamber, praised the stance adopted by Clegg.

On Wednesday, Clegg backed criticisms of the bill made by parliament's joint committee on human rights, which suggested giving judges greater discretion and narrowing the scope of the bill. He said: "I am very sympathetic to a lot of what the committee says and the government [is] considering its amendments with an open and, in many respects, sympathetic mind.

"I hope that we will be able to amend the bill to allay those concerns in line with many of the recommendations made by the joint committee on human rights." Lord Lester, a Lib Dem who supported the amendments, said secret courts had initially been supported by the human rights groups Liberty, Justice and Amnesty International "because there was no alternative".

The amendments proposed by the committee, Lester said, "recommended that the best way is to hedge rather than ditch – that is to amend the bill in accordance with the principle of natural justice".

Lord Macdonald QC, the former director of public prosecutions, accused the government of breaking its promises when it said judges would have the last say on whether or not there should be a secret court hearing.

"These amendments would give judges appropriate discretion to balance the interests of national security with justice," he said. "[They] secure a situation where CMPs will genuinely be a measure of last resort."

Lord Pannick, supporting the amendments that limit the expansion of secret courts, said: "Closed … procedures are inherently damaging to the judicial process. [They are] a departure, maybe necessary, from the principle of transparent justice."

There were massive Lib Dem rebellions in the early votes, with the party's peers overwhelmingly backing Pannick's set of amendments. Several Tories, including Lord Hodgson and Lady Berridge, also rebelled on a series of divisions.

The government will have to decide whether it makes further amendments before the bill goes to the Commons.

An amendment rejecting any further expansion of secret courts, backed by the Labour peer Lord Dubs, the Liberal Democrat Lord Strasburger, the Labour Baroness Kennedy and the Liberal Democrat Lord Thomas of Gresford was, however, lost by a large margin 25 to 164.

The historian Lord Morgan, a Labour peer, opposed the expansion of secret courts because there was a "tilting of the balance away from the free individual towards the interest of the state". But Lady Manningham-Buller, the former director general of MI5, said the "central dilemma was how to deploy a wealth of secret information without compromising it".

Commenting on the defeats, the shadow justice secretary, Sadiq Khan, said: "Transparency of justice is a hugely important principle and deviating from it should only be done in the most serious of circumstances with proper checks and balances in place.

"The government's plans were too widely drawn with too few safeguards placing too much power in the hands of government ministers to decide what would be kept secret in court hearings.

"These humiliating defeats for the government mean proper checks and balances are now being put in place, with the worst excesses rightfully watered down."Earlier this week, the new president of the supreme court, Lord Neuberger, stressed the importance of open justice. In a speech, he declared: "judicial decisions, at least in civil and family law, without reasons are certainly not justice: indeed, they are scarcely decisions at all.

"It is therefore an absolute necessity that judgments are readily accessible. Such accessibility is part and parcel of what it means for us to ensure that justice is seen to be done."Access to judgments carries with it access to law and access to justice ... our commitment to open justice which underpins the rule of law.

"The need, the duty, to provide a reasoned judgment is a well-established function of due process, and therefore of justice. A clearly reasoned judgment enables the litigants to understand why the court arrived at its decision."

Seizing on the comments, Clare Algar, of the civil liberties group Reprieve, said: "Lord Neuberger is absolutely right when he says that a judgement given without reasons is not justice. But this is precisely what would happen in a secret court: people would lose their cases without knowing why, as the government would not let them see the evidence used against them. The judgements themselves would also be secret, and so would not add to the body of case law accessible to lawyers and the public."

Commenting on the defeats, the shadow justice secretary, Sadiq Khan MP, said: "Open and transparent justice is a hugely important principle, and deviating from it should only be done in the most serious of circumstances with proper checks and balances in place. The government's plans were too widely drawn with too few safeguards placing too much power in the hands of government ministers to decide what would be kept secret in court hearings. These humiliating defeats for the government mean proper checks and balances are now being put in place, with the worst excesses rightfully watered down."