Coalition plans to introduce a new system of secret courts were thrown into disarray on Tuesday after Liberal Democrat delegates voted overwhelmingly to oppose them, rejecting the advice of Nick Clegg and the party leadership.

The vote at the party's conference in Brighton will force the party to go back to the former justice secretary, Ken Clarke, to see if any further compromises can be extracted. Clarke kept responsibility for the legislation despite moving in the reshuffle. It may also have to go to the intelligence services.

Liberal Democrat ministers believe the party has misunderstood the proposals, which are currently being scrutinised in the Lords, and claim the bill has already been improved by Clegg since the measures were first issued for consultation.

Shami Chakrabarti, the director of Liberty, welcomed the result. "Liberty fully expects the party in Westminster to vote accordingly alongside Conservative rebels and Her Majesty's opposition," she said.

Although Liberal Democrat leaders were stressing that the parliamentary party is not totally bound by conference votes, they recognise that the conference voted heavily to throw out the whole section of the bill introducing secret courts.

In a passionate debate, opponents argued that the law would allow ministers defending claims to trigger a hidden court procedure and limit public scrutiny of the government and the security services.

A number of MPs and senior party figures supported a wrecking amendment that would have robbed the motion of its primary purpose, opponents said. The amendment was defeated, prompting a cheer, hugs and strong applause.

The debate centred on the bill's most controversial element – the extension of secret courts, known as closed material procedures (CMPs), into the civil courts.

Opponents believe the powers will enable ministers, rather than judges, to manipulate the way evidence is withheld or presented in the courts – depriving claimants of a fair trial.

Jo Shaw, a former parliamentary candidate who tabled the motion to withdraw from part of the bill, told conference that if passed into law it would allow an unprecedented and deeply flawed system to be implemented in Britain's courts.

"Conference should remember that Kafka's novel was a warning, not a manual," she said. "Secret justice quite simply isn't justice, it is a dangerous perversion of justice. This motion is not saying that all the security services or government officials or police are bad or corrupt. But some may be.

"Some may make mistakes and wish to cover them up. Part two [of the bill] will allow a few bad apples to rot our judicial system from behind closed doors. In simple words, this is a bad bill.

"If you vote for the amendment, which is a wrecking amendment, then we are all responsible for secret courts in civil cases and we won't have anyone else to blame: not Labour, not the Conservatives. This will be Liberal Democrat policy."

Supporters of the amendment, including two MPs and Lord Wallace, claimed the reform will enable judges to hear a greater range of national security cases.

Lord Marks, a peer and QC, said the current system – where the government must apply for a public interest immunity certificate to allow sensitive material to be excluded – is itself deeply flawed and needs to be reformed.

He said: "There are some exceptional cases which now cannot get a hearing at all because evidence which is secret cannot be disclosed to a court. CMPs would not just help the government, they would help claimants too. Judges would be able to hear secret evidence helpful to claimants.

"This is a civil liberties issue, but to preserve civil liberties we need cases to be tried and governments to be held to account by judges".

He claimed the government case had been misrepresented. Conference organisers had earlier refused a request from Chakrabarti to address the conference.