The supreme court has upheld the ban on doctors helping patients to end their lives, but ruled that judges do have the "constitutional authority" to intervene in the debate.

The ruling challenges parliament to re-examine the predicament of those who are severely ill and wish to die but cannot do so without medical assistance. A private member's bill on assisted suicide will be debated in the Lords next month.

The cases were brought by Paul Lamb, who suffered catastrophic injuries in a car accident and now requires 24-hour care, Jane Nicklinson, widow of the right-to-die campaigner Tony Nicklinson, and a claimant known only as AM or Martin.

By a majority of seven to two, the justices dismissed the claims brought by Nicklinson and Lamb. But a narrow majorityof the justices – five to four – did decide that the court has the power to declare the law on suicide incompatible with human rights legislation. Only two justices, Lady Hale, deputy president of the court, and Lord Kerr, said they would exercise that authority in this case.

The court unanimously dismissed Martin's request that the director of public prosecutions should be ordered to rewrite guidelines on whether those accompanying patients to a suicide clinic overseas should be prosecuted, but anticipated that the director of public prosecutions (DPP) will be able to consider whether the policy needs amending.

The supreme court president, Lord Neuberger, said in the judgment: "Parliament now has the opportunity to address the issue of whether section 2 [of the Suicide Act] should be relaxed or modified, and if so how, in the knowledge that, if it is not satisfactorily addressed there is a real prospect that a further, and successful, application for a declaration of incompatibility may be made."

Hale said: "I have reached the firm conclusion that our law is not compatible with the [European human rights] convention rights. Having reached that conclusion, I see little to be gained, and much to be lost, by refraining from making a declaration of incompatibility. Parliament is then free to cure that incompatibility."

Lamb, 58, from Leeds, backed the case originally brought by Nicklinson, a sufferer of "locked-in syndrome" who died in 2012, a week after losing his high court euthanasia battle. Their lawyers argued the prohibition on assisted suicide in the Suicide Act 1961 was incompatible with their rights to respect for private and family life contained in article 8 of the European convention on human rights. A defence of necessity, they maintained, should protect a doctor who assisted in the suicide of someone who has made a voluntary, clear, settled and informed wish to end their lives but were unable to do so on their own.

Lamb is in constant pain and permanently on morphine. His condition is, under present medical knowledge, irreversible and he wishes to end his life with the help of a medical professional. He said after the judgment: "My first reaction was disappointment and I was angry hearing that it has not gone through. But it's a positive result. Parliament now have to face up to it, and if they don't we can go back to the supreme court. While I'm still here I will be doing all I can. It's a step forward. It's better than nothing. Overall I'm proud of myself but it's not just for me but for all those who have been injured."

Martin suffered a brain-stem stroke six years ago at 43, leaving left him unable to speak and virtually unable to move. He also wishes to kill himself, but is unable to do so without assistance from someone else. His legal challenge was slightly different. Close family members and friends who take individuals to the Dignitas clinic in Switzerland for assisted dying no longer face being charged after changes introduced by the DPP. But Martin's friends and family are unwilling to provide such help, so he would have to rely on others. As the legal guidelines now stand, anyone else helping him would be at risk of criminal prosecution.

Martin's lawyers had argued that the DPP's guidelines should be rewritten because they interfere with his article 8 rights and make the consequences of a person encouraging or assisting his suicide "insufficiently foreseeable".

Speaking after the hearing, using an adapted computer operated by sight, Martin said: "[It] takes me one step closer to being able to decide how and when I end my life. People in my situation, who have the right to choose how to end their lives, should be able get the assistance they need from people outside their family. The DPP has said as much to the court. She must now do the same to the public in general so there is no confusion."Under the Suicide Act, helping someone end to end his or her life is a crime punishable by up to 14 years in jail. A number of organisations have intervened in the case including Dignity and Choice in Dying, the British Humanist Association and Care Not Killing.

The former lord chancellor Lord Falconer has introduced a private member's bill in the Lords aimed at legalising assisted dying. It would permit doctors to administer a lethal dose of drugs to terminally ill patients given less than six months to live. Campaigners for assisted dying support it but believe its provisions are too narrow.

Outside the supreme court in Westminster, Jane Nicklinson said: "It's a mixed reaction. I'm disappointed that we didn't get the main points but I'm encouraged that they have said it's not down to parliament [alone], that judges can make decisions in a case like this.

"Parliament have been told they have to get a grip on this subject. Parliament don't want to discuss it. As Tony used to say: 'Politicians are a load of cowards who don't want to make a decision.'"

Nicklinson said she supported Falconer's private member's bill but that it did not go far enough because it only provided for those who are terminally ill.

She added: "Tony said that the terminally ill are the lucky ones because they know their lives are coming to an end."

Saimo Chahal, the solicitor from Bindmans law firm representing Nicklinson and Lamb, said: "It's a good decision. It's fallen short of what we wanted it to be.

"It's disappointing that they haven't gone so far as to declare [the Suicide Act] incompatible with human rights law. But they have given a clear message to parliament that it must review the ban on assisted suicide and the judges have said they may be minded to make a declaration next time around.

"It creates a national debate and a necessity for parliament to look at it. Parliament must consider this group of people who have suffered catastrophic injury and who have a settled intention to end their lives."

Chahal said they would consider taking an appeal to the European court of human rights in Strasbourg.

Rosa Curling, from the law firm Leigh Day who represents Martin, said: "We welcome today's judgment as it clearly indicates that the DPP should now revise her policy. We await her new guidance and hope she will now make clear that those, family members or otherwise, who wish, out of a sense of compassion, to help those unable to help themselves, to end their lives can now do so without real fear of prosecution."

Sarah Wootton, chief executive of Dignity in Dying, said: "The Suicide Act is now over 50 years old and is out of touch with the problems facing dying Britons in the 21st Century. Public opinion is resolutely in favour of change and now the supreme court has clearly indicated that it is only a matter of time before the law is reformed. If Parliament is unwilling to address the issue, then ultimately the courts will."

•This article was amended on 25 June 2014. The original subheading wrongly stated that justices had ruled that judges did not have the authority to declare the law incompatible with European human rights legislation. In fact, they ruled that judges did have that authority.