Maryland’s highest court has denied a new trial for a man whose murder conviction was chronicled in the hit podcast Serial.

In a 4-3 opinion, the court of appeals agreed with a lower court that Adnan Syed’s counsel was deficient in failing to investigate an alibi witness, but disagreed that the deficiency prejudiced the case. The court said Syed waived his ineffective counsel claim.

The court reversed a court of special appeals judgment, sending the case back to that court with directions to reverse a Baltimore circuit court judgment granting a new trial.

Syed was sentenced to life in prison after he was convicted in 2000 of strangling 17-year-old Hae Min Lee and burying her body in a Baltimore park. More than a decade later, the popular Serial podcast brought Syed’s case to millions of listeners. The 2014 show revealed little-known evidence and attracted millions of listeners, shattering podcast-streaming and downloading records.

In 2016, a lower court ordered a retrial on grounds that Syed’s attorney, Cristina Gutierrez, who died in 2004, did not contact an alibi witness and provided ineffective counsel. The state appealed. The special appeals court upheld the lower court’s ruling last year and the state appealed that decision too.

In the majority opinion, the court of appeals judge Clayton Greene concluded “there is not a significant or substantial possibility that the verdict would have been different” if Syed’s lawyer had presented the alibi witness, Asia McClain, who said she saw Syed at a library in Woodlawn around the time the state contends Syed killed Lee on 13 January 1999.

“Ms McClain would have been an alibi witness who contradicted the defendant’s own statements, which were themselves already internally inconsistent; thus Ms McClain’s proffered testimony could have further undermined Mr Syed’s credibility,” the court wrote.

The appeals court found that McClain’s account focused on “a narrow window of time in the afternoon” of the day Lee disappeared. It said her testimony would not have rebutted the state’s evidence about Syed’s actions that evening, and at best “would have highlighted Mr Syed’s failure to account precisely for his whereabouts after school” that day.

Judge Michele Hotten wrote separately with Chief Judge Mary Ellen Barbera and Judge Sally Adkins that she would have ordered a new trial. Hotten agreed with the majority conclusion that Syed’s counsel’s failure to investigate McClain as a potential alibi witness constituted deficient performance, but wrote that she believed the deficiency “was prejudicial against Mr Syed”.

Syed’s attorney Justin Brown said in a statement that they were “devastated” by the decision “but we will not give up on Adnan Syed”.

“Our criminal justice system is desperately in need of reform. The obstacles to getting a new trial are simply too great,” Brown said. “There was a credible alibi witness who was with Adnan at the precise time of the murder and now the court of appeals has said that witness would not have affected the outcome of the proceeding.

“We think just the opposite is true. From the perspective of the defendant, there is no stronger evidence than an alibi witness.”

The Maryland attorney general, Brian Frosh, said the state was “pleased with the court’s decision”.

“Justice was done for Hae Min Lee and her family,” Frosh said.

Thiru Vignarajah, a special assistant attorney general who worked on the case, said the state always said the conviction was “just and supported by overwhelming evidence”.

“This has been an arduous process for Hae Min Lee’s family, and we are hopeful the decision finally brings them a measure of comfort and closure,” Vignarajah said.