Photo

A New York State Supreme Court justice on Friday approved an $8.5 billion settlement between Bank of America and a group of mortgage securities investors, but the ruling contained a caveat that could create new problems for the bank.

Justice Barbara R. Kapnick blessed the 2011 agreement to cover some of the investors’ mortgage losses, but she also said that some of the legal claims by the investors were excluded from the settlement.

Those claims cover potentially billions of dollars of mortgages that were modified to help borrowers stay in their homes.

Banking analysts estimate that Bank of America could have to pay more money to settle those claims, but the amount is unclear. They said the ruling was a reminder that the banks — try as they might — were far from putting their mortgage problems behind them.

“This just made the bank’s murky legal situation even murkier,” said Mike Mayo, a banking analyst at CLSA.

The ruling on Friday involves mortgages issued by Countrywide Financial, which Bank of America bought in 2008. Bank of America has spent years in court dealing with investors who bought bonds backed by Countrywide loans that went into default, causing huge losses.

Lawyers on all sides of the case spent much of Friday afternoon scrambling to interpret the ruling, which initially appeared to be a clear victory for Bank of America and Bank of New York Mellon, the trustee for 530 mortgage bonds bundled by Countrywide Financial.

On many of the legal issues raised about the settlement, Justice Kapnick sided with the banks over the objections of the American International Group, which was a big investor in the bonds.

Lawyers for A.I.G. contended that the settlement shortchanged investors and that Bank of New York Mellon did not act aggressively enough to try to recover more money from Bank of America. They said that the $8.5 billion settlement was a mere fraction of the total estimated losses.

Those issues attracted a lot of the attention in the case, but Justice Kapnick ultimately found that Bank of New York, in many instances, acted properly in helping reach a settlement.

“The trustee did not abuse its discretion in entering into the settlement agreement,” she wrote in the 54-page decision. “And did not act in bad faith or outside the bounds of reasonable judgment.”

She took one exception. The justice said claims by investors regarding loan modifications were excluded from the settlement because the trustee had agreed to settle the claims “without investigating their potential worth or strength.”

“On this issue only, the court finds that the trustee acted ‘unreasonably or beyond the bounds of reasonable judgment.’ ”

The issue of the modifications was raised by the bond investor Triaxx, and had been largely a footnote in the case until Friday.

Triaxx said that the trustee failed to investigate whether Bank of America was obligated to repurchase from investors the loans it modified, and how much those claims could be worth to the investors in a settlement.

According to a lawyer for Triaxx, John Moon, the terms of the mortgage-backed bonds clearly stated that at least $11 billion of the modified loans had to be repurchased. Mr. Moon said approximately another $20 billion in modified loans could also be repurchased, but the language in the bond contracts was less clear.

“The impact of these claims is significant, but it’s not immediately apparent how significant they are,” he said after the ruling.

Bank of New York contends that the contract language is not as straightforward as Triaxx makes it out to be. It also says that federal mortgage modification programs encouraged servicers to modify loans without having to worry about the risk of repurchases.

The trustee could have a difficult time pursuing claims that run contrary to a government policy intended to keep homeowners in their houses.

A statement from Bank of New York said the ruling “vindicated the trustee’s actions by overwhelmingly approving the settlement.”

A Bank of America spokesman said in a statement, “Any outstanding issues raised in the opinion can be addressed without undue delay.”

The ruling could serve as a precedent for other mortgage securities cases still winding their way through the courts five years after the financial crisis.

The 22 bond investors that agreed to the settlement included the money managers BlackRock and Pimco and Metropolitan Life Insurance Company.

Bank of America shares closed down 1.06 percent at $16.75 on Friday, while Bank of New York was down 1.39 percent to $31.96, on pace with other financial companies.

A.I.G. lawyers are planning to carry on their legal battle. “This case is very far from over,” the company said in a statement, adding that the ruling sets a “dangerous precedent that could eliminate important protections for investors.”

The next steps for Bank of America are unclear. Analysts said the bank might have to negotiate a settlement on the modification claims. But first, Bank of New York will have to investigate the merit of the claims — something the judge said it failed to do adequately the first time around.

“If it was up to me,” said Mr. Mayo, the banking analyst, “I would throw a couple extra billion dollars in the pot and call it a day.”