SAN FRANCISCO — With a menu of legal options, a federal appeals court on Monday appeared generally inclined to support the right of same-sex couples to marry in California. But how the judges reach that historic conclusion remains unpredictable.

In more than two hours of legal sparring, lawyers on both sides of the battle over the state’s ban on same-sex marriage were pressed by each of the three 9th U.S. Circuit of Appeals judges hearing the challenge to Proposition 8. The 9th Circuit is reviewing the appeal of a federal judge’s ruling last summer striking down the proposition as a violation of the constitutional rights of gay and lesbian couples to equal protection under the law.

As cameras televised the federal court proceedings, the judges struggled with many aspects of the case, including whether the sponsors of Proposition 8 have the legal right, or “standing,” to even appeal Chief Judge Vaughn Walker’s ruling. Both Gov. Arnold Schwarzenegger and Attorney General Jerry Brown, the governor-elect, have refused to defend the state law on appeal — troubling to both the panel’s most conservative member, Judge N. Randy Smith, as well as its most liberal, Judge Stephen Reinhardt.

Voters approved Proposition 8 in November 2008 by a 52 to 48 percent margin.

“What we have here is the attorney general and governor with no ability to nullify the acts of the people, and then by not appealing, they do it,” Smith, an Idaho-based judge, told David Boies, one of the lawyers for same-sex couples challenging the law.

Added Reinhardt: “That does not seem consistent with the initiative system. “… It’s just tossing in the towel.”

But the judges also seemed skeptical that Proposition 8’s backers can defend the state law, and were even more skeptical of Imperial County’s argument that it could step in and lead the appeal. At one point, Proposition 8 lawyer Charles Cooper had to concede to Judge Michael Daly Hawkins that he had no legal precedent to support his argument that sponsors of a ballot measure can defend a state law in federal court.

Cooper had even more difficulty when the 9th Circuit turned to the central question in the case: whether Proposition 8 is unconstitutional. Cooper argued that society has a “vital interest” in limiting marriage to heterosexual couples because it encourages procreation and increases the prospect that children will be raised by both their mother and father.

“That sounds like a good argument for prohibiting divorce,” Reinhardt said sarcastically when child-raising came up.

Smith, an appointee of former President George W. Bush, also repeatedly challenged Cooper. Smith was particularly troubled by the fact that California has broad protections for same-sex couples under its domestic partnership laws, yet forbids them to marry.

“What is the rational basis then if homosexuals have all the rights heterosexual couples have?” Smith asked. “We’re left with a word: marriage.”

Hawkins and Reinhardt likewise did not mask their worries about the legal justification for Proposition 8. Hawkins wondered whether voters could choose to reinstate segregation in schools, which Cooper insisted was a different legal issue.

But the judges did not give lawyers for same-sex couples a free ride. In fact, Reinhardt, who has often been scolded by the more conservative U.S. Supreme Court for going too far in his rulings, asked plaintiffs lawyer Theodore Olson how far the 9th Circuit can go in the Proposition 8 case.

Among other things, Reinhardt suggested that the court could limit the impact of its ruling to California, by saying Proposition 8 violated the rights of same-sex couples because it stripped away the legal right to marry established in a 2008 California Supreme Court ruling. The 9th Circuit’s ruling could affect gay marriage bans in nine Western states if the judges take a broader view.

Olson made it clear he believes the law is unconstitutional.

“California has built a fence around its gay and lesbian citizens and it has built a fence around its institution of marriage,” he said.

After the hearing, lawyers on both sides expressed confidence in the outcome but declined to make predictions. There is no timetable for the 9th Circuit to rule. And its ruling will not end the case, as both sides have said they will appeal to the U.S. Supreme Court if they lose.

Legal experts, while uncertain of the path the 9th Circuit would take, came away with a sense that the judges would likely invalidate Proposition 8 — somehow.

“My sense is that they are going to go to the merits and find some standing,” said Rory Little, a Hastings College of the Law professor. “And, on the merits at least, they are inclined to rule against the (Proposition 8) proponents.”

Contact Howard Mintz at 408-286-0236.