WASHINGTON—Supreme Court justices appeared divided Tuesday during historic arguments over the fate of gay marriage in California, at some moments almost regretful they took the case and at others splitting neatly into well-worn ideological camps.

Justice Anthony Kennedy, seen holding a key vote, wrote both of the court's major gay-rights decisions, most recently in 2003. On Tuesday, however, he made clear that the case of Proposition 8, a 2008 California voter initiative that rescinded the marriage rights of same-sex couples in the state, had left him conflicted.

"You're really asking…for us to go into uncharted waters," he told Theodore Olson, the lawyer challenging Proposition 8. "And you can play with that metaphor. There's a wonderful destination or there's a cliff."

Tuesday's arguments came in the first of two cases in which the high court for the first time is directly tackling gay marriage, now legal in nine states and the District of Columbia. The court on Wednesday will hear a challenge to the federal Defense of Marriage Act, a 1996 law denying federal recognition and benefits, such as exemption from the estate tax, to same-sex spouses.

Lower federal courts have invalidated that law, saying the federal government has accepted state-authorized marriages in other instances. Decisions in both cases are expected before July.