Washington (CNN) As the Supreme Court wrestled with a clash between religious freedom and LGBT rights on Tuesday, all eyes were on Justice Anthony Kennedy, who might have to reconcile two strands of his jurisprudence.

The case concerns a Colorado baker who refused to make a cake to celebrate a same-sex couple's marriage because he believes that God designed marriage to be between a man and a woman.

Lawyers for Jack Phillips relied on two parts of the First Amendment -- free exercise and free speech -- to make his case, and at times Kennedy seemed torn during the lively and sometimes rapid-fire arguments.

As a lawyer for Phillips made his free speech argument on behalf of the baker's "artistic expression," Justice Elena Kagan and other liberals pounced, asking where they were supposed to draw a coherent line designating which business owners could qualify for an exemption from anti-discrimination laws. A jeweler? A makeup artist? A hairstylist?

Kennedy worried about the dignity of same sex couples. He wondered if Phillips prevailed "could the baker put a sign in his window, we do not bake cakes for gay weddings?" The 81-year-old justice expressed concern about an "affront to the gay community."

But when the subject turned to free exercise of religion, Kennedy was infuriated by a part of the record where one of the commissioners on the Colorado Civil Rights Commission, who ruled against Phillips, noted that using freedom of religion to justify discrimination is a "despicable piece of rhetoric."

Kennedy wondered if that showed a "hostility to religion."

Kennedy later seemed to side with Phillips when he told a lawyer for the commission: "Tolerance is essential in a free society. And tolerance is most meaningful when it's mutual. It seems to me that the state in its position here has been neither tolerant nor respectful of Mr. Phillips' religious beliefs."

More than two years ago, Kennedy penned the Supreme Court's opinion clearing the way for same sex marriage. LGBT rights advocates often point to his soaring language about equal protection. But they acknowledge this case also taps into a different side of his jurisprudence: an expansive view of the free speech rights.

They fear that if the Supreme Court ultimately sides with Phillips, it will diminish its landmark ruling in Obergefell v. Hodges. Both sides agree that a ruling in favor of Phillips would also open the door to claims from others who engage in professional services -- florists, for example -- that their religious liberty exempts them from public accommodation laws applicable to other businesses.

It was back in 1993 that Phillips opened the bakery, knowing at the outset that there would be certain cakes he would decline to make in order to abide by his religious beliefs. "I didn't want to use my artistic talents to create something that went against my Christian faith," he said in an interview, noting that he has also declined to make cakes to celebrate Halloween.

Flash forward to 2012, when same sex marriage was not yet legal in Colorado, but two men walked into the bakery.

"The conversation was fairly short," Philips remembered. "I went over and greeted them. We sat down at the desk where I had my wedding books open."

The men told Phillips they wanted a cake to celebrate their planned wedding, which would be performed in another state. Phillips said he knew right away that he couldn't create the product they were looking for without violating his faith.

"The Bible says, 'In the beginning there was male and female,'" Phillips said. He offered to make any other baked goods for the men.

"At which point they both stormed out and left," he said.

JUST WATCHED People line up to hear religious liberty case Replay More Videos ... MUST WATCH People line up to hear religious liberty case 01:52

The couple filed a complaint with the Colorado Civil Rights Commission, which ruled in their favor, citing a state anti-discrimination law. Phillips took his case to the Colorado Court of Appeals, arguing that requiring him to provide a wedding cake for the couple violated his constitutional right to freedom of speech and free exercise of religion. The court held that the state anti-discrimination law was neutral and generally applicable and did not compel Phillips' Masterpiece Cakeshop to "support or endorse any particular religious view." It simply prohibited Phillips from discriminating against potential customers on account of their sexual orientation.

Phillips then took his case to the Supreme Court and the justices agreed to take it up after mulling it for several weeks.

In court papers, Kristen K. Waggoner, a lawyer from the conservative Alliance Defending Freedom who is representing Phillips, argued that the First Amendment guarantees him the right to decline to make wedding cakes that celebrate marriages that are in conflict with his religious beliefs. She said that Phillips is protected by two parts of the First Amendment: its protections of religious exercise and free speech. While she argued that the free exercise clause forbids the commission from targeting Phillips "and like-minded believers for punishment," she reserved the bulk of her brief for the free speech clause, perhaps targeting Kennedy, who has at times shown an expansive view of free speech.

Waggoner argued that a person viewing one of Phillips' custom wedding cakes -- his "artistic expression" -- would "understand that it celebrates and expresses support for the couple's marriage." She said the Supreme Court's compelled speech doctrine "forbids the commission from demanding that artists design custom expression that conveys ideas they deem objectionable."

In the interview, Phillips said, "I feel I'm being compelled to create artwork for an event -- an inherently religious event -- that goes against my faith, and I'm being compelled to do so under penalty of jail time and fines."

Not surprisingly, the same sex couple, David Mullins and Charlie Craig, see the case through an entirely different lens: discrimination.

"This case is about more than us, and it's not about cakes," Mullins said in an interview. "It's about the right of gay people to receive equal service."

"This isn't about artistic expression," said Craig. "I don't feel like we asked for a piece of art, or for him to make a statement, we simply asked him for a cake, and he denied that to us simply because of who we are."

The couple is being represented in court by the American Civil Liberties Union.

"In essence, the bakery seeks a constitutional right to hang a sign in its shop window proclaiming, 'Wedding Cakes for Heterosexuals Only,'" the ACLU's David D. Cole wrote in court briefs.

Cole said that whether a cake is an artistic expression is not at issue. "The question, rather, is whether the Constitution grants businesses open to the public the right to violate laws against discrimination in the commercial marketplace if the business happens to sell an artistic product." The answer, Cole contends, is "no."

Twenty other states and the District of Columbia likewise expressly prohibit places of public accommodation from discriminating on the basis of sexual orientation, according to the National Conference of State Legislatures.

The Trump administration sides with Phillips in the case, arguing that it falls "within the small set of applications of content-neutral laws that merit heightened scrutiny" from the courts. "A custom wedding cake is not an ordinary baked good; its function is more communicative and artistic than utilitarian," Solicitor General Noel Francisco argued. "Accordingly, the government may not enact content-based laws commanding a speaker to engage in protected expression: An artist cannot be forced to paint, a musician cannot be forced to play, and a poet cannot be forced to write."

But the government lawyers did draw a line when it comes to race, arguing that laws targeting race-based discrimination may survive heightened First Amendment scrutiny in part because racial bias "is a familiar and recurring evil that poses unique historical, constitutional and institutional concerns."