The Supreme Court heard oral arguments today in the case of Masterpiece Cakeshop, Ltd. v. Colorado Civil Rights Commission. On its surface, the case pits religious freedom against gay rights. Those on the left would have you believe it’s far simpler than all this, but in reality, it’s about much more.

The case centers on Jack Phillips, a Colorado baker who refused to make a wedding cake for Charlie Craig and David Mullins, a gay couple who went to his store, Masterpiece Cakeshop. Phillips is a religious Christian, who said he would not make a cake for a gay wedding. Litigation ensued.

If you read this morning’s column by my colleague Elura Nanos and nothing else, you might think that Phillips is a horrible bigot who refuses to sell any of his baked goods to gay people. You might also think that there’s no reason why this case should have gotten to the Supreme Court in the first place, because obviously religious people are evil and it’s impossible to hold good faith beliefs that aren’t a guise for expressing hatred.

Of course, none of this is true.

Phillips has gone on record saying that he doesn’t judge anybody for how they live their lives. He just doesn’t see why he needs to be a part of it. He has said that if Craig and Mullins wanted to buy anything else from his shop, he would have been happy to sell it to them. Phillips just didn’t feel like he should have to create something for an occasion that he doesn’t want to be a part of, as it conflicts with his personal religious principles. Historically, marriage has been a religious institution, as is recognized by the authority this country gives to clergy members to officiate them. If a straight person had asked Phillips for a cake for a gay wedding, I’d imagine he’d refuse, because for Phillips, it’s not about the identity or sexual orientation of the customer, but the nature of the work.

If Phillips refused to sell any kind of cake to a gay person because they were gay, I’d agree that this was an open and shut case of blatant unjust discrimination. It’s not though, which is why I’m on Phillips’ side, even though I’m pro-marriage equality and sympathetic to Craig and Mullins. When you boil it down, this case is much bigger than just whether or not it’s okay to discriminate. It’s about whether or not the government should be able to exert control over people and their businesses by forcing them to devote time and resources to something they don’t want to be a part of.

If a customer walked into Masterpiece Cakeshop and wanted a cake with Jake Phillips’ face on it, with text that says “Jake Phillips is the Worst,” should he be forced to do that? No, of course not. The current case isn’t much different, except it involves sensitive issues of gay discrimination and religious beliefs. Those on the right can argue that it’s not discrimination because Phillips would have sold the same gay couple something else. Those on the left can argue that it’s not religious discrimination because no one is telling Phillips to marry another man. What remains is whether a business should have the freedom to decide what kind of work they want to do, regardless of who the customer is. Should a cheese shop have to provide non-dairy alternatives if a lactose-intolerant customer asks? Should a butcher have to sell tofu?

As a decorative cake baker, Phillips considers himself an artist who expresses himself through his work, so that adds a First Amendment freedom of speech wrinkle to it as well. By making a cake for something he doesn’t support, he feels he’s being forced to say something he doesn’t want to say.

On the flip side, just as Phillips should have the right to bake or not bake whatever cakes he wants, those who are upset by his decisions have the right to react. If Colorado residents are mad at Phillips, they can stop supporting his business. Hell, they can organize a boycott and write letters to the editor of local newspapers denouncing his practices. If his sales were to take a hit, Phillips would be faced with a business decision over whether to keep his policy or change it, or maybe relocate to an area with a population more in line with his values. That’s how these issues should be handled.

Of course, it’s a lot easier to just label someone a bigot.

This is an opinion piece. The views expressed in this article are those of just the author.