On May 5, the PEN American Center will present their annual Free Expression Courage Award to Charlie Hebdo—the French satirical magazine that was attacked by ISIS-inspired terrorists in January. A number of writers objected to the award by PEN and pulled out of the ceremonial event.

This controversy has sparked a debate along fairly predictable lines. On one side are those who believe the right to free speech must be upheld and celebrated, and those who think Charlie Hebdo is a xenophobic publication unworthy of such accolade.

But there is a third angle. The dividing line isn’t between those who believe in absolute free speech and those who do not. It isn’t between those who will tolerate racism and those who will not. The dividing line here is between those who understand the threat of blasphemy to religious fundamentalists, and those who do not.

Charlie Hebdo may very well have published racist or Islamophobic caricatures. Its defenders say, with good reason, those accusations misunderstand the cartoons, and most weren’t even about Islam. Its critics reply that white people don’t get to be the only ones to decide what is racist and what isn’t. But this debate is circular and beside the point; Charlie Hebdo wasn’t attacked because of its (alleged) racism but because of its alleged blasphemy. And that distinction is very important, even if largely ignored.

A fortnight ago, screenings of an Indian film on Sikhism were canceled after British Sikhs stormed cinemas across the country and demanded its censorship. They alleged the film was blasphemous—a ridiculous charge, considering the film’s quite favorable content. Similarly, in Pakistan, 24 year-old Esha has been languishing in prison for three years on charges of “blasphemy” largely because judges are too scared to touch her case. A popular Pakistani governor opposed to the blasphemy laws was assassinated in 2011.

Such cases aren’t rare, but on the rise across South Asia. In India and Pakistan, Hindu and Muslim groups are persecuting religious minorities on the basis that others have insulted their religion. It is not unusual for Ahmadis in Pakistan, or Christians and Muslims in India to face mob violence around trumped up charges of “blasphemy.”

I mention this because the writers boycotting PEN think they are standing against racism in the media. Yet, the victims of “blasphemy” are primarily religious minorities who have no power against organized mobs. Who will stand up for them? When will these writers recognize the importance of being free from threats for simply criticizing religious dogma?

Critics of Charlie Hebdo have repeatedly stressed their dedication to free speech and say their objections to Charlie Hebdo are distinct. But how many of them have criticised religious fundamentalists and faced threats? Do that, and two points become clear very quickly: what constitutes as offensive is not as clear cut as many like to think, because religious fundamentalists would like to live in a world where no one can question their dogma. Secondly, without the right to be offensive we would quickly face severe restrictions on what we are allowed to say in general. The groups most likely to face these restrictions would be religious minorities, whether across South Asia and the Middle East or in the West.

So yes, Charlie Hebdo may have published offensive content. But we should celebrate the right to be offensive. Only celebrating the right to free speech when its speech you agree with isn’t principled, it’s opportunistic and shallow. A world where writers safely celebrate only those they agree with isn’t a world that values diversity of speech and thought. And writers who want to live in that cozy world of consensus don’t deserve our respect.