In a world where few would deny the existence of racism but even fewer would ever admit to propagating it, there will always be the problem of agency. We have racism but no racists – a noun without a subject, a consequence that nobody caused, a system that nobody operates creating victims without perpetrators.

On the web people have the added cover of anonymity, creating an environment where individual writers and entire groups of people are abused because of their race or religion but few have the courage to stand openly behind their statements. On these pages I have been slated as the Guardian's "affirmative action hire", told I was "raising my son to hate white people" and asked to entertain black people's genetic propensity to crime, idiocy, violence and sloth. As I don't know of a single black writer, here or elsewhere, who doesn't deal with this, I know this is not about my work per se.

But what is truly frustrating is the rarity with which those who peddle this intolerance will take responsibility for their own actions and the climate it engenders. When called on their bigotry, usually by other commenters, people usually either escalate their attacks or bristle at the accusation and insist upon their free speech.

As a means of avoiding conversations about what they have done, they instead insist on what they are not. Instead of taking responsibility for what they do they insist instead on their right to do it. Faced with the concrete fact of their actions they retreat into the abstract notion of their entitlements.

The internet did not create this situation. When I started my column, before it was possible to comment online, one journalist wrote a letter in response to a piece I'd done about Britishness: "You clearly know nothing about your country," he wrote. "This piece may set you on the road towards partial knowledge." The piece, written by the reactionary historian Paul Johnson, was also sent to me by another reader who made personal threats. That one, I was told by the paper, I had to take to the police. It turned out the sender had links to organised racists with a track record of violence.

But the internet has certainly exacerbated it, because people feel empowered to be far more insulting when communicating anonymously through a modem than they do in person.

So let us take these instigators of racial animus at their word and assume, since we cannot prove otherwise, that they are trying to engage in good faith. Let's imagine for a moment that no racists comment on the Guardian site or anywhere else and acknowledge that (within legal limits) everyone has the right to say anything they want. You have the right to disparage whole races, ethnicities, faiths and faith itself; you have the right to abuse people on the grounds of their race and religion; you have the right to offend people; you have the right to ignore what the writer is actually arguing and attack them instead for what you imagine they are arguing because of their race and religion; you have the right to be rude, obnoxious, dismissive, ignorant and aggressive. Let us also acknowledge that these rights are important and should be defended.

Even that utopia of absolutisms comes with at least one caveat and two consequences. The caveat is that the right to offend is not the same as an obligation to be offensive or a duty to disparage. We have the right to sleep with our in-laws and fart loudly in lifts. Why, generally, don't we? Because to do so would be antisocial, diminish us in the eyes of others (including those we don't know) and eventually leave us isolated. "I tend to think some things are off-limits," the eminent academic Stuart Hall once argued. "Not in the sense that you should not say them, but you need some care about how and when you go into them. If you wanted to make a joke about concentration camps, you should think twice. At least twice."

But some people not only seem to never think at all, they actively resent the idea that they should have to think about the effect their words have on others.

Those who cry censorship should understand that this is a shared space and like all shared spaces it has rules and standards. If you don't like them, the internet is a big place.

Sensitivity is not a sign of weakness – it is how we all get through the day. The trouble is that not everybody is deemed worthy of it. People feel comfortable saying things about Muslims or black people, as a group, that they would never say about white people or Christians, as a group. Similarly they feel comfortable saying things about Muslim and non-white writers they would never say about other writers.

Which brings us to the first consequence. That what you write has an effect. If you write something racially offensive then those you have offended will be less likely to participate. The hostile environment to which you have contributed will also become, by definition, a limited and limiting one. What you end up with is a community, where people are excluded because of who they are that then shrinks to a fetid ecosystem including only people who are just like you.

It certainly has an effect on me. When Comment is free started I regularly came in below the line. After a few months of abuse I mostly gave up. As much as I believe in open journalism I don't believe it's my job to wade through racist filth in search of the many gems that undoubtedly may lurk beneath. If I can get into a thread early enough I respond. Otherwise I only do so when moderators have the time to point me to engaging comments worthy of a response.

Second, the right to offend is a two-way street. It means nothing if it is not accompanied by the right to be offended. Those who believe racism is fair game should at the very least understand that calling them on their racism is no less so. You don't have to accept the accusation but if you want to be taken seriously then you have to take the accusation seriously and engage with it. To hide behind your right to free speech is little short of pathetic.

As I wrote in my most recent book, insensitivity never achieved much. Baiting, ridiculing and humiliating are poor substitutes for satire, irony and humour, although they often masquerade as such. But oversensitivity never achieved very much either. Not every nuance, challenge, wordplay or display of ignorance is a slight; not every slight is worth of being escalated into an incident; not every provocation need be indulged. We all have a duty to help create the "safe spaces" for people to both engage in self-criticism and accept criticism from others. But we have no less a duty to engage in an honest and open manner that makes difficult discussions possible.

That's not censorship, its common sense. There's nothing inevitable about the prevalence on the internet. People have made it so, which means people can make it otherwise. Just because comment is free that doesn't make bigotry permissible.

• Follow Comment is free on Twitter @commentisfree