Watching the events of the last 10 days makes it hard to have much confidence in the sanity of the human race. After Friday prayers in Pakistan – the day of Love for the Prophet in which an estimated 19 people died and 200 were injured – protesters burned down two cinemas to express their outrage about Innocence of Muslims, a work that was made available only on YouTube, not in cinemas. It's as logical as smashing your computer because you object to what's being shown at the local Odeon.

But things could have been much worse. Even though so many lives have been lost, there was another side of the story that escaped us – the fact that a very tiny proportion of Muslims actually went on the streets. In France, Muslims showed restraint after the publication of cartoons by Charlie Hebdo. In Egypt and Tunisia, the governments intervened to stop the riots, while in Yemen and Pakistan, the military prevented rioters reaching western embassies. More hopeful was the action by cheering crowds that swept through Benghazi, in eastern Libya, clearing the strongholds of Ansar al-sharia, the militant group thought responsible for the death of US diplomat Chris Stevens a week ago.

Reason did eventually prevail. The American administration got the picture quickly. Obama and Clinton moved to condemn the film, even buying advertising time on Pakistani TV, and the vast majority of Muslims saw Innocence of Muslims for what it was and stayed at home.

The Gawker website had it about right in lampooning the cover of Newsweek, which shows two angry Muslim men under the headline: "Muslim Rage". The website ran a series of pictures of smiling Muslim kids on a garden swing, canoodling in the park, doing handstands, swimming. My favourite is of a man slouching in a chair outside his store with a newspaper. The caption reads: "This Egyptian guy is filled to the brim with Muslim Rage".

On YouTube, you will find a man named Syed Mahmood, who makes an eminently sensible assessment. Innocence of Muslims was "a load of crap that was made to ignite rage – being violent makes a success of the film". This is the voice I recognise from covering the Arab Spring – calm and rational, but no less devout for its modernity. I encountered it in Egypt and Tunisia, where young people expressed their ambition to combine the freedoms and prosperity of the west with respect for Islam and local traditions. It's a voice you haven't heard much in the media because idiots dancing round a fire makes news, but it was the force behind the Arab uprisings and, despite the repression and manipulations of an older generation, it is still there.

The film and the cartoons are plainly not some faux pas that will easily be forgotten. People are genuinely offended. The grievance will be nurtured and there are forces in Islam that will continue to whip up anger and to insist that the free expression of the west and the lack of government control on the internet and media was responsible for the film… in fact, tantamount to a blessing. That is where my sympathy ends, because it as rational as blaming me for the sentiments expressed in the graffiti at the end of my street.

We have an obligation to use free speech responsibly and there are laws in Europe and the US against religious and racial incitement, which the film and cartoons may have broken. However, as Muslim spokesmen (they are nearly always men) demand greater restrictions on free speech, I find myself thinking that if every major Arab newspaper did not habitually publish antisemitic cartoons, which in their levels of racist hatred would not be out of place in the pages of Julius Streicher's tabloid Der Stürmer, I'd be more inclined to listen to their complaints.

To the people who rioted and killed last week, there is no equivalence between an insult to religion and an insult to race – the first unquestionably trumps the second. But like most people in the west, I believe that racism is on a par with religious intolerance and causes just as much suffering. Newspapers across the Middle East allow this outpouring of racial hatred, without the slightest understanding that it is profoundly offensive to demonise and dehumanise Jews by portraying them as Nazis, cannibals, snakes, pigs and cockroaches.

It's much harder to admit to the faults in your own society than reach for the status of aggrieved party – and if there is one criticism of Muslim countries it is this reflexive victimhood that entirely overlooks the routine torture practised in so many Muslims states: the failure to accord women equal rights and a proper voice in the running of society; the absence of free expression, association and assembly; the lack of civil society and the persecution of religious minorities.

Religious intolerance is everywhere. A report by the Pew Research Centre shows a rising tide of religious restriction. The number of countries with high restrictions on religious belief rose from 31% in 2009 to 37% in 2010.

Muslims suffer as much as the followers of other faiths – they have been forbidden to build mosque minarets in Switzerland, for instance – but the growth in the persecution of Christians in Muslim societies is striking. In Pakistan, Christians face punishment and discrimination; in Indonesia, churches have been closed. In Egypt, the Copts are openly discriminated against in the courts. Across Africa, Christians are being attacked in their churches; in Kuwait, the "villa-churches" used by foreigners are being forcibly shut; in Iraq, Christians are targeted by kidnappers and bombers; in Iran, 300 Christians have been arrested since 2010.

And let's not forget that minority sects in Muslim societies that suffer as well. The point is that while Muslim spokesmen demand respect for their faith they are not often prepared to give it to others, and in the west this inconsistency is a significant obstacle to a better understanding of the offence taken from the film and cartoons.

Free speech is at the heart of this issue, but western governments should not be bamboozled into imposing restrictions that exceed present laws. On the contrary, we should proclaim its value, because the kind of free debate the young people in the Arab Spring yearned for will bring about transformation in states where torture, discrimination against women and religious minorities and the failure to build modern civil societies all go unchallenged.

• This article was amended on 27 September 2012. The original referred to the ban on building mosques in Switzerland. That should have been mosque minarets, and has been corrected.