On 24 June, while I was on stage at the Comedy Store in London, a man told me to “go home”. The eagle-eyed among you will recognise that date as the night after the UK voted to leave the EU. Obviously, it was jarring. No one has told me to go home for 16 years, and I assumed the idea had been forgotten. It’s the racist equivalent of Limp Bizkit, or being worried about the millennium bug. Also, I come from Croydon, so telling me to go home really is incredibly cruel.

I dealt with the gentleman at the time, using some language and terminology that could generously be described as “fruity”. I hoped that this would be an isolated incident, the consequence of discussing this subject in the eye of the storm. However, I was wrong. According the government data, 6,193 hate crimes have been reported since the referendum, a 20% rise on the same period in 2015. This led to home secretary on Tuesday to announce that police handling of hate crimes will be reviewed.

Now, clearly not everyone who voted to leave the EU is a racist (a phrase I’ve found myself saying so often in the last three weeks, I’m thinking of having it printed on a T-shirt). But these figures do suggest that the intolerant have been emboldened by the result. Last week, Barclays claimed that the referendum had “uncorked a genie” that was causing economic instability. Well, it turns out that genie is also a huge racist. While that isn’t ideal, it is a great idea for a gritty Christopher Nolan-style reboot of Aladdin, where the racist genie has to reluctantly do the bidding of an Arab while intermittently muttering: “Coming over here, taking our wishes.”

It’s easy to be angry with people. But you have always got to look at the root causes of the problem. In his song Only a Pawn in Their Game, Bob Dylan takes on the story of the murder of civil-rights activist Medgar Evers, but instead of just blaming the assassins, Dylan widens his scope:

A South politician preaches to the poor white man

“You got more than blacks, don’t complain

You’re better than them, you been born with white skin,” they explain

And the Negro’s name

Is used it is plain

For the politician’s gain

As he rises to fame

And the poor white remains

On the caboose of the train

But it ain’t him to blame

He’s only a pawn in their game.

Those words have rattled around my head recently along with two key questions: if we are all the pawns, then who are the grandmasters playing us? Also, when you think about it, what is chess, if not a way of encouraging slow, tactical racism?

For me, at least some of the blame lies with the leadership of the leave campaign. They stoked intolerance, and then refused to take responsibility. In the lead-up to the vote, a pro-Brexit minister told Newsnight: “The only issue we can go on the offensive is on immigration.” The hostility was relentless, from the campaign and its supporters in the media. The writer Liz Gerard found that between 1 January and the date of the referendum, the Daily Mail and the Daily Express each published 34 front-page articles about immigration.

In case you were wondering, none of them were positive. There were no headlines such as: “Immigrant saves boy from killer shark” or “Foreigners make a lovely jam”. There were 174 days between those dates – which means two national newspapers carried front-page stories about immigrants once every five days. What other subject dominates the headlines once a week for six months? The answer is football, a sport that, ironically, in this country is played largely by a talented group of economic migrants.

Boris Johnson and Daniel Hannan have both claimed that there is no relationship between the campaign and the rise in hate crimes. Then last week, Nigel Farage made a cameo at the Republican national convention, bemoaning the xenophobia of other countries and claiming that Donald Trump’s attitude to Muslims makes him “very uncomfortable”. I’d say that this was the pot calling the kettle black, but Farage would probably suggest that I was accusing him of being racist to kettles. Honestly, it’s political correctness gone mad.

In his convention speech, Trump claimed that immigration has impacted negatively on African-Americans and Latinos, simultaneously attempting to position himself as the champion of non-white people, while legitimising the views of people who hate them. His doublespeak is working – a CNN poll on Monday put him ahead of Hillary Clinton. An army of racist genies are licking their lips in anticipation, sensing the cork loosening. In spite of this, I remain optimistic. The audience at the Comedy Store that night came to my defence. Leave and remain voters united in shouting my heckler down. It’s good to remember that non-racists are in the majority. But we need to ensure we make our voices heard. Otherwise, we’re just pawns in their game.

Nish Kumar’s Actions Speak Louder Than Words Unless You Shout The Words Real Loud runs daily during the Edinburgh fringe festival, before touring in the autumn