My first ever gig was at a Pontins holiday camp when I was nine. It had a talent competition and I decided to enter as a standup, with a book of jokes from which I took all of my material. A lot of the jokes seemed to have no problem playing with the stereotype of Irish people being stupid. They were incredibly racist, but the bigger crime here, comedy-wise, was that my set was entirely built on stolen material.

There was a joke about a man buying a Rolls-Royce who is a bit short of cash. He goes out into the street to see if someone can lend him 10p to make up the price. He bumps into an Irish man – we know he’s Irish, because his name is Paddy. Paddy says, “Here’s 20p. Buy me one as well.”

The point of that joke is that Paddy is incredibly stupid, because he’s assumed that Rolls-Royces cost 10p. I’d never even met an Irish person, but I had the clear impression that they were all thick – and the Pontins crowd seemed to agree: the joke went down a storm.

That was one of the many hugely racist jokes I told as part of my set that night at Pontins, Camber Sands, which I also delivered completely in a Sri Lankan accent – a very early recognition of the fact that my ethnic minority roots could be exploited for comedic value.

When I was working in a supermarket I spoke to customers in as many different accents as I could – Irish, Welsh, Chinese

Years later, when I was working as a trolley wally in a supermarket, I tackled the boredom by talking to the customers in as many different accents as I could manage. I started with one that I didn’t think would alert any suspicion – generic Asian – then moved on to Irish, Welsh, Australian and American. I am pretty sure I even smashed out a Chinese voice. I discovered it takes a very ballsy person to say, “I’m sorry, but I don’t think that’s how you really speak”, particularly to an Asian-looking, English-speaking guy with a lazy eye.

Anyway, I won the Pontins competition. There were only two other contestants, and one was a girl playing a kazoo. She was shit, but at least she was performing her own material. I can’t remember what the other guy did. I didn’t care, because this fat Asian kid reading out loud in a fake accent blew them both out of the water.

***

Twenty two years later, I finally decided to capitalise on that Pontins buzz. I was 31 and a teacher at the time, and I think I was actually aspiring to be a frustrated artist of some kind, in the way all teachers are. I Googled “comedy night” and found the Comedy Cafe in Shoreditch, London. It had a new-act session on a Wednesday, and you could phone up for a spot. I called and was told by a very nice lady that you were allowed to do the show only if you had done 20 or more gigs before. I lied and was booked for that week.

When I arrived at the venue, I became immediately aware that I had made a mistake. The other comics on the bill hadn’t lied. They all knew each other, too, and were chatting excitedly about the evening ahead. I looked over the jokes I’d written on the train on the way to the gig. There is a magical, unexplainable phenomenon that still occurs to this day where, however funny you think the material you’ve written is, as soon as you turn up to a gig to try it out, it becomes almost incoherent.

Gradually the club started to fill. The people looked terrifying – comedy connoisseurs who were going to hate everything I was doing. I started my set using my Sri Lankan accent. My plan was to reveal partway through that I don’t actually have a Sri Lankan accent. I imagined this would lead to the audience collapsing in fits of laughter, before I was carried out on their shoulders.

What actually happened was that I did my big reveal and it got absolutely nothing. Then I remembered I’d nicked the idea off Omid Djalili. This was a lot of information to process, and I quickly descended into a panicked garble. I overran badly. I went on for such a long time that the guy running the night flicked the stage lights on and off, to signal to me that I’d outstayed my welcome. I didn’t understand at the time why the lights had flickered, and proceeded to deliver some witty riffs about the electricity failing.

This was delivered to more silence. I then said my thank yous and left the stage to the sort of rapturous applause that you might expect if you started juggling at a funeral. People were just grateful I’d finished. I soon realised I’d got myself blacklisted at the Comedy Cafe for quite some time. On the plus side, I’d got some stage time under my belt and a possible introduction to Djalili if he wanted to have a go at me for ripping off his material.

***

I don’t believe in make-or-break moments in your life. If you screw something up, it can knock you down, but that only means you’ll be better when you get back to where you were before. I also think luck has a huge part to play. You can be good at comedy, which means you’ll be given spots, but beyond that it is luck that pushes you to the next level. There are loads of brilliant comedians who haven’t had the breaks, and plenty of average comedians who have. I don’t think I’m the best comedian they could have got for anything I’ve done. And if I ever forget that, I can just look at Twitter, where I’ll quickly find someone saying, “Can somebody explain to me why somebody as unfunny as @RomeshRanga keeps managing to get on TV?”

The one thing that all ethnic minority comedians seem to have to contend with is people telling them they talk about race too much. This happens if they mention it at all. There are certain people who, when they see a female or an ethnic minority comic, are basically waiting for them to mention something that they can accuse them of talking about all the time. For women, it’s periods and relationships; for ethnic minority comics, it’s race. I’ve talked about this with the comic Rob Beckett, who said he joked about his teeth and being working class a lot, yet he’d never been accused of talking about either too much. If a woman wants to go on about periods or I want to talk exclusively about race, that should not be an issue. We should be letting market forces decide. If audiences stop laughing, that’s your clue – but if not, then it’s nobody’s right to pick you up on it. They can always go elsewhere.

‘Suggesting I talk about race too much is a pretty racially intolerant position to adopt.’ Photograph: Pål Hansen for the Guardian. Grooming: Kristopher Smith for Terri Manduca

One night, we were filming Mock The Week, on which I’ve appeared a few times, when Hugh Dennis made an Isis joke. I said it would have gone better if I’d laughed, too, so that people felt reassured. It became a running joke throughout the recording that every time somebody told a gag that had any connection with race, I would laugh loudly and everyone would feel better.

The night that show went out I got absolutely slaughtered on Twitter: “@RomeshRanga talking about race again. Quelle surprise”; “@RomeshRanga waited a whole two minutes before talking about his race”. I found it unfair. People seem to want a moratorium on anybody discussing anything that doesn’t directly affect them, which means I’m being dictated to by people who have no direct experience of the subject I’m discussing.

I even had a former student do it. I was walking through a bar with a couple of friends when he tapped me on the shoulder. “Hey, Sir [all ex‑students call you “Sir” for ever], loving your shows, but just a little thing: drop the race stuff.” I was extremely disappointed that somebody I’d taught had fallen into that trap.

If you can make all of my life experiences the same as a white guy’s, and you can stop me experiencing racism, which I do all the time, then I’ll stop talking about race. And if you have ever tweeted me or spoken to me suggesting I talk about it too much, that’s a pretty racially intolerant position to adopt.

To give you an example, I recently received this beautifully written message on Facebook: “I seen ur program about Albania, it was very intentional to make albania look bad,a u paki or smelly indian!!! I seen many ptograms about your race,about Asian gangs raping underage girls,maby you one of them!!!” If I’m receiving messages like that, surely I’m allowed to do a joke or two with a head wobble, mate.

On another Mock The Week, I observed that old people should be allowed to be racist because they have enough going on. I thought nothing of it. The day after the show went out, I got a message saying something along the lines of, “Saw what you said about old people on Mock The Week. My grandparents didn’t fight in the war so that Pakis like you could express your views. If I ever found out where you lived I would end your worthless life.” I don’t recall ever claiming that his grandparents fought in the war so I could do Mock The Week, but I could confidently argue that they didn’t fight in the war so losers could type abuse at people from the comfort of their bedrooms.

I reported a death threat. The guy was so scared that the police asked if I would be satisfied with a letter of apology

I reported it to the police, not because I thought it was a credible threat, but because people shouldn’t be able to spout off like that without repercussions. Also, the guy had posted it to my website, which requires you to put in your email address, so it wasn’t as anonymous as he’d perhaps assumed.

The police took it seriously and got a full statement from me, then said they’d find this person and take action. They eventually located the guy’s house and spoke to him. He was so scared that the police told me it was probably overkill to press charges, and asked if I would be satisfied with a letter of apology. I said that would be fine; if anything, it would be nice to get a letter.

A couple of days later, I received a letter that painted a picture of a very different man from the one who’d sent a death threat to someone he’d never met. The letter said this person had been drinking wine while watching second world war documentaries, and was getting very worked up about history. He’d switched to BBC2, found me talking about old people and become enraged. He sent me an angry message. It all seemed a bit sad. But I still think he’s not a great guy. Nobody goes, “Isn’t it a nightmare when you have a touch too much chablis, then issue a death threat to a Paki?”

I’ve come to think of these social media trolls as cowardly hecklers, but I get my fair share of the real kind, too. One of the most effective I’ve faced came at a gig about a week after my dad died in 2011. (That sounds insane, but I really thought it would be good for me to get straight back into things; plus at that point, I really needed the money.) This was at Up The Creek, a great comedy club in Greenwich, London, and I love playing there – but the audiences can be feisty, to say the least. I started my set, feeling proud of myself for getting up onstage and doing 20 minutes of standup, like a brave little soldier, when a bloke in the front row decided to give me a racist heckle.

It was so effective that I’m actually slightly nervous to divulge it here. What the bloke did was deliver a “bud bud ding ding”. There was a lot to admire in the way he did it. First, he said, “Bud bud ding ding” – that’s proper old-school racism and I respect that. Second, he said it loud enough only for me and his mates to hear, which meant that if I addressed it, I’d be drawing it to the attention of people who weren’t aware it had happened. Third, and possibly the most impressive of this heckle’s attributes, he repeated it every minute or so during my set. I performed the whole thing without acknowledging I was having “bud bud ding ding” repeated at me from the front row. If that bloke reads this, which is unlikely: well played. You’re a prick.

Some comedy clubs are really hot on making sure hecklers get dealt with. Komedia in Brighton is particularly good at it, but on one occasion it did backfire. I was doing a set about the difficulties of taking your children on holiday when one of the crowd shouted, “Just sack off the little ragamuffins!”

I was surprised to hear such an old-fashioned word, so I said, “Sorry, mate, did you just use the word ‘ragamuffin’?” The Brighton in the crowd came out then, as somebody else said: “You cannot say that to him. That’s racist!”

I have no idea how it’s racist. This person just heard “ragamuffin”, made an association with Jamaican patois and took that to mean the guy was being racist, which I think in itself might be quite racist. Suddenly the bouncers came up and grabbed the guy to throw him out.

I started protesting, but they put their hands up to say, “We’ve got this, don’t worry.” Next thing I knew, he was being frogmarched out.

I finished my set, then ran outside to find him and get him back into the club, but he had gone. I felt terrible. If by any chance he’s reading this now, I’d like him to know that I’m sorry.

But, really, you shouldn’t be so racist, mate.

• Adapted from Straight Outta Crawley: Memoirs Of A Distinctly Average Human Being, by Romesh Ranganathan, published by Transworld on 4 October at £20. To order a copy for £17.20, go to guardianbookshop.com, or call 0330 333 6846.

If you would like a comment on this piece to be considered for inclusion on Weekend magazine’s letters page in print, please email weekend@theguardian.com, including your name and address (not for publication).

Comments on this piece are premoderated to ensure the discussion remains on the topics raised by the article.