He was always on a knife edge, Spike Milligan. Watching him could go either way: it was either going to be fabulous, or like having lemons squeezed in your eye. It was a bit like watching your ancient grandmother having breakfast – you never knew whether she was going to find her mouth that morning. And, the evil little shit that I was, I quite liked all that.

I had The Goon Show on tape, and I remember feeling that I had to watch these guys. Even though the show could be uncomfortable and embarrassing beyond belief, it could also be the funniest thing you'd ever heard. I don't think the Pythons would have been in existence had it not been for Milligan and the Goons.

Milligan showed me that there are no limits. He changed the way that people thought about comedy. I was brought up going to the theatre and seeing safe jokes about a woman's knickers. I could hardly believe that the BBC had given Milligan a job.

He refused to fit it, and I liked that as well, because he just didn't give a toss. When Prince Charles said that Milligan was his favourite comic, the comedian went on the Parkinson show and called the prince a "little grovelling bastard" on live TV. That's a massive part of his appeal, but even bigger than that was the fact Milligan made you laugh so much that you had to leave the room because it was getting ridiculous. I can't remember anyone else doing that at the time or since.

I met Milligan when he was much, much older on the set of Gormenghast, which we were both in. I was over the moon about meeting him – I was going to push Spike Milligan around in a wheelchair. I thought it would be great. But it wasn't, it was just incredibly sad. They say you should never meet your heroes – there was nobody in; Milligan was gone.

As told to Vicky Frost

• Gregor Fisher plays Rab C Nesbitt. The show returns to BBC2 this autumn.