Michael Fagan makes an unlikely criminal mastermind. The architect of the biggest royal security breach of the 20th century – sitting in a Wetherspoon's pub, sporting socks, sandals, an oversized parka and a winter hat with ear flaps – is more of a contender for the title of Britain's Most Embarrassing Grandpa. Nevertheless, in 1982 a 32-year-old Fagan scaled the barbed-wire-topped, 14ft wall of Buckingham Palace and shinned up a drainpipe before wandering into the Queen's bedroom and a place in history.

As the Queen celebrates her Diamond Jubilee, it is unlikely she'll want to dwell on the memory of the security debacle that allowed the unkempt, bare-footed and slightly tipsy Fagan to pull back the curtains of her four-poster. It has been decades since he gave an interview, but he agreed to meet The Independent on Sunday last week at his local pub in north London's Holloway Road.

He says he doesn't like giving interviews, but warms to telling his tale pretty quickly. "I was scareder than I'd ever been in my life," he says, widening his eyes theatrically as he recalls the moment he pulled back the curtains to see the Queen staring up at him. "Then she speaks and it's like the finest glass you can imagine breaking: 'Wawrt are you doing here?!'"

Download the new Independent Premium app Sharing the full story, not just the headlines

He insists he has "great respect" for the Queen. Not apparently as great as the pleasure he takes in sharing the details of his moment in the royal chamber: "It was a double bed but a single room, definitely – she was sleeping in there on her own," he giggles. "Her nightie was one of those Liberty prints and it was down to her knees."

Reports at the time suggested the Queen had a long conversation with Fagan to stall him while security was summoned. Fagan tells it differently: "Nah! She went past me and ran out of the room; her little bare feet running across the floor."

Why he also had bare feet has long been a mystery. He clears this up: "I got my sandals returned to me two years later by the security guard. 'These are Michael's sandals, we found them on the roof,' they said."

Before pitching up rudely at the Queen's bedside, Fagan wandered around the palace, via King George V's multimillion-pound stamp collection, triggering the alarm twice. Police turned it off – assuming the warnings were errors. The resulting scandal prompted the then Home Secretary, Willie Whitelaw, to offer his resignation to the Queen.

When, finally, the Queen managed to summon help, it was an unarmed footman who stood watch until the police came. Fagan recalls, with increasing licence: "The footman came and said, 'Cor, fucking hell mate, you look like you need a drink'. His name was [Paul] Whybrew, which is a funny name for someone offering you a drink, innit? He took me to the Queen's pantry, across the landing, where I presume she cooks her baked beans and toast and whatever – and takes a bottle of Famous Grouse from the shelf and pours me a glass of whisky."

He says the night of his arrest was not the first time he had sneaked into the palace: a month earlier he broke in and managed to spend most of the night inside before leaving again undetected. His wife, Christine, had just left him. He climbed in through the window of a shocked maid's bedroom; she ran straight to security. But when they found nobody in her room they thought she'd imagined it, leaving Fagan free to explore undisturbed.

He didn't make it as far as the Queen's chamber on that occasion: "I found rooms saying 'Diana's room', 'Charles's room'; they all had names on them. But I couldn't find a door which said 'WC'. All I found were some bins with 'corgi food' written on them. I was breaking my neck to go to the toilet. What do I do? Pee on the carpet? So I had to pee on the corgi food. I got into Charles's room and took the wine off the shelf and drunk it. It was cheap Californian.

Daily coronavirus briefing No hype, just the advice and analysis you need Enter your email address Continue Continue Please enter an email address Email address is invalid Fill out this field Email address is invalid Email already exists. Log in to update your newsletter preferences Register with your social account or click here to log in I would like to receive morning headlines Monday - Friday plus breaking news alerts by email Update newsletter preferences

"I was loving it... It was like Goldilocks and the Three Bears; I tried one throne and was like 'this one's too soft'. I was having a laugh to myself because there was one right next to it, so I tried another. He demonstrates how he reclined on a chair to view the Queen's art, putting his feet up on the pub table: "I was sitting like this – see. I liked the picture and thought I'd look at it till someone comes, but nobody came.

"It was harder to get out than get in. I eventually found a door and walked out into the back gardens, climbed over the wall and walked down the Mall, looking back and thinking 'ooh'. I hadn't thought about going in there until that last second when it came into my head to do it, so I was shocked."

Days after the first break-in he was arrested for stealing a car in London, driving it to Stonehenge in search of his wife. He was sent to Brixton prison and after three weeks was released on bail. The next day he went to the palace for the second time. He has no regrets. "It's brought me adversity, but I can laugh about it and that's the main thing. I wouldn't do it again. I think security is tightened up now."

Even all these years later, he cannot explain his motivation. "I don't know why I did it, something just got into my head," he says, breaking into a Pink Floyd song: "There's someone in my head and it's not me..." Describing his second visit, he adds: "I went back because I thought 'that's naughty, that's naughty that I can walk round there'." He suggests the whole incident stemmed from putting too many magic mushrooms in his soup five months earlier. "I forgot you're only supposed to take a little handful. Two years later I was still coming down. I was high on mushrooms for a long, long time."

Drugs have played a big part in his life, one way or another. The oldest of three children, he was brought up in north-east London by his mum, Ivy, and dad, Michael, who was a steel erector and a "champion safe-breaker".

He left home at 16 and started working as a painter and decorator, but never really developed an accommodation with the world of work. Before the palace break-in, life revolved around petty crime, drugs and getting up to mischief. When the brief flare of fame in the Eighties after the break-in – he released a version of "God Save the Queen" with the Bollock Brothers – sputtered, he went back to what he knew.

In the past three decades he has been charged with – among other things – assaulting a police officer, dealing heroin and indecent exposure. The last was a "misunderstanding" while smoking dope and fishing with friends. Diving into the water to retrieve a net, he took his trousers off because they were wet and was seen by a woman from a distance. He is indignant: "It was said in court 'he had a huge erection', but this woman can't have been from this planet! Her husband must be like that," he says, measuring a tiny distance with his thumb and forefinger.

In truth he is more prankster than gangster. He blames his heroin-dealing conviction – his most serious – on getting back together with his ex-wife, Christine, "When she came back I got into it. I had to try it and it did cause me a bit of grief. I got four years for dealing. The people I was serving, one was a company director, the son of a lord. They were all business people and they liked coming to me."

Though he talks about drug dealing in the past tense he seems high during our interview: his pupils like pinpricks as he leaps constantly up and down on his seat. I ask if he still does drugs? He sniffs and allows his mouth to stretch into a knowing grin.

He describes himself as "retired" now, though he's never been much of a nine-to-five man.

As we take our leave, I ask if he has a message for Her Majesty in her Diamond Jubilee year. "Yeah, 60 years – that's fucking great! I hope she beats Victoria. I hope she lives to be a hundred. If she does, I'll send her a hundredth-birthday telegram."

Curriculum vitae

1950 Born in Clerkenwell, London, to Michael and Ivy Fagan; two younger sisters, Margaret and Elizabeth.

1955 Attends Compton Street School, London.

1966 Leaves home at 16 to escape his father, who, Fagan says, was violent. Works as a painter and decorator.

1972 Marries Christine, with whom he has four children.

1982 Breaks into Buckingham Palace twice in a month; the second time he makes it into the Queen's bedroom and speaks to her. Home Secretary Willie Whitelaw offers his resignation over the security breach; the Queen refuses. Fagan is sent to Brixton Prison and Park Lane secure mental institution on unrelated offences of taking a car and assault.

1983 Releases a version of "God Save the Queen" with the Bollock Brothers.

1984 Attacks a policeman in a café in Fishguard, Wales, and is given a three-month suspended jail sentence.

1987 Found guilty of indecent exposure after a woman motorist saw him running around with no trousers on at a waste ground in Chingford, Essex.

1997 Fagan, his wife and their son Arran, 20, are charged with conspiring to supply heroin. Fagan goes to prison for four years.

2002 Witnesses Uri Geller and Michael Jackson board a train at Paddington Station.