Fairytale of New York video. Warner Music

Once upon a time a band set out to make a Christmas song. Not about snow or sleigh rides or mistletoe or miracles, but lost youth and ruined dreams. A song in which Christmas is as much the problem as it is the solution. A kind of anti-Christmas song that ended up being, for a generation, the Christmas song.

That song, Fairytale of New York by the Pogues, has just been reissued to mark its 25th anniversary; it has already re-entered the Top 20 every December since 2005, and shows no sign of losing its appeal. It is loved because it feels more emotionally "real" than the homesick sentimentality of White Christmas or the bullish bonhomie of Merry Xmas Everybody, but it contains elements of both and the story it tells is an unreal fantasy of 1940s New York dreamed up in 1980s London. The story of the song is a yarn in itself: how it took more than two years to get right and became, over time, far bigger than the people who made it. As Pogues accordion-player James Fearnley says: "It's like Fairytale of New York went off and inhabited its own planet."

Appropriately for a song that pivots on an argument, there is disagreement as to where the idea originated. Fearnley, who recently published a memoir, Here Comes Everybody: The Story of the Pogues, remembers manager Frank Murray suggesting that they cover the Band's 1977 song Christmas Must be Tonight. "It was an awful song. We probably said, fuck that, we can do our own."

Singer Shane MacGowan maintains that Elvis Costello, who produced the Pogues' 1985 masterpiece Rum, Sodomy & the Lash, wagered the singer that he couldn't write a Christmas duet to sing with bass player (and Costello's future wife) Cait O'Riordan.

Either way, a Christmas song was a good idea. "For a band like the Pogues, very strongly rooted in all kinds of traditions rather than the present, it was a no-brainer," says banjo-player and co-writer Jem Finer. Not to mention the fact that MacGowan was born on Christmas Day 1957.

The Pogues had formed amid the grimy pubs and bedsits of London's King's Cross in 1982. Although their name ("Pogue mahone" means "kiss my arse" in Gaelic) and many of their influences were Irish, most of the band weren't, and their interest in folk songs and historical narratives roamed far and wide. They aspired to timelessness.

'It's for the underdog' … Kirsty MacColl and Shane MacGowan promote Fairytale of New York. Photograph: Tim Roney

Finer first tried writing a song about a sailor missing his wife at Christmas but that was dashed on the rocks by his own wife, Marcia Farquhar, who called it "corny", says Finer. "So I said OK, you suggest a storyline and I'll write another one. The basic plotline came from her: this idea of a couple falling on hard times and coming eventually to some redemption." He says there's a "secret history" to the story: "a true story of some mutual friends living in New York."

MacGowan, whose contribution to this piece comes in the form of a dialogue written by long-term partner and biographer Victoria Mary Clarke, declines to elaborate: "Really, the story could apply to any couple who went anywhere and found themselves down on their luck."

While Finer retained the uptempo reel from his abandoned maritime tale, MacGowan worked on the slower verses and chorus. The singer had never seen New York but it was on his mind. As the Pogues toured Europe in autumn 1985, they almost wore out a video of Once Upon a Time in America, Sergio Leone's epic tale of Jewish mobsters in interwar New York. (Ennio Morricone's elegiac title theme seeped into Fairytale's opening melody: don't all good fairytales start with "Once upon a time"?)

In Here Comes Everybody, Fearnley writes: "A stable perception was never reachable as to whether Shane was a genius or a fucking idiot." There is the public image of MacGowan as a wayward alcoholic with a bombsite of a mouth and a wheezing ghost of a laugh. Then there is the clever, diligent craftsman who sweated for two years to make Fairytale of New York perfect.

The first demo was recorded by Costello at the same time as The cinematic romance of A Rainy Night in Soho, MacGowan's first song to draw on his love of Frank Sinatra and Judy Garland. When he brought that song into the studio in early 1986, Fearnley remembers: "He meant business, much more than before. It was awe-inspiring to see him in the rehearsal room with his suit on and an attitude."

But Fairytale of New York, from the same session, doesn't sound like an embryonic classic. The melody limps, the lyrics stumble and the action begins in Ireland: "It was a wild Christmas Eve on the West coast of Clare," sings Cait O'Riordan. "I looked 'cross the ocean, asked what's over there?" Clearly, it needed work. "I don't think the band was capable of playing the song as it needed to be played at that point," says Finer. "Shane and I batted arrangements around for ages and we'd periodically try and record it. Shane's a tireless and meticulous editor."

"Every night I used to have another bash at nailing the lyrics, but I knew they weren't right," says MacGowan. "It is by far the most complicated song that I have ever been involved in writing and performing. The beauty of it is that it sounds really simple."

Costello prosaically suggested calling it Christmas Day in the Drunk Tank. MacGowan pointed out that this did not sound like a hit. At the time Finer was reading JP Donleavy's 1973 novel A Fairy Tale of New York, the picaresque story of a bereaved Irish-American's return home from Ireland to Manhattan. MacGowan later visited the novelist to ask his blessing to borrow the title. (Years later, Donleavy told the BBC that he loved the song but "realised straight away that it didn't really have anything to do with my book".)

A short time later, in February 1986, the Pogues finally made it to New York itself, to start their first ever US tour, and they weren't disappointed. "It was a hundred times more exciting in real life than we ever dreamed it could be!" says MacGowan. "It was even more like New York than the movies!" After their debut at a club called the World, their backstage visitors included Peter Dougherty, who came to direct the video for Fairytale of New York, and actor Matt Dillon, who appeared in it. MacGowan remembers Dillon, the rising star of Rumble Fish and The Outsiders, kissing his hand and saying: "I dig your shit, man, I love your shit!"

It was a year later that Murray approached U2 producer Steve Lillywhite to helm the next Pogues album. The sessions at London's RAK studios in the unusually hot summer of 1987 went so well that the band decided to have another crack at Fairytale. When they said they were struggling to blend MacGowan and Finer's sections, Lillywhite's solution was absurdly simple: record them separately and edit them together later. "It was a beautiful time," says Lillywhite. "I got the Pogues when they were really firing and before too much craziness got involved. As long as I got them early in the day it was great."

Fearnley was tasked with arranging the strings (a job completed by Fiachra Trench) and colouring in the scene- setting piano part, drawing on Tom Waits, Aaron Copland and Leonard Bernstein's score for On the Waterfront. "I wanted to get American music into it," he explains. MacGowan originally wanted the orchestra to interpolate the refrain from Have Yourself a Merry Little Christmas but "Phil Chevron [Pogues guitarist] told me that was a bad idea. He was right."

Only one hurdle remained. Cait O'Riordan had left the band in October 1986, leaving nobody to complete the duet. "I think at some point almost any female with a voice was a contender," Finer jokes, mentioning fellow RAK clients Chrissie Hynde (feasible) and Suzi Quatro (less so). "One person I certainly hadn't thought of was Kirsty [MacColl] and I don't think anyone else had."

"To be honest they weren't 100% convinced that Kirsty was the right person," says Lillywhite, who was married to MacColl. She was well-liked but her solo career was becalmed due to stage fright and contractual problems. Lillywhite suggested recording MacColl's part at his home studio over the weekend and seeing what the band thought. "I spent a whole day on Kirsty's vocals. I made sure every single word had exactly the right nuance. I remember taking it in on Monday morning and playing it to the band and they were just dumbfounded."

But MacGowan, who was so impressed that he re-did his own vocals, insists: "I was madly in love with Kirsty from the first time I saw her on Top Of The Pops. She was a genius in her own right and she was a better producer than he was! She could make a song her own and she made Fairytale her own." (Since MacColl's tragic death in 2000, her part has been taken by singers including Sinéad O'Connor, Cerys Matthews, Katie Melua, Victoria Clarke and Jem Finer's daughter Ella.)

In the finished version the story finally acquires the ring of truth, but it's still teasingly elliptical. Does the argument take place after the man leaves the drunk tank or does the whole song unfold in his sozzled head? After all, Once Upon a Time in America is told almost entirely in flashback. And while the "cars big as bars" and the singing of Galway Bay (a 1948 hit for Bing Crosby, beloved of Irish immigrants) place the action in the 1940s, MacGowan suggests that the characters are much older, remembering their glory days.

And can we trust the narrator anyway? "The guy is a bum who is living on the street," says MacGowan. "And he's just won on a horse at the unlikely odds of 18-to-one, so you're not even sure he is telling the truth." He says that both characters are versions of himself. "I identified with the man because I was a hustler and I identified with the woman because I was a heavy drinker and a singer. I have been in hospitals on morphine drips, and I have been in drunk tanks on Christmas Eve."

The song's brilliance is sealed by its final verse when MacGowan protests, "I could have been someone", and MacColl shoots back: "Well, so could anyone." Then MacColl accuses, "You took my dreams from me," and MacGowan responds, with all the warmth he's been withholding: "I kept them with me babe/I put them with my own." So in its final iteration the chorus is no longer a tauntingly ironic reminder of better times but the tentative promise of reconciliation. "You really don't know what is going to happen to them," says MacGowan. "The ending is completely open."

The Pogues shot the video in New York during Thanksgiving week. The air was bitterly cold and fairy lights twinkled in the trees. Matt Dillon played the NYPD officer who arrests MacGowan but he was too nervous to manhandle him until the shivering singer snapped: "Just kick the shit out of me and throw me in the cell and then we can be warm!" Contrary to the lyrics, the NYPD didn't have a choir, so Dougherty hired the force's pipe band instead. When it turned out that they didn't know Galway Bay, they mouthed the only lyrics they all knew: the Mickey Mouse Club chant.

In the black-and-white performance footage, closely modelled on a BBC2 documentary about Billie Holiday, it was decided that MacGowan should sit at the piano while Fearnley wore the singer's rings to imitate him for the closeups. "I'm the fucking piano player and I wanted people to know that," says Fearnley. "It was absolutely humiliating. But it looks better. You have to find your proper place for the benefit of the project."

Fairytale of New York had a galvanising effect on everyone involved. When MacColl joined the Pogues on tour, she gained the confidence to relaunch her solo career, and the Pogues only narrowly lost the Christmas No 1 to the Pet Shop Boys' Always on My Mind. "Going to No 1 in Ireland was what mattered to me," MacGowan says now. "I wouldn't have expected the English to have great taste!" For Lillywhite: "I love the fact that it's never been No 1. It's for the underdog."

This Christmas, as the song enters the charts for the 10th time, the Pogues will play a show to celebrate their 30th anniversary. Although they fired MacGowan in 1991 ("What took you so long?" he replied), they reunited a decade later. So Fairytale of New York has ended up being a parable of the band's life together: the youthful optimism, the bitter recriminations, the uncertain detente.

"We told a similar story ourselves," agrees Fearnley. "We've all had hopes and we've had our conflicts, but there's some other damn thing that's binding us all together and hopefully always will."

The ending is completely open.

Fairytale of New York is out now on seven-inch vinyl on Rhino and digitally on iTunes. The Pogues play the O2 in London on 20 December. With thanks to Victoria Mary Clarke.