In Sex & Drugs & Rock & Roll, the new Ian Dury biopic, there is a scene that faithfully records the first time Dury met his songwriting partner, Chaz Jankel. It is May 1976 and the singer has just hobbled off stage after a particularly ramshackle London pub gig with his band of bedraggled misfits, Kilburn & the High Roads. The young, clean-cut Jankel strolls into the dressing room, grinning widely, and introduces himself. "Do I know you?" asks Dury, fixing him with a malevolent stare. "No," replies Jankel, still grinning. "Well do us a favour then," barks Dury, "and fuck off!"

Kilburn's guitarist Ed Speight convinced Jankel to return to the dressing room. In doing so, Jankel changed the course of Ian Dury's music and his life.

"The Kilburns were just about finished," Dury's onetime manager, Peter Jenner, told me, "and Ian was seen in the music press and the business in general, if it noticed him at all, as someone who'd had his chance and blown it. Then along comes this nice middle-class Jewish kid who turns out to be a genius with melody and composition. It was just what Ian needed but his first instinct is to tell Chaz to fuck off. That was Ian in a nutshell."

Next year will be the 10th anniversary of Ian Dury's death, at 57, from cancer. Back then, in March 2000, the string of brilliantly absurd, effortlessly funky and oddly old-fashioned sounding songs he created with Jankel – "Sex & Drugs & Rock & Roll", "Hit Me With Your Rhythm Stick", "Reasons to be Cheerful, Part 3" – were played constantly on the radio, while Paul McCartney and Elvis Costello paid tribute to his songwriting. The funeral at Golders Green crematorium was attended by an array of friends that included a cabinet minister, Mo Mowlam, as well as pop's then-current cheeky chappie, Robbie Williams, who had succeeded Dury as celebrity campaigner for Unicef. Madness, who occupied a similar place in the nation's heart in the 1980s as Dury and his group the Blockheads had in the late 1970s, acted as pallbearers. A Times obituary praised the singer's "Swiftian satirical streak" and acknowledged his "lasting place in the corpus of the English popular song".

In death, Dury, if only for a moment, seemed to have become what he could never have been while alive: a national treasure. He escaped that dubious accolade because his persona, like many of his songs, was too rough and ready, too dark and edgy, to be truly subsumed into the nation's collective consciousness. Now, with the imminent release of the film Sex & Drugs & Rock & Roll, featuring an extraordinarily convincing central performance by Andy Serkis, and the publication in January of journalist Will Birch's Ian Dury: The Definitive Biography (Sidgwick & Jackson), Dury's canonisation may yet happen, but somehow I doubt it.

As both film and book show, Dury was, in his own parlance, "a tricky customer", by turns needy, wilful and truculent and possessed of a savage wit that could regularly reduce those closest to him to tears. He was way too complicated – and clever – to be truly lovable. "He was a brilliant storyteller and incredibly funny but he was also difficult and very stubborn," says his son, Baxter, now a singer and songwriter too. "As he got older he could be extraordinarily difficult but he was saved from wankerdom by his humour and his honesty. It was all very complex."

The film is a not altogether faithful, but often insightful, evocation of Dury's wild life and times. It does not shy away from his difficult nature or his edginess. The pace is dictated by the unruly thrust of Dury's relentlessly chaotic life and his complicated relationship with the two women in it, his first wife, Betty, and his girlfriend, Denise Roudette. It's exhilarating, if exhausting, to watch: the chaos increased as Dury's long-awaited success finally arrived, and was fuelled by copious amounts of the requisite sex and drugs as well as an ever-increasing retinue of colourful hangers-on and properly dodgy geezers, the most loyal and scary of whom became Dury's minder and was christened by him "the Sulphate Strangler".

"It was a scarily male world at times," says Dury's daughter, Jemima, "but some of the dubious criminal types turned out to be the most charming and thoughtful. The dicey ones were the rock'n'roll misfits that punk threw up who always seemed to find their way on to the bus and into the hotel room. I guess that was just how it was back then."

One thing the film does highlight, perhaps accidentally, is the lunar distance between today's pop universe and that of 30 years ago. It is well nigh impossible to imagine a Dury figure – eccentrically brilliant, utterly English, essentially untameable and physically disabled – emerging into today's post-Pop Idol pop mainstream after years of dues-paying on the pub-circuit wilderness.

When the song that gives the film its name was released in 1977, capturing the dissolute dynamic of the time, Dury was 35 – old in pop terms, and even more so in the punk revolution of that year. Despite his age, though, he was regarded as an honorary punk, revered by the likes of the Clash and the Sex Pistols as both a survivor and a kindred spirit, as well as someone they had studied closely on their way up.

"You went to see Dr Feelgood, the Alex Harvey Band and Kilburn & the High Roads, and that was it as far as raw British rock'n'roll was concerned in the mid-70s," remembers Paul Simonon, later of the Clash, "They were the tough guys and Ian was one of the toughest. You felt he was coming from the same place as you."

Dury's press officer, Kosmo Vinyl, remembers seeing the Kilburns in their heyday. "I was only a kid and I was taken aback and then mesmerised. The trouble was I could never find anyone to go again. They were just too chaotic, too threatening. Then suddenly punk made it all right to be like that."

Dury, though, was initially less than generous towards the emerging punk figureheads, often castigating Johnny Rotten for stealing his self-styled razor-blade-on-a-safety-pin earring and that hunched-over, amphetamine-eyed microphone stance. The Clash, until they invited him on tour, were dismissed as "public school punks".

For all that, it was punk's anarchic thrust and scorched-earth ideology that gave Dury, with Jankel at his side, the space to realise himself musically. "His idea of rhythm was based on Zulu," says Jankel, laughing, "That scene where you can hear the Zulus before you can see them, that incredible stomp before they come over the hill. It was the rhythm of survival and, as it happened, it suited the times perfectly."

Dury's first, and best, album, New Boots and Panties!!, from 1977, was originally going to be called Live at Lourdes. That irreverent title nodded at Dury's disabled status and, as it turned out, signalled his semi-miraculous reinvention as a performer. For a few exhilarating years, when the musicians he christened the Blockheads took their pl aces on stage and he ambled wonkily on behind them, be-suited in best vintage, draped in polka-dot scarves and gypsy kerchiefs, with one of several exotic titfers perched on his head, you knew that something extraordinary was about to happen.

"There was a bit of Max Wall in there and a bit of music hall plus a bit of art school rock," says Peter Jenner, who produced New Boots…, "but the way he put it all together on stage made it utterly unique and utterly riveting. He was truly a one-off."

Just as punk exploded, Dury, minus the Kilburns, who had finally disintegrated, found a home on a fledgling independent label called Stiff, where the only rule was that every rule in the music business was made to be broken. "We were anarchists, I guess," says Stiff founder Dave Robinson, "but not in the way that punks used the word. We knew Ian's music and we saw his potential and we were prepared to put up with his antics. He was one of the most remarkable performers I ever worked with, and the one I most wanted to throttle on a regular basis. He had a lot of baggage. Whatever he had suffered as a child, I don't think he ever really processed. It was always there. It made him angry and impossible from time to time but it also made him a great artist."

He was born Ian Robins Dury on 12 May 1942 in Harrow – though he always claimed it was Upminster – to Margaret and William Dury, a nurse and a chauffeur respectively. "I'm a Mockney," he would cheerfully confess to Will Birch years later, "insofar as my mum spoke beautifully and my dad didn't. Dad was always telling me to sound my aitches, because he would worry about it, whereas my mum's lot, who were bohemians, didn't."

Birch memorably describes the seven-year-old Dury as "Upminster's very own Little Lord Fauntleroy", constantly being taken for rides though the borough in the Rolls-Royce his father drove for a living. Then, on a day trip to Southend-on-Sea, Dury contracted polio while splashing about in the shallow end of the open-air swimming baths. He became so ill that he was not expected to survive. After several weeks in an isolation unit he was taken back by ambulance to Essex, where he spent another two years undergoing often-painful physiotherapy. The virus left him with a paralysed left arm and leg.

In April 1951, just before his ninth birthday, his withered left leg now encased in a calliper, he was sent to Chailey Heritage Craft School, where children with varying degrees of disability were taught, usually in the open air, the benefits of hard work and self-sufficiency. It was a tough environment where the hierarchy among the pupils was established by vicious playground fights. "Once I got strong, I became quite a little tyke," Dury told Birch, adding, "Being in that place is one of the reasons I talk the way I talk. Before that I talked not-quite BBC. A third of the kids were funny in the head as well as being disabled. It was a very tough place, very cold and very brutal. The law of the jungle ruled there."

In 1954 Dury enrolled at the Royal Grammar School in High Wycombe, which seems to have tested his survival skills even more. "He was not popular," a fellow pupil later remarked, "because he was not nice." Dury's disability certainly set him apart at High Wycombe – he was the only disabled boy among 800 pupils – but by then he had also retreated into himself and seems to have adopted a bully-or-be-bullied attitude that, to a degree, stayed with him into adulthood. It was there, though, that he began to read voraciously, the lowbrow pulp novels of Micky Spillane mostly, and discovered a precocious gift for drawing, specialising in sketches of busty pin-ups copied from soft-porn magazines and teddy boys with outrageously extended quiffs. An aesthetic of a kind was being formed, though it would not find its full expression until two decades later.

He left High Wycombe with a bad reputation and three O-levels – art, English language and English literature – as well as a serious Gene Vincent fixation. Now a fully-fledged ted, he worshipped the leather-clad American rocker who had been left severely lame following a car accident.

At Walthamstow College of Art he finally found an educational environment that accepted him for what he was: a bohemian misfit with attitude to burn. There he was taught by Peter Blake, perhaps his most important mentor. "Pop Art, I suppose you would call it," Dury said later of the dynamic there. "Jazz was involved. It was OK to be rude or common in our art, nobody was aiming to be academically clever."

Though Dury never applied himself seriously to being an artist, he did work for a time as an illustrator for the Sunday Times. More important, like John Lennon, he was a product of the post-war thrust towards social inclusion that allowed the brightest of the young working class to attend university and art college. Out of that semi-bohemian academic milieu came many of the singular talents that reshaped British pop culture in the 60s and 70s.

"When I think of the young Ian, I always think of his song 'Reasons to Be Cheerful'," reflects his best friend, the artist Humphrey Ocean. "It sums up where he was at and where he came from. The earnest young Dury – Pelican books, intelligent aunties, the welfare state, grammar school. It's nothing to do with rock'n'roll really, it's all to do with postwar England at a certain, incredibly positive, moment."

All that found its way into Dury's own art – his songs. He created a cast of extraordinary ordinary real-life characters – Billericay Dickie, Clever Trevor, Plaistow Patricia – and painted a world around them that was both instantly recognisable and utterly surreal. He sang about lawless brats from council flats and wrote odes to outsiders of every hue, both the marginalised and the self-styled. He, of course, was both. Consider the song "Blockheads". Its best, and darkest, lines seem to be indebted to Philip Larkin rather than any pop precursor.

You must have seen parties of blockheads

With blotched and lagered skin

Blockheads with food particles in their teeth

What a horrible state they're in

They've got womanly breasts under pale mauve vests

Shoes like dead pig's noses

Cornflake packet jacket, catalogue trousers

A mouth what never closes.

In a way, Dury was the poet laureate of the English working class at a time when even their class status was being demolished by the Thatcherite ideology that insisted there was no such thing as society. He was more than that, though. "I was a working-class kid," says Kosmo Vinyl, "and most people I knew had a few quid in their pocket but Ian was a revelation. He had no money and no regard for money. Him and Denise lived on doughnuts and cigarettes and seemed pretty happy. You'd go round there and he'd play you Gene Vincent and Charlie Mingus. He didn't differentiate. He was a bohemian, basically, and I've never met anyone that free before or since."

Musically, too, he was a bohemian though a hard-working one. "We worked all day every day refining the songs," says Jankel of their initial writing stint. "He had phenomenal energy and dedication. He wrote reams and I edited it down. He always said he wanted to give you a snapshot in song and that's what the best songs did."

Dury's uniqueness as a lyric writer rests in this snapshot aesthetic and the merging of all the traditions that formed him: music hall, art school, rock'n'roll, pulp fiction and the English suburbia he grew up in as much as the semi-mythical East End he hymned. He took these conflicting elements and made of them something else altogether, something new and at times astonishing. "I think New Boots & Panties!! is Ian's version of Pop Art," says his long-time manager, Andrew King. "Basically, it's stories about the kind of people that Peter Blake painted."

Ian Dury's time at the top was brief, his creative peak bracketed by two brilliant singles: "Sex & Drugs & Rock & Roll" (released August 1977) and "Reasons to Be Cheerful Part 3" (July 1979). He had his moments after that, of course – the plaintive rehab anthem "I Want to Be Straight", and the angry, uncompromising "Spasticus Autisticus", which he wrote for the International Year of the Disabled and which the BBC banned for being offensive.

"Even though he had quite a nice time at the top, fame did not sit easily with him", says Humphrey Ocean, "Around the success of New Boots, he seemed strangely unsettled. He had the most awful difficulty adjusting to hotels. He was deliberately leading the life that went with pop stardom but it struck me that what he really needed was a proper gaff. There were always these incredible paradoxes with Ian. Always."

They defined him to the end but no British songwriter since has come close to such economy of expression matched with such linguistic mischievousness. He remains a one-off, a flawed genius, as well as a reminder of how beautiful and alarming a humble pop song can be. Consider these lines from "What A Waste":

I could be a lawyer with stratagems and ruses

I could be a doctor with poultices and bruises

I could be a writer with a growing reputation

I could be the ticket man at Fulham Broadway station…"

Luckily for us, he wasn't.

Sex & Drugs & Rock & Roll is released on 8 January

• This article was amended on Tuesday 1 December 2009 because we included 'What A Waste' in a list of songs Ian Dury wrote with Chaz Jankel. That song was co-written with Rod Melvin.