One morning in 1979, Brian Duffy, then one of the most famous photo-graphers in the world, came into work. One of his assistants told him they had run out of toilet paper. His memory is hazy, he admits, but what happened next became an ­episode of snapper folklore.

"I realised," he recalls in a documentary that airs on BBC4 ­tonight, "that I was making decisions about toilet ­paper. And I thought, 'This has got to end.' Either by me murdering my staff, killing myself, or setting fire to the whole fucking thing." So he gathered every negative and transparency he had ever shot and burned them on a fire in his back garden. After that, he never took another picture.

Except, as it turns out, negatives do not easily catch fire. And when they do, they produce an acrid black smoke: this bonfire ended when an official from Camden council peered over the fence and insisted Duffy put it out. Duffy packed what remained away in shoeboxes in his attic and turned to painting and furniture-restoring. It was only in 2007, when his son Chris went through the boxes, that he reluctantly agreed that they were worth another look. This led to a show in London last year – the first, anywhere, of his career.

To devotees of photography, these surviving pictures were like a salvaged stack from the library at Alexandria. Lost portraits of Michael Caine, John Lennon, Nina Simone, William Burroughs, ­Reggie Kray and many more. And ­fashion shots that remind us how this man (alongside the other "cockney photographers" David Bailey and ­Terence Donovan) created the visual spirit of the swinging 60s: making ­fashion fun, colloquial, young.

The son of an IRA man who did time for murder, Duffy grew up "a thug", in his words, running wild on the bombsites of 1940s London, until he was enrolled in an ­experimental school that believed in rescuing delinquents through art. It worked, though it did little for his manners: once, when a model smoked cigarettes carelessly in Duffy's studio, he tipped the ashtray into her handbag. "Difficult" is how Bailey describes his great pal: "Duffy and aggravation go ­together like gin and tonic."

But when we meet at BBC Television Centre in London, he couldn't be sweeter, smiling and extending an ­enthusiastic hand. He is 76 now, suffering from lung disease and accompanied by his wife, a friend and a BBC publicist. But he's full of anecdotes, which he ­delivers, when his breath allows, with a hand clasped on my arm. "How long have we got?" I ask. "What, before I pass away?" he grins. "What are you suggesting – that we go drinking, ­picking up birds or what?" His bright eyes dare me to take him up on it.

Perhaps later, I mumble, but first let's talk about the pictures. Who did he like photographing best? "I quite enjoyed Terry Stamp," he says. "He was so self-interested, which is fascinating, and terrific to look at. The camera loved him." It is clear, from the enthusiasm in his voice, that he has not lost all his fondness for his old career.

Duffy was a young fashion designer in 1955, but turned to photography ­because it was "a darn sight easier than drawing". He says: "I wanted to make women look good. That really intrigued me. To make a model look as though she owned the clothes." It was a ­philosophy that took him to British Vogue and then French Elle, before he started his own studio – a hub for the era's ­in-crowd, including Peter O'Toole, Michael Caine and the Beatles. "It was terrific. There were very few people to know, so you could know everybody."

Duffy shot Pirelli calendars, prime ministers, and, in the 1970s, the ­revolutionary (and surreal) Benson & Hedges campaign, as well as three ­album covers for David Bowie. He even produced a couple of films. With all this going on, why did he stick with photography? "I loved it," he says. "But it was only enjoyable because it was mysterious. The revelation [about why a picture worked] came after – when I'd go, 'Christ, that's interesting!'"

So when did it cease to be interesting? Duffy offers some clues in the BBC's documentary. Talking with Bailey, he says he feels the US photographers ­Irving Penn and Richard Avedon "fucked photography for us". What does he mean? "They got there," he says, ­referring to their revolutionary work, which pushed at the boundaries of photography. "You're a bit annoyed when someone does something and you go, 'Shit! I was just about to do that!'"

The result, says Duffy, was that "photography was dead by 1972". He takes a rare pause, then explains: ­"Everything had been resolved between 1839 and 1972. Every picture after 72, I have seen pre-72. Nothing new. But it took me some time to detect its death. The first person who twigged was Henri Cartier-Bresson. He just stopped – and started painting and drawing. God, he was useless."

Duffy came to photography through art school, and he remains, at his core, a theoriser and a provocateur. Certainly, he is a man for whom swearing is not so much a habit as a hobby – and he clearly loves a good wind-up. "I'm not very keen on ­Yorkshiremen," he says at one point, adding: "I used to love watching ­[racing] drivers being killed." Yet what also comes across is his playfulness and good humour: he enjoys experimenting with an audience, just as he did as a photographer – trying out techniques, poses, situations, everything. In what he calls the "insecure-making" world of photography, he says, you have to prove yourself anew with every picture, because anybody can use a ­camera. So Duffy experimented, until he felt the scope for experimentation had ran out. By the 1970s, he was doing most of his work in advertising – with people he didn't like, on briefs that bored him. "The more I got into it, the more I ­realised I was hanging out with things I was diametrically opposed to. And they wanted me to keep a civil tongue up their rectum."

So he burned everything. Is it true he actually beat people up as well? "Yeah, I think probably I hit a few people," he says, without pride. Who? "Oh, all sorts of people." Assistants? "No way!" ­Models? "No!" People he was photographing for? "Yeah. People who . . . "

"Erm," says the woman from the BBC. "I'm not sure this is the kind of thing . . . " Duffy frowns delightedly. "You look very worried," he says.

The Man Who Shot the Sixties is on BBC4 tonight at 9pm.