Pete Townshend arrives in his Richmond office clutching a portion of chips. As befits a former figurehead of the British mod movement, he looks dapper, that late lunch notwithstanding: suit, pocket handkerchief, suede loafers. He also looks knackered, having been on a promotional blitzkrieg for his new autobiography, Who I Am, which he has found somewhat gruelling. He's barely done any interviews in the last 20 years, after decades as the Who's de facto spokesman. "No one else really spoke but me," he explains. "I just talked for 28 years."

Who I Am is occasionally a rather harrowing read, particularly when it comes to his chaotic childhood. Townshend's parents had an unstable marriage, and he was farmed out to his maternal grandmother, who seems to have been mentally ill. Moreover, one of her lovers sexually abused him. Nor does it stint on Townshend's own failings: his alcoholism and cocaine addiction; his infidelities; the collapse of his first marriage. One radio interviewer suggested the Who's guitarist and chief songwriter had performed "a hatchet job on himself".

"When I first started writing it in the mid-90s," says Townshend, "the publishers said they wanted a book about sex and drugs and rock'n'roll, and I said I'm not going to write that book. I think I have now. I've written about as much sex and drugs and rock'n'roll as I'm entitled to, and a lot of people have said to me it's quite sad, really – it's almost like the bits of sex and drugs and rock'n'roll that you pulled off, you felt ashamed about." He chuckles.

Of course, there's more to the book than that. The day before we meet, I'm sent an email via Townshend's publicist that suggests he is weary of discussing the concluding chapters, which deal with his 2003 arrest on child pornography charges and his subsequent decision to accept a caution rather than fight the charges in court. "This is not a subject I can talk about without being triggered," he says. "I get angry. This is too important to chat about on a TV show for a few minutes. Please, if you are really interested to hear my side, read the book. That's where the facts are … Or if you can't afford my book, just take a position based on your heart. I have to accept that."

The book certainly covers Townshend's side of the story in meticulous detail: how the Who had set up a charity in the 70s called Double O, dealing with victims of domestic violence and sexual abuse; how he subsequently became obsessed with the idea that child abuse and pornography involved "a financial chain that ran unimpeded from the Russian mob to our high-street banks"; how he used his credit card on a site he was certain had been set up by the police as a sting (he didn't think it was a genuine child porn site); how he did this to prove that banks were happy to accept the business of websites offering child porn; how he immediately cancelled the payment; how the police who confiscated his computers found nothing on them; how, given half an hour to make a decision whether to accept a caution, he agreed because he didn't feel emotionally capable of going to court.

Today, he seems if not exactly happy, then at least willing to discuss it. The only time he stonewalls is when I mention the allegations against Jimmy Savile: "I don't want to comment on him because I know too much. I know him from the past, and I don't want to comment on him because he's dead." He says the Who's singer Roger Daltrey – who stuck by him throughout, as did his partner Rachel Fuller and his three children – told him not to write the book, worried it would bring it all up again. So why did he? "I had to nail it down – and say, 'Come on, please look at this more closely. Look at the train of events.' If what you're going to say is, 'You accepted a caution and went on the sex offenders list, therefore you must have done it' – I know what I did. I used a credit card. It's whether or not I used a credit card for the reason that most men who logged on to a fucking child-porn site would've used a credit card for. My reasons were very, very different."

In the aftermath, he became used to public opprobrium. "Everywhere I went to eat out, sometimes people would put down their plates and walk out of the restaurant. It was tough, and I do feel I'm on the other side of it now. I can get a bit self-righteous now: 'There you are, you owe me an apology.' But that's not appropriate." He recently received a letter from "the guy who ran the case, saying 'none of us ever believed that you were guilty of anything'. He's offered that I can use the letter, but I don't think I will. My sense is that you can read the book, you'll be able to make a decision. And if you don't read the book, you can do what normal assholes do, which is to look at my big nose and think, 'He looks like a paedo.'"

He seems gloomy while he says all this, but then Townshend always looked a bit like that, as evidenced by just about any photo of the Who in their mid-60s mod pomp, when Townshend had just singlehandedly invented the notion of art-rock, applying to pop what he'd learnt about Peter Blake and auto-destructive art pioneer Gustav Metzger at Ealing art college in London. The band were in the midst of a remarkable run of singles: the ferocity of their sound rubbed up against lyrics that shoved pop into areas so new that not even the man who wrote them seems entirely aware of what he was doing. "I remember two boys, on different occasions, came up to me and said about I'm a Boy, 'That song really helped me to get to grips with…' And I would say, 'What do you mean?' And they mentioned a word I'd never heard before: 'I'm a transgender individual.' And we're talking about 1966. I had no idea what they were talking about."

A couple of years before, they'd been playing covers in pubs. Now they were at the vanguard of what may have been rock's greatest period of innovation and excitement – and yet, in every photo, Townshend had the baleful expression of a cat that has fallen into a bath. As Who I Am makes clear, there may have been a good reason for this: he doesn't seem to have enjoyed the 60s very much.

There he was, hanging out with Jimi Hendrix and Mick Jagger, taking Eric Clapton to see Syd Barrett's Pink Floyd (alas on a night their errant frontman decided to play the same note over and over again for hours), and delivering a career-defining performance at Woodstock in 1969 despite being spiked with LSD. Yet Townshend appears to have spent the entire decade crippled by self-doubt, mired in business problems, increasingly bored with the Who's destructive stage act, and attempting to deal with living in close proximity to the band's drummer Keith Moon, who seems to have alternately amused Townshend and driven him up the wall. He concludes his retelling of the Who's infamous appearance on America's Smothers Brothers TV show (during which Moon "let off an over-sized theatrical charge of gunpowder" that permanently damaged the guitarist's hearing and set his hair on fire) with the plaintive observation: "Keith could be such a twat sometimes."

It's all the more surprising when you see the 1960s iconography dotted around his office: a gleaming Vespa; a Rickenbacker guitar (smashed); another Rickenbacker guitar (intact); an ancient keyboard; a painting of the late Small Faces guitarist and sometime Townshend collaborator Ronnie Lane; a mirror decorated with targets, union flags and photos of mods. His secretary serves me tea in a mug that looks like something you might pick up in a tourist shop, featuring a cartoon of a scooter and some 1960s buzzwords: Twiggy, Carnaby Street.

Although he admits it wasn't all gloom, Townshend insists he was unhappy a lot (he wonders if he might be a manic depressive). Part of the issue was sheer over-work. "I would be in my home studio with my guitar and my tape machine, possibly with a baby crying in the distance, possibly with my beautiful wife lying in bed wondering where I was, thinking, 'We've got an album to make in three weeks – and between now and then, I've got to write the songs and we've got eight gigs.'" Then there was the way the band members' personalities chafed against each other in what Townshend describes as "an unruly gang". Quite aside from Moon, there was Daltrey: "I wouldn't say Roger has ever been fun to be around. I don't mean that disparagingly – I just don't think he's a fun guy. He's very serious, very committed, very strong. I needed that."

Bassist John Entwistle, meanwhile, seems to have been both deeply odd and completely unknowable. At his funeral in 2002, Townshend was startled to discover the man he'd been in a band with for 40 years was a lifelong freemason. "Perhaps that's why we did so well," laughs Townshend. "He was a very strange fellow. I loved John, obviously, loved him for his eccentricities. He had an extraordinary way of eating. Wouldn't touch anything on his plate until it was all cut up. Then he would eat it not quickly, but methodically, precisely. I think he was a little bit Asperger's."

There was something else: the sense that Townshend didn't feel he belonged in the Who at all. "To be completely honest, I think if I hadn't been bullied into the band, I would have been happier as an art student. I would have been happier in a Brian Eno world. That was the stuff I was trained to do – to think differently, to think outside the box, about installations, the coming of computers, the way that computers would change the language, blah blah fucking blah. I've been talking about it all my life as a rock star and people have been going, 'For fuck's sake – shut up and play your guitar.'"

At the height of their 60s fame, he wrote a manifesto, now lost, suggesting the Who should split up. "What I wrote was: 'The Who must destroy themselves.' I tried to find it years later, because I wanted to fucking wave it at [Sex Pistols manager] Malcolm McLaren. I think it was my thesis. I thought, 'I'll go and do the band and then I'll come back to art school and be an artist.' I remember being in Sweden, in the back of this little car, smoking a bit of grass. We didn't even have our own instruments with us. I just thought, 'This is fucking ridiculous, but I'll be back at Ealing soon.'"

The problem was, he smiles, he was good at it. "I was quite brilliant on stage, and I found it incredibly easy. I could see Keith Moon found it incredibly easy as well. And I couldn't work out why I wasn't fulfilled by it. So there were periods when I was sarcastic. I look back and think, 'Didn't I realise what I had?' I had this fantastic band and I don't think I realised what I had in the guys until we made Quadrophenia in 1973. I thought, 'I'm going to write this thing and it's going to be really challenging and there's no fucking hope they're ever going to do it properly.' And they did – they did it brilliantly and quickly."

By then, they'd made 1969's Tommy and 1971's Who's Next, the albums widely considered their masterpieces; but Townshend was haunted by the deaths of friends such as Hendrix and Rolling Stones guitarist Brian Jones, and concerned that artists had become remote from their fans. At one juncture, he took to wearing a boilersuit on stage in an expression of working-class solidarity: "We're people like you – we don't dress up like Christmas trees." He sighs. "But I was the only one in the band that did it. One day, I decided to wear a gold lamé boilersuit with a crown. I came off stage and said to Mick Jagger, 'How do I look?' He said, 'If you want to know the truth, Pete, you look like a bit of a cunt.'"

At another point, he seems to have invented the notion of open journalism 30 years before it was to become so prevalent: he wrote a column in the music press, which he hoped would lead to an exchange of ideas between himself and his fans that he could use to spur his songwriting. "But the editor wouldn't print their comments," he says. "We did get letters back from people. They were mainly, 'What the fuck are you talking about?'"

Townshend seems more comfortable with his role today as an elder statesman of rock. He and Daltrey are about to embark on a tour of north America, performing Quadrophenia in its entirety. They last made a new album in 2006; he isn't sure they'll ever make another. "I've felt too old to be doing it since I lost my hair and people started to say, 'Roger Daltrey looks in good shape, but that guy Townshend looks like a bank manager', or more recently 'a vicar', or more recently still 'a paedophile vicar'. And there has been a sense, too, that I can do what I do simply because I'm healthy and I've survived. I can jump about and ape the old stuff. But for me, the job was to write the songs, to do the creative work – and I still feel if there's a brand there … "

His voice tails off. "If it's possible to make a record that echoes what we've done in the past, even if it's 'Let's do something audacious and mad that won't work', then that's OK, too. I would never have wanted Philip Roth or John Updike or Gore Vidal to suddenly go, 'Oh, I'm too old now to write books about masturbation.'" He frowns, as if considering the idea. "Funnily enough, with Roth and Updike, they wrote more books about masturbation as they got older."