A string bean of a man in a khaki parka, smart jeans, stripy socks and flared sideburns walks past. "Bradley," I say, gently. He jumps.

"Bradley, I'm Simon, your interviewer."

"Oh," he says, unnerved. "The thing is, I've just been round the corner for coffee and was stopped three times for photos." He says it wasn't like this a few months ago, before he became the first Briton to win the Tour de France, and he doesn't quite know how to cope.

Has victory sunk in? "No. I don't think it will, to be honest. I don't think it has to sink in. I accept that I won the Tour, but I don't know if I should feel any different. I understand what it means, because I love the history of the sport, but I don't think I'll ever…" He thinks through every sentence so thoroughly that he ties himself in knots – doubling back on ideas, qualifying every thought, adding last-second caveats. He's got a cold, and sounds like a man coming down from a high without having fully experienced it in the first place.

The problem is, he says, there wasn't time. After the Tour, it was the Olympics. So what about now? Does it hit him; does he turn to his wife and kids, and say, "You know what? I've won the Tour." He smiles. "No. My wife says it to me occasionally. She'll say, 'Fuck! You won the Tour, Brad!' like it dawns on her now and again."

Does he wake up smiling these days? "Sometimes I feel… I wouldn't say I wish I hadn't won the Tour, but sometimes, especially with recent events, the Lance Armstrong stuff, I find it hard being the winner of the Tour and everything that goes with it. I wanted to be the winner for the challenge of what the sporting event is about and how hard you can train to do that, and I never wanted all the stuff that went with it." In short, no, he doesn't wake up smiling.

It's no wonder, really. Just as he should be bathing in the glory of his Tour win, and Olympic gold, he is forced to defend his sport's tattered reputation. And it gets worse. "Fans" are telling him he's changed – that he's got up himself; tweeters are suggesting that he couldn't race as he has done without taking drugs. A few days after we meet, a newspaper reports that he has joined a disreputable tax avoidance scheme, and that he's in danger of becoming the new Jimmy Carr. Welcome to the world of celebrity, where we build up our heroes only to knock them down.

And yet we couldn't have handpicked a better British hero – Wiggins has struggled plenty (he's been racing professionally for 11 years, often couldn't make ends meet, and has had a drink problem), he is funny (he does great impressions), cool (a retro mod), a bit of an extrovert (he jumped on a car after winning the Tour) but more of an introvert (he did it only because he didn't know how he was expected to react). The French nicknamed him Le Gentleman when he slowed to allow rivals to repair their bikes and catch up after saboteurs had thrown tacks on the ground. Fellow cyclist Chris Hoy described his victory as the greatest achievement by a British athlete.

In an annus mirabilis for British sport, nobody has been embraced quite like Bradley Wiggins: despite the achievements of Rory McIlroy and Andy Murray, Jessica Ennis and Mo Farah, David Weir and Jonnie Peacock, he is odds on to win Sports Personality of the Year. The Tour is regarded as sport's most demanding event; to win it requires superhuman discipline, strength, endurance, speed and, all too often, illegal drugs. Wiggins had all of these in spades – except the drugs.

We are sitting on the roof terrace of a London hotel and his agent is flicking through photographs for his new autobiography, aptly titled My Time and ghosted by the Guardian's cycling correspondent, William Fotheringham. Is he pleased with the book? "It's just really odd," he says in a monotone, his accent an unlikely mix of London and north-west that meets somewhere in the Midlands. "I said to William, I hope there's a fucking story there because it's me just doing my everyday life. What happened in the Tour… I suppose some people might find it interesting."

However much he protests, his is a remarkable story. Wiggins was 12 when he started racing at the Herne Hill velodrome in London – the same track on which his father had raced years before – and says he knew from then on this would be his future. He loved everything about it – the freedom of track and road, the sweet agony of training to peak fitness, the natural high, the rivalries, and the battle of man versus machine.

He was brought up by his mother, Linda, after his Australian father, Garry, walked out when he was two years old. Actually, it was worse than that. They were living in Belgium where Garry Wiggins was making a living as a professional cyclist. When Bradley and Linda were on a visit to her parents in London, Garry wrote a letter telling them not to bother returning home because he had a new girlfriend. He took the ferry to London with four black bin bags full of their possessions, dumped them outside Bradley's grandparents' house in Kilburn, and erased them from his life. Linda and Bradley moved into her parents' flat, then into a one-bedroom church commission flat. When Bradley was seven, Linda had a second son with her new partner, and the family moved into their own home.

Wiggins on his first bike: ‘At 12, I told my art teacher, I’m going to be Olympic champion, I’m going to wear the yellow jersey in the Tour’

Bradley didn't know any of this when he started cycling. Nor did he know that his father was an alcoholic, street fighter, wife beater, drug cheat and dealer who had smuggled drugs into Belgium in his son's nappy, served time in jail and hadn't contributed a penny to his upbringing. All he did know was that his father had been a professional cyclist, and his mother suggested he take up the sport because he was likely to be pretty good, too.

Was it a way of getting in touch with his father? "No," he says instantly. I tell him I don't mean literally. "You don't really think about things like that when you're 12, but maybe it was, looking back. I only knew what he was like from reading the odd article about him. They described him as a cyclist, but I knew nothing of the person."

Did Linda tell him about the things he'd done?

"No. Not at that age."

When did he find out?

"I suppose as I got into my late teens. I still don't know the half of it. As I've got older, I've become aware of what he was like. Shagging about when me and my mum were at home." He nurses his coffee, still and self-possessed. There is something understated, almost dispassionate, in the way he talks about his father – as if he dealt with the extreme emotions way back. He pauses. "Because of what he did to us, I'm, sort of, the way I am with my own family."

I ask him what he means. "I cherish them. It was a lesson of how not to be with my own children. So it's not 'like father like son'. It's the complete opposite… Not having my father around has made me a better person."

At 18, he went to meet his father in Australia and was horrified by the man he encountered – a raging, pathetic, self-absorbed monster. "He wasn't a nice person," he says quietly. The funny thing is, Wiggins says, he probably wouldn't have become a cyclist without his father, but he definitely wouldn't have if his father had stayed with the family. "If he was around throughout my childhood, having been a cyclist himself, I would never have taken up cycling. He'd have been incredibly pushy and critical of everything I was doing, because he was when I met him when I was 18." In 2008, Garry Wiggins was beaten to death in Australia. Nobody has been charged. Bradley decided not to go to the funeral. Why reopen wounds that were beginning to heal?

Although the new book is ostensibly about the triumphant Tour, family is at its heart. Real family, and surrogates from the world of cycling. I ask Wiggins if he was aware of the number of times he talks about father and brother substitutes when writing it. "No," he says. "No, I wasn't. This thing for father figures is a result of the sport I'm in and how close you get to people."

His first and most important father substitute was his grandad, George, whom he adored. Every Saturday, young Bradley would collect George's Racing Post for him and a handful of betting slips from Ladbrokes, and in the evening George would take him greyhound racing at Wimbledon dog track. "All my mannerisms I learned from him. He became my male role model in a family that was predominantly women. The grandparents had three daughters, so there were a lot of aunties."

Meanwhile in his professional life, coach Shane Sutton took on a paternal role. The world of cycling is famously macho, and perhaps no one exemplifies this more than Sutton, another tough Australian. In My Time, Wiggins says that when he was devastated by his grandfather's death in 2010, Sutton's response was to tell him to get a grip. "Shane is quite a hard character," he says. "He's merciless at times in terms of shrugging off events that other people would consider serious." How long was this after his grandfather died? "A couple of days." He segues into an Australian accent. "Man up, mate, you've got to get on with things."

Has he given Sutton lessons in emotional intelligence? He grins. "No, no… I'm emotionally retarded. That's what my wife describes me as." Why? "Because I struggle expressing emotion." While many Olympic gold medal-winners burst into tears, Wiggins simply sat on his throne, made a V for victory sign with both hands, and turned it into a big joke. He says he's grown a little more emotional in recent years, and has even been known to cry since having children. Then he quickly corrects himself. "Not in public, obviously. A lot of that is feeling that everyone is looking at you. It was like that after the Tour. I felt the need to do something, to perform. I don't know if you saw the shot when I climbed on top of the car in Paris? Everybody was looking at me, and I felt the need to do something, so I jumped on the car."

Is he shy by nature? "Yeah, I am shy and introverted. If I've got to go on stage and talk, I end up drinking to stem that… then my character comes out, and I relax a bit and perform better."

Wiggins in the Pyrenees wearing the Tour leader's yellow jersey. Photograph: Getty

Despite the diffidence, he was sure he'd become a successful cyclist from the moment he first pedalled a racing bike. "That was it. I had no doubt, I'm going to make millions from cycling, so I don't need to concentrate at school. I remember telling my art teacher when I was 12. She said, 'What are you going to do when you leave school, Bradley?' I said, 'I'm going to be Olympic champion, I'm going to wear the yellow jersey in the Tour.' She laughed and said, 'What are you really going to do?'"

Sure enough, by the age of 20 he had won bronze at the Sydney Olympics in the team pursuit. Four years later, in Athens, he became the first British athlete in 40 years to win three Olympic medals, including his first gold. He returned home and found nothing had changed. "It got me down. You think if you win the Olympics, you'll become a millionaire overnight. But I was still scraping the barrel, looking down the back of the settee for pound coins to buy a pint of milk."

Cath was pregnant with their first child and working as a radiographer. He had too much time on his hands. So he drank. And drank and drank and drank. Wiggins has always had an obsessive personality – the psychiatrist at Team Sky, Steve Peters, told him that one of his personality traits is a passion for discovering something new and feeling the need to become a world expert on it. So when he started collecting boxing memorabilia he had to have the lot, and when he went on to guitars he bought any number of Gibsons and Fenders. And when he found himself fascinated by Belgian beer, he was determined to collect all 365 different varieties (supposedly one for every day of the year, but actually, he says, when you count the different varieties of the same brand, it's closer to 800). And this was probably the start of his drink problem. He regularly set off for Belgium and returned with a van-load of beers. He would sit at home, admiring his collection, and then he started drinking his way through it. "I probably get my addictive streak from my father," he has said previously. Before long he'd be at his local for opening time every day. Often he'd get through 12 pints in a day. "I was just bored shitless and didn't know what to do." He says many athletes experience depression after great victories – it's the anticlimax. The bender lasted nine months.

Wiggins is six foot three and weighed in at a skinny 69kg for this year's Tour. Did he get a belly when he was drinking heavily? "I got quite big. I wasn't huge. I was probably 83 kilos." Could he drink 12 pints now? "No, I don't drink beer now." How did he get sober? "We had a baby. So then it was a case of, well, I've got to earn some fucking money and the responsibility takes over."

In 2008 he won two more golds at the Beijing Olympics, and this summer he won his fourth, becoming the first British athlete to win seven Olympic medals. As he talks, I stare at his thumbs and the two elaborately calligraphed Bs on them. Great tattoos, I say, thinking that's a bit solipsistic.

"They're my kids. Ben and Bella. So when I'm time trialling, I'm like that." He stares at his thumbs. "It's a reminder of why I'm doing it. When I'm on the start line of an Olympic final, I look at them and think, you know what? This ain't life or death. So all I can do is try and do my best. Nobody's going to shoot me if I don't win. And at that moment I go, 'Let's do it'."

Perhaps it is guilt that most fuels Wiggins – about all the time he spends away from the family in training, about missing the kids' birthday parties, that Cath has to carry the suitcases when they travel lest he injure himself, that his team-mates at Sky have to sacrifice their personal ambition for his success. You sense he feels beholden to so many people that he thinks the least he can do is win his races.

Cycling is an extraordinary sport, with its polarities of selfishness and selflessness. His triumph in the Tour was very much a team effort, but the victory is solely his. Eight team members ride alongside him, the leader. They are worker bees, there purely to service his needs. So they will chase another rider who tries to break away, provide Wiggins with enough food and drink, and shield him from the wind. Not surprisingly, this causes friction at times. At one point in this year's Tour, Wiggins's team-mate Chris Froome, who finished runner-up, looked as if he was racing for himself rather than his leader, and it led to a Twitter spat between Cath and Froome's girlfriend. In the book, Wiggins criticises Froome's knowledge of the sport and his tactical naivety. He also makes a point of saying that he suggested to Sutton in 2011 at the Vuelta a España, when it became obvious Froome was stronger than Wiggins, that "we should concentrate on Froomie now… The answer was, 'No, no, no. We don't have faith in him.'"

His relationship with Mark Cavendish has also suffered. In Beijing, the pair were favourites for the madison team event, but finished ninth. Cavendish, the world's top sprinter, felt Wiggins couldn't give his best because he was too exhausted from previous events. The pair didn't talk for months. Cavendish recently announced he would be leaving Team Sky to join the Omega Pharma-Quick Step team.

Wiggins says his departure was inevitable. "I think he needed to go. I'm sad he's going, but he needs a dedicated team round him, as I had this year at the Tour." Wiggins is proud that he managed to lead Cavendish out on the final day in Paris, setting the pace for his fourth successive victory in the Champs Elysées sprint (most leaders would not take such a risk, preferring to be cocooned by their team at the final stage). But he still feels this wasn't enough. "We'd normally do what we did for him in Paris every day, but we couldn't some days because we were going for the yellow jersey." In other words, Cavendish had to be sacrificed to protect Wiggins' lead.

Did that create tension? "No, no, no, not at all. He was totally supportive of what we were trying to do for me. At the same time he deserved better than that. But we simply couldn't feed two mouths. He'll always be like a little brother to me, so I'm happy to see him shine again."

Wiggins celebrates winning gold at the London Olympics. Photograph: Getty Images

Wiggins's greatest regret is reserved for his family. Ask him if he enjoyed the Tour and he'll tell you about the months away training, the monastic lifestyle in the mountains, the 7,000 calories he consumes a day with a diet of pasta, omelettes and croissants, oat and honey energy bars, the 8,000 calories he burns daily, the self-imposed chocolate and alcohol ban, the five to seven hours on a bike every day and the 100,000 metres he'll climb in the lead-up to competition. He usually misses his children's birthdays because he is away training and says when his family come out to see him he feels as if he is receiving a prison visit. Yes, he enjoyed the victory, but that's different from enjoying the race.

Now there's a new source of guilt. Even last year, he says, he could go out largely unbothered, but not any more. "They ask your wife to take the photo, which is a bit rude. And after a while that becomes tiresome, especially when you're having a pizza with your children, or you have to have a photo with somebody else's kids while yours stand to the side." Don't get him wrong, the public's reaction has been astonishing, but that doesn't make it easier to cope with. "It's quite humbling that sport can do this to a nation, but there comes a point when I've got to start getting on with my life. It would be hard to live my life as it is for ever."

In his early 20s he lived in France, and he speaks the language fluently. Does he think of leaving Britain again? "There were a few times we considered it. Where d'you go, though? I like living here. I wouldn't go to Monaco or anywhere like that. It's a shit hole. I couldn't think of a worse place to live."

How does he think he will regain normality? "I have no idea." At least, he says, he lives in a village, just outside Wigan, where people aren't suckers for fame. "It's a humble town. Working-class people don't tend to be wooed by celebrity." He pauses. "I hate that word," he mutters. This lugubriousness is quite at odds with the insouciant Wiggins we normally see in public.

One of the things he finds toughest to deal with is the way people treat Cath – as if she doesn't exist. "Nobody ever asks her how she is. It's always, 'How's Brad doing?' Nobody ever says to her, 'How are you doing, Cath? How are you handling it all?' It's very difficult for her."

They met when they were 15 and aspiring cyclists, and got together at 21. Does she still compete? "Yeah, she's British champion for her age group. She's 31." I didn't have a clue, I say. "Exactly. No one knows, that's what I mean. So she won that while the Tour was on. And we go to cycle events and it's all about me, me, me."

Does Cath make money from racing? "There isn't much money in women's racing." Does she have to go away and train? "Yeah, but not to the level I do. She does what she can around staying at home and looking after the kids. And she runs me. She does everything. She's sacrificed herself for my career." Does he feel he owes her? "Yes, as soon as I retire, it's her turn. She wants to go back to university and do a master's. She might go back and do physiotherapy, she's quite academic."

Their son Ben is seven and a promising cyclist. "I'm not just saying this because he's my son, but he is fucking good. I think he'll be better than me. But I'm not pushing him at the moment." Even here there is scope for concern. "I feel sorry for him that he's my son, because he'll never get a fair shot; he'll always be compared to me, which is sad really. He did a few races last year and everyone was like, 'Oh, he's got his dad's genes.' He can't just be one of the other kids." Does that bother Ben? "No, he doesn't care, he's just proud. It's hard for my wife, that." In what way? "Well, she's still a cyclist, and she's his parent as well."

Of course, there are plenty of positives that come with success. Not least the money. This year he is estimated to have made £5m. I ask what bonus he got for winning the Tour. His agent, who is sitting at a nearby table, intervenes. "It's confidential," he says. "We'd literally get our knees broken if we say."

"I feel like I'm getting payback for my whole career," Wiggins says. "Now, you get on to footballers' wages… Everybody goes, 'Footballers get paid too much' and maybe they do, but it's nice for a sport that's as hard as cycling. It makes the sacrifice worth it for your family."

In light of the Armstrong affair, it feels as if everything Wiggins says about cycling and its rewards is tainted by the shame the American has visited on the sport. Wiggins is irate that his victory has been tainted by Armstrong's cheating. Had Armstrong been a hero of his? "No, not really a hero," he says flatly. "He was someone I respected and admired. I've met quite a few sportsmen, but I don't think I've met anybody as… powerful as him." How? "In his persona. He's quite an intimidating person to be around. He has an aura. He doesn't live his life like anybody else in our sport. I sit here now as the winner of the Tour and I don't live my life how he lived his life when he was the winner." How did he live his life? "Fucking entourage around him, chauffeur-driven car outside."

The more he talks about Armstrong, the more apparent his contempt. "If I'm going to Kilburn, I get on a bus. He'd have a car waiting for him with a bodyguard. He'd go to races on a private jet. I take my kids to school. It's what keeps you normal. I don't want my kids growing up as fucking idiots, d'you know what I mean? We don't have a nanny or an au pair. We could afford one, but we don't. Cath stayed at home because we want to bring our own children up and not have them brought up by someone else."

A week after we meet, Armstrong is stripped of his seven Tour titles. The American is by no means the sport's only drug cheat. Fourteen of the past 17 Tours have been won by riders later penalised for doping. How did Wiggins feel when the Armstrong report came out? "It wasn't a surprise." It was confirmation? "Yeah. The anger is more: I've got to pick up the pieces. He's still a multimillionaire, and he's not here to answer the questions. I can't not answer them because I've got to go and race next year, and I hate talking about it."

Will it have a significant impact on Team Sky? "Yes, a massive impact." Since the team re-stated its zero-tolerance approach to doping, two people have already left, having admitted taking performance-enhancing drugs in their pre-Sky careers. "The sport is going to have to change… The level and organisation of it is shocking. It wasn't one or two people going off and doing it, it was an organised ring."

Wiggins, who has always prided himself on racing clean, has found himself under scrutiny since winning the Tour. During the race, tweeters suggested he was taking drugs. He answered in the only way he knew: "They're just fucking wankers. I cannot be doing with people like that."

And now, the media is digging dirt on its golden boy. Last week he was attacked for investing money in a Cayman Islands tax-avoidance scheme that the Treasury has condemned. After the interview, he emails to explain the situation. "I had a small investment in Twofold, following guidance from my professional advisers. I had, however, claimed no tax relief of any amount in regard to this investment. Given the concerns raised, I have now instructed my advisers to withdraw me from the scheme with immediate effect."

For the first time in his life, he is facing hostility. Last night, he tells me, he was verbally abused by an autograph hunter at a charity bash he held. He recognised him as a professional who collected autographs to sell, and refused to sign his book. "Today, this guy put something on Twitter saying, 'Brad you're a right cunt, you've gone up your own arse, you're too big for your boots.'"

Then there is the pressure to be the poster boy for honest cycling. He feels he's done his bit by racing without drugs. "I've got other things to do with my life. I'm not going to become a bloody soapbox preacher for clean sport. I've done my bit by winning the Tour, and I shouldn't have to justify that for the rest of my life."

Wiggins still loves cycling, and has signed on for another couple of years with Sky, but he is looking forward to retirement. Last week it was announced that he might not lead next year's Tour, which will be more mountainous and probably better suited to Froome. Actually, Wiggins says, he's not sure the desire is there to do another Tour. After all, his ambition was to win it once, and he's done that. Soon it will be time to set himself new targets. He has always fancied participating in the 2016 Olympics in a fresh sport. He knows it's extremely unlikely, but you've got to dream. Running, he says with a smile. Or rowing. "Imagine that," he writes in My Time, "going and winning the coxless lightweight four: Olympic gold in rowing four years off."

Since winning the Tour, people have been speculating whether he will be offered a knighthood – and if so, will he accept it. Wiggins says he's been thinking about it, too. His instinct was to politely turn it down, but he asked his grandma what George would have said if he'd rejected it and she told him he would never have spoken to him again. "Kids from Kilburn aren't supposed to win the Tour and they're not supposed to become knights, either. Know what I mean?" So that's that settled – he'll be accepting, with a knowing wink. "Yeah. I wouldn't use the title. If I did, I'd use it in a comedy sense. Not in any kind of seriousness or grandeur. Not Sir Bradley Wiggins on your credit card or whatever. Just Sir Wiggo." He repeats it to himself, and smiles. Yes, Sir Wiggo, that sounds just fine.

My Time is published by Yellow Jersey Press on 8 November. To order a copy for £14 with free UK mainland p&p, go to guardian.co.uk/bookshop or call 0330 333 6846. Read exclusive extracts in the Guardian this week. Bradley Wiggins – A Year In Yellow is on Sky Atlantic HD on Wednesday 21 November, at 10pm. Bradley Wiggins wears his Fred Perry autumn/winter 2012 collection.