There have been a couple of questions asked about doping this week and I don't feel I've been able to give a full answer. I understand why I get asked those questions given the recent history of the sport, but it still annoys me. It's hard to know what to say, half an hour after finishing one of the hardest races you've ridden, when you're knackered. The insinuations make me angry, because I thought people would look back into my history, the things I've said in the past, such as at the start of the 2006 Tour when I turned up for a first go at the race and Operación Puerto kicked off, what I said when Floyd Landis went positive, and what I said when I was chucked out with Cofidis after Cristian Moreni tested positive in 2007.

On the way home after that, I put my Cofidis kit in a dustbin at Pau airport because I didn't want to be seen in it, and swore I would never race in it again, because I was so sick at what had happened. Those things I said then stand true today. Nothing has changed. I still feel those emotions and I stand by those statements now.

To understand me, I think people need to look at the bigger picture, where I have come from, in the context of how the sport has changed, and how I've progressed. They see me put in a great time trial like I did on Monday: I can do it because I've worked hard to close the gap between me and Fabian Cancellara and Tony Martin. What seems to be forgotten is that the margin between me and the best guys wasn't that large in the past, even when I wasn't putting in anything like the effort I have in the past couple of years.

I do think that over the years I've laid down a few markers as to what I could do. I was fifth in the time trial in Albi in the 2007 Tour, behind Alexandr Vinokourov, Andrey Kashechkin, Cadel Evans and Andreas Klöden. The first two later tested positive for blood doping so I was effectively third, two weeks into the Tour, at a time when I wasn't concentrating on the race.

I had the engine already, and it showed that year when I won the prologues in the Dauphiné and the Four Days of Dunkirk. As early as 2005 I was seventh in the world time trial championships in Madrid: two of the riders in front of me, "Vino" and Kashechkin, were again, later, done for doping; a third, Rubén Plaza, was implicated in Operación Puerto. That year, I won a mountain stage in the Tour de l'Avenir.

When I look back, we now have an idea of what was going on in the sport back then, and it was a different era. Personally, I used to find it difficult. You'd be trying to negotiate a contract – say £50,000 – I had two kids to worry about, a livelihood to earn in the face of what was going on, and people beating me because they were doping. I had a chip on my shoulder as a result, and I wasn't shy of saying what I thought about doping because it directly affected me and the lives of my family.

Since then, drug tests have begun to work better, the blood passport has come in, so it's harder for people to dope. The chances of getting caught are far higher than they were. I do believe the sport is changing, if you look at what Ryder Hesjedal did at the Giro and what Chris Froome did at the Vuelta. As that change has happened, my performances have gone up, and at the same time I've begun to work far harder than I did before.

I'm not claiming the sport is out of the woods but doping in the sport is less of a worry to me personally, it's less at the forefront of my mind, because I'm no longer getting beaten by people who then go on and test positive or whatever. If there is a difference in my attitude now compared to back then, it's that I'm more focused on what I am doing, I pay less attention to what's going on outside my bubble because I'm not coming second to riders who dope.

It affects me less, in terms of my worrying about it, but the important thing is that nothing has changed in how I stand morally. Nothing has changed about the reasons why I would never dope. In fact, the reasons why I would never use drugs have become more important. It comes down to my family, and the life I have built for myself and how I would feel about living with the possibility of getting caught. I wrote it all in my autobiography back in 2008 and I still feel the same now. It's just I say it less. There is more attention on me, which makes me more withdrawn, and I don't feel easy in a leader's role, as [the cycling author] Richard Moore correctly wrote in his book.

The question that needs to be asked is not why wouldn't I take drugs, but why would I? I know exactly why I wouldn't dope. To start with, I come to professional road-racing from a different background to a lot of guys. There is a different culture in British cycling. Britain is a country where doping is not morally acceptable. I was born in Belgium but I grew up in the British environment, with the Olympic side of the sport as well as the Tour de France. I don't care what people say, the attitude to doping in the UK is different to in Italy or France maybe, where a rider like Richard Virenque can dope, be caught, be banned, come back and be a national hero.

If I doped I would potentially stand to lose everything. It's a long list. My reputation, my livelihood, my marriage, my family, my house. Everything I have achieved, my Olympic medals, my world titles, the CBE I was given. I would have to take my children to the school gates in a small Lancashire village with everyone looking at me, knowing I had cheated, knowing I had, perhaps, won the Tour de France, but then been caught. I remember in 2007 throwing that Cofidis kit in the bin at that small airport, where no one knew me, because I didn't want any chance of being associated with doping. Then I imagine how it would be in a tiny community where everyone knows everyone.

It's not just about me. I've always lived in the UK. All my friends in cycling are here, and my extended family. Cycling isn't just about me and the Tour de France. My wife organises races in Lancashire. I have my own sportif, with people coming and paying £40 each to ride. If all that was built on sand, if I was deceiving all those people, I would have to live with the knowledge it could all disappear just like that. My father-in-law works at British Cycling and would never be able to show his face there again. Their family have been in cycling for 50 years, and I would bring shame and embarrassment on them. It's not just about me: if I doped it would jeopardise Sky – who sponsor the entire sport in the UK – Dave Brailsford and all he has done, and Tim Kerrison, my trainer. I would not want to end up sitting in a room with all that hanging on me, thinking: "Shit, I don't want anyone to find out."

That is not something I wish to live with. Doping would simply be not worth it. This is only sport we are talking about. Sport does not mean more to me than all those other things I have. Winning the Tour de France at any cost is not worth the possibility of losing all that.

I am not willing to risk all those things I've got in my life. I do it because I love it. I don't do it for a power trip: at the end of the day, I'm a shy bloke looking forward to taking my son to summer rugby camp after the Tour, where he could maybe bump into his hero, Sam Tomkins. That's what's keeping me going here. What I love is doing my best and working hard. If I felt I had to take drugs, I would rather stop tomorrow, go and ride club 10-mile time trials, ride to the cafe on Sundays, and work in Tesco stacking shelves.