I used to be a smoker. The first thing I did when I woke up at 6am was to reach out of bed, grab a cigarette and light up. It was my start to the day, every day. I tried several times to give up, but only lasted a few months before going back. During one of the periods when I was off cigarettes, I went to the pub. Somebody bought a round, then someone passed round a packet of fags. I foolishly took one – I'd had a few drinks – and the next thing I was a smoker again.

That was about 30 years ago, and I finally managed to kick the habit soon after that, partly because I met a woman who didn't smoke. We married and had four children.

Much has changed in our behaviour and attitude towards smoking. Happily, far fewer people now smoke – it's down to 21% of those over 16 in England, compared to 39% in 1980 – and smoking has been restricted in a number of useful ways. The action this government has taken against smoking has been very welcome and it deserves credit for that. They have banned most tobacco advertising, brought in graphic picture warnings on cigarette packets, and raised the age at which it's legal to buy cigarettes from 16 to 18.

The government also introduced the ban on smoking in public places, such as pubs and restaurants, which has made a big difference to public health, although they did wait until Scotland and Ireland had changed the law before following suit.

Tomorrow, the House of Commons will see the third and final reading of the Health Bill. I believe MPs must support the clause that would ensure that cigarettes being sold in shops must be stored out of sight, though I'm puzzled and concerned as to why so many seem reluctant to back this move.

Ending point-of-sale displays in this way is an excellent idea and necessary. Whether by accident or design, the primary colours used on both the outside of cigarette packets and the shelves they are kept on attract children. It's the same marketing technique that is used with confectionery.

Tobacco companies can't be trusted to make cigarette packets a colour that doesn't work in this way, so they should be put out of sight. This is very important because most smokers get hooked as teenagers – few start at 18 – so if we can stop children being addicted tobacco companies will soon run out of customers.

The government's "de-normalisation" of tobacco is welcome, but it's taking too long. The Health Bill proposes to restrict cigarette-vending machines in pubs. But they should be banned altogether. Even smokers don't like them, because they typically give you only 16 cigarettes instead of a normal packet of 20 and cost £6, about £1 more than in the shops. And many pub landlords think the government's halfway-house proposals are unworkable because bar staff would have to check people's age ID before operating the machine by remote control.

In my view smokers who currently stand outside a pub or restaurant having a fag should have to stand at least several yards away from the front door, to save the 79% of us who don't smoke from breathing in their smoke when we go in or out. We should curtail the rights of the 21% and increase their responsibilities towards the 79%. In other words, we should stop them killing us and our children.

Studies estimate that about 11,000 people a year die because of passive smoking. This isn't nanny statism, Big Brother, or wrongful interference in people's personal freedoms – it's the right thing to do to protect the health of the vast majority of us who don't smoke from the declining minority who do.

Smoking should be banned in cars, and particularly any vehicle with children in it. On a school visit I met a 12-year-boy who wanted to be an athlete who told me that every morning his mother lit up when she was driving to school, even though he'd begged her to stop. He should be able to report her to the police.

It should also be illegal to smoke at home in front of children. I accept that enforcing such a law would be difficult, but it would send a message that such behaviour is unacceptable. And shops should need a licence to sell cigarettes. They need a licence to sell alcohol, which is sometimes addictive and certainly harmful, just like tobacco, so why not? That would make shopkeepers less likely to sell fags to people under age.

Some shopkeepers are genuinely afraid of a ban on tobacco displays. But that is because the tobacco industry have been up to their old tricks. They tried to convince pubs that the smoke-free law would drive them out of business so they would lobby against the law.

Now they are doing the same thing with shopkeepers. A retailer from the northeast recently went to Ireland to find out the truth and the shopkeepers he spoke to told him that now they had won their displays back from the tobacco companies who controlled them, they were free to promote products that allowed them to make a healthy profit.

Recently I spent some time in Mauritius; not the six-star hotel bit, but the parts where the poor people live where so many die of heart attacks because of smoking. The island's surgeon-general said that he wanted Mauritius to become the first totally no-smoking country in the world. I would like the UK to get there first.

Duncan Bannatyne is president of No Smoking Day and a patron of the charity QUIT.