We live in a time when government seems to have the Sadim touch: everything politicians lay their hands on turns into the opposite of gold. So it is a pleasant surprise when a significant piece of policy affecting the futures of millions of people is working as intended.

Many folk park pensions in that segment of the brain where they keep things they know to be important, but find boring. Many folk would prefer to spend any surplus income today rather than save it for tomorrow. As a result, Britain has a serious problem. Its citizens are saving far too little for their retirement. Five years ago, the government did something to try to remedy this. It changed the way in which workers make pension decisions by introducing auto-enrolment. Where previously employees had to take a series of steps to opt into a company pension, now you are automatically signed up unless you actively choose to opt out. This subtle-sounding switch has had a rather dramatic result. More than eight million people have started saving for the first time, which means they also receive a pension contribution from their employers.

That still leaves many unwilling or unable to save enough for their older age, but it is a substantial advance that we now have nearly 40 million people, a record number, in workplace pensions. Hurrah. A government policy that is working.

This policy success is in part a tribute to the influence of Professor Richard Thaler, the pioneer in behavioural economics, whose work has just been recognised with the Nobel prize. One of his many insights is that people do not always behave in their own best interests. Human beings are, well, human. This might not be a complete shock to you or me, but it was a challenge to classical economic theories that assumed people were always rational actors. From this observation, he developed an argument that nuanced changes in the “choice architecture” of society can trigger desirable shifts in behaviour.

His “nudge theory” was seized on by politicians, especially liberal ones in the west. They were attracted to the idea that their citizens could be induced to make wiser choices without clubbing them over the head with coercive, nannying and opposition-arousing legislation. Barack Obama’s White House got extremely excited about nudge theory. David Cameron set up a behavioural insights team at the Cabinet Office, which was nicknamed the “nudge unit”. The timing was important. Nudge, the bestseller written by Thaler and Cass Sunstein, was published in 2008, the year of the Great Crash.

It is almost certainly not a coincidence that the theory became popular with politicians and other decision-makers when their countries were being crunched by recessions and money was tight. This created a big appetite for low-cost solutions to public policy challenges. Nudging appeared to offer easy ways of reforming society without committing to large spending programmes. Nudge units were put to work in countries as diverse as Australia, Germany and Japan. The concept was also embraced by international bodies such as the UN and the World Bank. Nudge became one of the most globally influential ideas among policymakers.

Nudge has also gone rampant in the market place. The Fitbit on your wrist is a nudge device. The calorie count on food packaging is there to tilt you to healthier eating options. It was once a novelty to find a notice in a hotel bathroom asking you, “dear guest”, to consider whether you could reuse the towel. It is now unusual if you are not greeted with a winsome plea to think about the planet. The hotel is doing a little bit for the environment and, of course, saving on its laundry bill. You know you are being nudged, you know why and you probably don’t mind. I don’t. I did mind going through Gatwick the other day where the only route from security to the gate was a long and snaking forced march through miles of “duty-free”. Well, it felt like miles. This did not make me want to fill my boots with the ciggies, spirits, choccies and fragrances. It made me want to walk faster.

Work and Pensions secretary David Gauke has scrapped the charge of up to 55p a minute for the universal credit helpline. Photograph: PA

The application of nudge theory in politics has been a similarly mixed blessing. When it works effectively, it does so because it exploits human weaknesses for human benefit. It is a frailty of many of our species to procrastinate. Turning pension contributions into an opt-out rather than an opt-in uses inertia to achieve a desirable end. Another winning example is organ donation. In countries where there is a presumption of consent unless the deceased has declared against donation, organ availability has shot up. No one has been deprived of a freedom. You can still decide that you prefer to take all your bits with you to the grave. But you have been influenced to make a different choice that is more beneficial to society.

In truth, nudging was being practised by artful governments long before it was given a name. The pre-eminent example of that is smoking policy. The tobacco habit is highly addictive and deadly, so government could decide to make it illegal. Which would be a terrible idea, because a blanket ban would criminalise millions of otherwise law-abiding citizens and provide a massive opportunity for organised crime to create an underground market. So instead, successive governments have used the nudge. First, TV advertising of cigarettes was prohibited. Then, all advertising was banned. This was followed by a stop on the open display of cigarettes in shops. Now fags have to be wrapped in packaging plastered with pictures of horrible diseases. One of the best and cheapest contributors to the improvement of our nation’s health is the prohibition on smoking in public places, which was introduced by Tony Blair’s government. That decision tore apart his cabinet. Some ministers feared a furious backlash from the millions of voters who would no longer be able to puff away in pubs and restaurants. As it turned out, it was introduced with minimal fuss. Predicted pub riots by fuming smokers never materialised. You are part of a small and eccentric minority – and probably also a member of Ukip – if you still think it was a bad idea to end smoking in restaurants and pubs. You are part of a smaller, and frankly weird, minority if you think smoking should still be allowed on trains and planes, in cinemas and at football stadiums and on the London underground.

A big reason for this policy success is that it has gone with the grain of human desires. You’d expect that ban to be attractive to non-smokers. It has worked because it wasn’t opposed by smokers. Most nicotine addicts want to quit. Importantly, it wasn’t a total ban. You can still smoke, so long as you don’t mind going outside. That nudge has worked because it guides, rather than compels, folk to go in the right direction. Nudge doesn’t work when it loses touch with the human factor. A topical example is the trouble the government has got into with the introduction of universal credit. This simplification of the benefits system is founded on the excellent principle that work should always pay: no one should be worse off by deciding to take a job or put in more hours. The implementation is going wrong because it failed to take into account how lives are lived. People on low incomes can be a day’s pay away from not being able to put dinner on the table. So a delay of five weeks or more in paying the credit is an atrocious design fault. Charging up to 55p a minute for calls to the helpline was simply stupid, as ministers have belatedly realised.

This illustrates one of the downsides of nudge. It is highly dependent on the nudgers getting it right and we know that politicians and civil servants are also fallible. Just like other human beings, they miscalculate risks, prioritise short-term gratification over long-term achievement and can act irrationally. While technocrats quite often really do know what is good for us, sometimes they don’t, and even the best-intentioned can make bad mistakes. For years, the tax system was used to incentivise drivers to move away from petrol cars and towards diesel vehicles because experts declared diesel to be less harmful to health and the environment. That turned out to be a faulty nudge.

“Liberal paternalism”, the posh label for nudge, assumes that there is an elite that knows what is good for the citizenry. This idea – the establishment knows best – is precisely the one that significant numbers of voters have been rebelling against. One of the more powerful critiques of nudge is that it concentrates on the psychological manipulation of voters rather than properly educating them about choices, and the ultimate effect of this is to infantilise the citizenry.

Experience has shown that nudge is not the miracle cure for every political challenge. Some problems are just too big to be fixed by adjusting the “choice architecture”. Britain’s housing crisis is not solvable with a few tweaks to the tax system and the planning regime. That won’t be cracked without bolder and stronger measures. Not a gentle hand on the elbow, but a muscular kick up the arse. Nudge has some proved beneficial uses for governing, but it is not the answer to everything. Some things need the push and the shove.