People are unlikely to react positively to the idea of using citizens as guinea pigs; many will be downright disgusted. But there are times when government must experiment on us in the search for knowledge and better policy.

I am not condoning the type of mass experimentation that has troubling effects on people’s health and is potentially dangerous. There are some horrific examples littering the history of scientific experiments, such as the ‘Tuskegee study of untreated syphilis in the negro male’ – a notorious experiment that ensured that 399 African-American men with syphilis went untreated. Or the Ministry of Defence experimentation on a million unwitting people on the south coast of Britain between 1940 and 1979, to see the impact of germ warfare.

Though history calls into question the ethics of experimentation, unless we try things out, we will never learn. The National Audit Office says that £66bn worth of government projects have no plans to evaluate their impact. It is unethical to roll out policies in this arbitrary way. We have to experiment on a small scale to have a better understanding of how things work before rolling out policies across the UK. This is just as relevant to social policy, as it is to science and medicine, as set out in a new report by the Alliance for Useful Evidence.

Whether it’s the best ways to teach our kids to read, designing programmes to get unemployed people back to work, or encouraging organ donation – if the old ways don’t work, we have to test new ones. And that testing can’t always be done by a committee in Whitehall or in a university lab.

Experimentation can’t happen in isolation. What works in Lewisham or Londonnery, might not work in Lincoln – or indeed across the UK. For instance, there is a huge amount debate around the current practice of teaching children to read and spell using phonics, which was based on a small-scale study in Clackmannanshire, as well as evidence from the US. A government-commissioned review on the evidence for phonics led professor Carole Torgerson, then at York University, to warn against making national policy off the back of just one small Scottish trial.

One way round this problem is to do larger experiments. The increasing use of the internet in public services allows for more and faster experimentation, on a larger scale for lower cost – the randomised controlled trial on voter mobilisation that went to 61 million users in the 2010 US midterm elections, for example. However, the use of the internet doesn’t get us off the ethical hook. Facebook had to apologise after a global backlash to secret psychological tests on their 689,000 users.

Contentious experiments should be approved by ethics committees – normal practice for trials in hospitals and universities.

We are also not interested in freewheeling trial-and-error; robust and appropriate research techniques to learn from experiments are vital. It’s best to see experimentation as a continuum, ranging from the messiness of attempts to try something new to experiments using the best available social science, such as randomised controlled trials.

Experimental government means avoiding an approach where everything is fixed from the outset. What we need is “a spirit of experimentation, unburdened by promises of success”, as recommended by the late professor Roger Jowell, author of the 2003 Cabinet Office report, Trying it out [pdf].

But such a spirit is rare. Over-promising, unfortunately, dominates political debate. Particularly so before a general election, when the stalls are being set out for all sorts of new policies or tax gifts, with the false certainty that they will be delivered in the next parliament. According to Geoff Mulgan [pdf], chief executive of Nesta and former No 10 director of strategy: “No plans survive their first encounters with reality intact”.

In one sense, governments experiment on us all the time. By rolling out big policies with little idea if they will work or not, we are unwitting guinea pigs. Would it not be better to have a more formal experiment, where new policies are first trialled and evaluated before they are unleashed on all of us?

Sign up for your free weekly Guardian Public Leaders newsletter with news and analysis sent direct to you every Thursday. Follow us on Twitter via @Guardianpublic