Black Box Thinking: The Surprising Truth About Success. By Matthew Syed. John Murray; 352 pages; £20. To be published in America in November by Portfolio; 336 pages; $27.95.

Inside the Nudge Unit: How Small Changes Can Make a Big Difference. By David Halpern. W.H. Allen; 400 pages; £20.

EMBRACING failure is a cliché of the business world. The Harvard Business Review devoted an entire issue to it in 2011. People should be open about their mistakes and tolerant of others’; this is the route to improvement, goes the thinking. But as Matthew Syed, a journalist at the Times, shows in a new book, “Black Box Thinking”, in practice a “stigmatising attitude towards error” pervades everyday life. This has big implications.

Success brings its own rewards, but the world comes down hard on those who are deemed failures. The desire to avoid such opprobrium prompts people to cover up mistakes, argues Mr Syed. Doctors tell patients of “complications”. Police fail to drop cases against people accused of committing a crime, even after clear evidence emerges of their innocence. Politicians plough on with policies even when it is obvious they are not working. All are psychological strategies to avoid admitting fault.

Fear of failure can have devastating consequences, as Mr Syed shows in a story about United Airlines. In 1978, as a plane approached its destination, the pilot worried that the landing gear had not come down. Desperate, he tried to establish what was wrong, becoming blinded to the plane’s dwindling fuel reserves. Eventually the tank was empty and the plane crashed. The worry of making a mistake—subjecting the passengers to a bumpy landing—blinded him to bigger problems.

The story is a metaphor. Investors hold on to losing stocks longer than they should. Unable to face the shame of a bad return, they end up with a much bigger loss. Fred Goodwin of RBS, a bank, fretted about the colour of the carpets at head office while his firm collapsed under the weight of the financial crisis. The medical profession is especially intolerant of mishaps, says Mr Syed. This means that mistakes are not scrutinised and people do not learn from them. Small wonder that blunders are pervasive. According to one study of acute care in hospitals, one in ten patients “is killed or injured as a consequence of medical error or institutional shortcomings”.

What to do? One solution is making it easy for people to own up or speak up, as the airline industry has learned to do better than any other. Mr Syed’s more novel suggestion, though, is the rigorous testing of business strategies. This forces people to make improvements. The gold standard is the “randomised control trial” (RCT), in which a treatment group is compared with a control group. Capital One, a credit-card company, has used RCTs obsessively—over the fonts it uses, for example, and the scripts at its call-centres—to assess which initiatives fail and which do not. James Dyson, a technology entrepreneur, and Google are other cheerleaders for this hyper-rational school of management.

This approach may also hold benefits for governments. David Halpern is the boss of the British government’s Behavioural Insights Team (BIT), known as the “nudge unit”, which uses RCTs to improve policy. His new book, “Inside the Nudge Unit”, offers an interesting, if familiar, discussion. Identifying points of failure and making small changes, he argues, reaps disproportionate gains. By including a message on a car-tax form appealing to people’s sense of humanity, the BIT sharply boosted organ donations.

Much still needs to be done. Between 2010 and 2012 the BIT saved the British government only £300m ($457m), a negligible proportion of GDP. Few businesses incorporate RCTs as extensively as Capital One. Much more could be done. Hospitals could subject doctors to RCTs; identify the mistake-prone and then help them. Civil servants could randomly test the economic impact of policies, such as changes to income tax, before rolling them out. It sounds extreme, but confronting failure rationally would bring huge rewards.