From jeans to dating partners and TV subscriptions to schools, we think the more choices we have the better. But too many options create anxiety and leave us less satisfied. Could one answer lie in a return to the state monopolies of old?

Once upon a time in Springfield, the Simpson family visited a new supermarket. Monstromart’s slogan was “where shopping is a baffling ordeal”. Product choice was unlimited, shelving reached the ceiling, nutmeg came in 12lb boxes and the express checkout had a sign reading, “1,000 items or less”. In the end the Simpsons returned to Apu’s Kwik-E-Mart.

In doing so, the Simpsons were making a choice to reduce their choice. It wasn’t quite a rational choice, but it made sense. In the parlance of economic theory, they were not rational utility maximisers but, in Herbert Simon’s term, “satisficers” – opting for what was good enough, rather than becoming confused to the point of inertia in front of Monstromart’s ranges of products.

This comes to mind because Tesco chief executive Dave Lewis seems bent on making shopping in his stores less baffling than it used to be. Earlier this year, he decided to scrap 30,000 of the 90,000 products from Tesco’s shelves. This was, in part, a response to the growing market shares of Aldi and Lidl, which only offer between 2,000 and 3,000 lines. For instance, Tesco used to offer 28 tomato ketchups while in Aldi there is just one in one size; Tesco offered 224 kinds of air freshener, Aldi only 12 – which, to my mind, is still at least 11 too many.

Now Lewis is doing something else to make shopping less of an ordeal and thereby, he hopes, reducing Tesco’s calamitous losses. He has introduced a trial in 50 stores to make it easier and quicker to shop for the ingredients for meals. Basmati rice next to Indian sauces, tinned tomatoes next to pasta.

What Lewis is doing to Tesco is revolutionary. Not just because he recognises that customers are time constrained, but because he realises that increased choice can be bad for you and, worse, result in losses that upset his shareholders.

Facebook Twitter Pinterest Scrapping 30,000 products ... Tesco chief executive Dave Lewis is streamlining the supermarket experience. Photograph: Neil Hall/Reuters

But the idea that choice is bad for us flies in the face of what we’ve been told for decades. The standard line is that choice is good for us, that it confers on us freedom, personal responsibility, self-determination, autonomy and lots of other things that don’t help when you’re standing before a towering aisle of water bottles, paralysed and increasingly dehydrated, unable to choose. That wasn’t how endless choice was supposed to work, argues American psychologist and professor of social theory Barry Schwartz in his book The Paradox of Choice. “If we’re rational, [social scientists] tell us, added options can only make us better off as a society. This view is logically compelling, but empirically it isn’t true.”

Consider posh jams. In one study cited by Schwartz, researchers set up two displays of jams at a gourmet food store for customers to try samples, who were given a coupon for a dollar off if they bought a jar. In one display there were six jams, in the other 24: 30% of people exposed to the smaller selection bought a jam, but only 3% of those exposed to the larger selection did.

Now consider – and there’s no easy way to say this – your pension options. Schwartz found that a friend’s accounting firm was offering 156 different retirement plans. Schwartz noted that there was a shift of responsibilities from employer to employee in this seemingly benign transfer of choice: “When the employer is providing only a few routes to retirement security, it seems important to take responsibility for the quality of those routes. But when the employer takes the trouble to provide many routes, then it seems reasonable to think that the employer has done his or her part. Choosing wisely among those options becomes the employees’s responsibility.”

Facebook Twitter Pinterest Posh jam conundrum ... too many options can be baffling. Photograph: Graham Turner for the Guardian

But that’s the problem. Which of us, really, feels competent to choose between 156 varieties of pension plan? Who wouldn’t rather choose to lie in a bath of biscuits playing Minecraft? And yet, at the same time, we are certain that making a decision about our workplace pensions is an important one to get right. But instead of making that choice, Schwartz says, many defer it endlessly. One of his colleagues got access to the records of Vanguard, a gigantic mutual-fund company, and found that for every 10 mutual funds the employer offered, rate of participation went down 2% – even though by not participating, employees were passing up as much as $5,000 a year from the employer who would happily match their contribution.

But even if we do make a choice, Schwartz argues, “we end up less satisfied with the result of the choice than we would be if we had fewer options to choose from”. When there are lots of alternatives to consider, it is easy to imagine the attractive features of alternatives that you reject that make you less satisfied with the alternative that you’ve chosen.

Increased choice, then, can make us miserable because of regret, self-blame and opportunity costs. Worse, increased choice has created a new problem: the escalation in expectations. Consider jeans. Once there was only one kind, says Schwartz – the ill-fitting sort that, fingers-crossed, would get less ill-fitting once he wore and washed them repeatedly. Now, what with all the options (stone-washed, straight-leg, boot-fit, distressed, zip fly, button fly, slightly distressed, very distressed, knee-holed, thigh-holed, knee and thigh-holed, pretty much all holes and negligible denim), Schwartz feels entitled to expect that there is a perfect pair of jeans for him. Inevitably, though, when he leaves the store, he is likely to be less satisfied now than when there were hardly any options.

Facebook Twitter Pinterest In the good old days there was just one kind of jeans ... Photograph: Ben Margot/AP

Schwartz’s suggestion is that, at a certain point, choice shifts from having a positive relationship with happiness to an inverse one. So, what’s the answer? “The secret to happiness is low expectations,” he says, sensibly.

No wonder, then, we aren’t happy. In the 10 years since Schwartz wrote his book, the ideology of unlimited choice has expanded into unlikely areas – schools, sex, parenting, TV – and expectations have risen as a result. Equally importantly, new tactics have developed to help consumers deal with the downsides of choice. For instance, Schwartz notes, there is an increasing reliance on recommendation engines to help people cope with choice. “The internet hath created a problem that it is now trying to solve,” he says.

One of the areas affected is dating. Relationships are being treated like any other product – online we can browse and compare prospective sexual partners.

“I think dating sites are now the most common path for meeting romantic partners, and the overwhelming amount of choice that dating sites have created is a real problem,” says Schwartz. One of those problems was noted by the comedian Aziz Ansari in his book Modern Romance. In it, a woman recounts meeting a man on the dating app Tinder, then spending the journey to their first date swiping through the service to see if anyone better was available. Failure to commit to a date or a relationship can itself be a choice – indeed, the sociology professor who helped Ansari with his book, Eric Klinenberg, wrote Going Solo: The Extraordinary Rise and Surprising Appeal of Living Alone to account for those who have stepped off the treadmill of dating, the nightmare of having more choice but less reason to choose. Hence, too, Japan’s soshoku danshi or herbivore men who, so corrupted by the endless choices offered by online pornography, are no longer interested in real sex or romantic relationships. Psychologist Philip Zimbardo fears that, thanks to how online pornography is offering more choices for masturbatory satisfactions, becoming more interactive and immersive, choosing real-life romantic relationships will become even less appealing.

Facebook Twitter Pinterest BT vying with Sky for football coverage is good news, right? Not necessarily ... Photograph: Rex Shutterstock

There’s another problem with choice: it can be more apparent than real – that is, a seeming increase in choice masks the fact that you’re paying more for the same stuff you had before. My Guardian colleague Barney Ronay identified this when he considered football on TV recently. The ostensibly good news is that BT Sport is competing with Sky for football rights. It now exclusively shows European Champions League football, which should mean more choice, lower customer outlay and more joy, shouldn’t it? But if you are already a Sky Sports subscriber (or, perhaps more pertiniently, watched the free-to-air games on ITV), it means the opposite. If Barney wants to watch the same amount of football as last year, he will now have to pay more.

That sort of phenomenon repeats itself across TV more generally. To watch all the good stuff on TV now involves paying money in the form of monthly subscriptions to Amazon Prime, Netflix, Sky, BT and Blinkbox, as well as having a Freeview box. But who can afford that kind of outlay? A decade ago everything you could ever wish to watch was on Sky (if you were prepared, admittedly, to pay a monthly subscription to Murdoch). A decade before that, all good TV was on terrestrial, so once you had paid for the telly and the licence you were set. What is sold to us as increased choice has thus made us poorer and, if Ronay’s experience is anything to go by, more disappointed. As Ronay says: “for the captive consumer this isn’t really a proper choice at all, but an opportunity to spend the same and get less, or alternatively spend more and get the same.”

UK TV: how much does it cost to watch everything? Read more

Anger at this state of affairs is comprehensible to anyone who lives in an advanced western society in 2015 and has to choose between mobile phone plans, schools, and water, gas and electricity suppliers – not to mention minimally distinguishable prospective dates. Admittedly these are the choices typical of decadent westerners in the era of late capitalism, but that thought doesn’t make the burden of choice any easier to bear.

Consider electricity, says Professor Renata Salecl, author of The Tyranny of Choice. “Privatisation of electricity did not bring the desired outcome – lesser prices, better service – however, it did contribute to the anxiety and feeling of guilt on the side of the consumers. We feel that it is our fault we are paying too much and we are anxious that a better deal is just around the corner. However, while we are losing valuable time doing research on which provider to choose, we then stop short of actually making the choice.”

So we do nothing, and corporations profit from this inertia.

All of this confounds the idea that human beings act in such a way that they maximise their wellbeing and minimise their pain. “People often act against their wellbeing. They also rarely make choices in a rational way.”

The political idea of allowing parents to choose between schools was to apply the presumed rigour of the market to education so that underperforming schools would improve or close. Standards would rise and formerly illiterate brats in key stage II would relax after double quantum physics by dancing around the maypole singing settings of Horace’s verse in Latin, just like in Michael Gove’s dreams.

Facebook Twitter Pinterest Education has become a consumer good, and could all go wrong ... Illustration from Charles Dickens’ Oliver Twist. Mansell/Time Life Pictures/Getty Images

So education has become a consumer good, and my daughter’s is something I’m encouraged to think about as though it were a trip to the shops to buy shoes. Can I buy the best education for my daughter, possibly by moving house, lying about my real address or selling a kidney for private schooling? I’m not sure, but one thing I am becoming increasingly convinced of is what Salecl says: “Ideology that convinces us that everyone can make it if only he or she makes the right choice relies on blindness – we do not see that social constraints stop us making out of our lives what we wish for. And when we think about choice as a primarily individual matter, we also become blind about broader social, political choices.”

What she means, I think, is that the ideology of choice makes us forget that some things shouldn’t be bought and sold, and they are the most important things of all. What’s more, having made a decision, we don’t want to hear we have got our choices wrong. “We are constantly under the impression that life choices we made after careful planning should bring us expected results – happiness, security, contentment – and that with better choices, traumatic feelings that we have when dealing with loss, risk and uncertainty can be avoided.” No wonder, then, that Slacel’s most recent work is on the power of denial and ignorance. “When people are overwhelmed by choice and when they are anxious about it, they often turn to denial, ignorance and wilful blindness.”

Schwartz demurs, arguing that some extensions of choice can be a good thing. When I tell him about the preponderance of academies and free schools that, seemingly, increase choice for British parents, he says a similar phenomenon, charter schools, has arisen in the US. “There is something good about this, since public education in much of the US is dreadful, and competition might make it better. But there is no doubt it is stressing parents out big time.”

As with choosing a pension, choosing a school leaves scope for regret, shame and fear of missing out. And, in extremis, the terrifying sense that I might inadvertently choose an option that will mess up my daughter’s future.

Facebook Twitter Pinterest Challenging the rhetoric of choice ... Jeremy Corbyn. Photograph: Mary Turner/Getty Images

In 2015, though, there are counter-tendencies to the stress-inducing extension of choice. Not only is Tesco reducing its number of products, but the new leader of the Labour party has just been elected on a political platform that, in part, challenges the rhetoric of choice. Jeremy Corbyn proposes to renationalise not just the rail network but public utilities (gas, electricity and water), partly in the hope that the reduction of choice will provide a fairer, less anxiety-inducing experience for their users.

Perhaps, Corbyn’s political philosophy suggests, what we need is not more choice, but less; not more competition but more monopolies. But before you counter with something along the lines of “Why don’t you go and live in North Korea, pinko?” consider this: Paypal founder Peter Thiel argues that monopolies are good things and that competition, often, doesn’t help either businesses or customers. “In the real world outside economic theory, every business is successful exactly to the extent that it does something others cannot. Monopoly is therefore not a pathology or an exception. Monopoly is the condition of every successful business.” Competition, in short, is for losers.

That, of course, doesn’t mean that successful capitalists like Thiel would be supporting Corbyn in his plan to recreate the state monopolies of yore or submit schools once more to local education control, but it does mean the rhetoric of choice and competition is at least being challenged and not only from the political left.

“At least we are talking about a political and economic choice,” says Salecl, “and are not simply following the ‘desires’ of the market.” Perhaps: if she’s right about that, then we are opting for something we haven’t done for a long time.