A shopper wants to buy some cheese, but he must decide between the 197 varieties on offer. Conventional wisdom says that such a huge choice makes it more likely he'll make a purchase. But that doesn't take into account human unpredictability, according to the new discipline of behavioural economics. Aditya Chakrabortty reports

The Sainsbury's on London's Cromwell Road is hardly a superstore, yet it stocks 197 varieties of cheese (plus another 27 diet versions), 42 types and sizes of bottled water, and enough sub-species of tomato to perplex a pizzeria. It is, in short, a slalom course of options and decisions, of a sort we face every day - from picking gas suppliers to taking out a mortgage. The question is: do we always choose what is right for us?

No, argues Dan Ariely, one of an increasingly influential group of behavioural economists, who analyse how people behave everywhere from supermarkets to stock markets - and they have found a chasm between what traditional economists and regulators presume we do, and what really happens. One of the most exciting areas of research, behavioural economics could overturn many of the assumptions and assertions that shape modern policy-making.

Ariely presents his case, as is the behaviouralist way, based on evidence from real life, such as Sainsbury's. Instead of algebra, he uses shopping trolleys; in place of textbooks, he refers to shelf prices. And to attack the cliche that people always want more choice, he deploys ... jam.

"Economists know all about choosing jam," he says, ambling down an aisle with 73 varieties. He describes an experiment where academics set up a tasting booth in a store in California. On some days they put out six kinds of jam, on others 24. When the booth had 24 types, it was mobbed - "there was more colour, more excitement". But it was the sales that were truly remarkable: with six jams on show, 30% of customers bought a jar; when 24 were out, only 3% did. "Jams are hardly complex things, but people saw 24 stacked together and thought: 'I have no idea how to deal with this.'"

If that is how choosing between strawberry or plum makes us feel, imagine the toll looking at mortgage options takes on the nerves. What Ariely's jam study suggests is that, contrary to economic belief that more choice is better, confronted with too much complexity, we make bad decisions, or stick with what we have already got. An example of that kind of inertia is right behind him. Three Chinese-origin boys are motoring down the aisle, their trolley stacked high with cheese-and-tomato spaghetti ready meals. They are engineering students.

"Did your parents teach you this was good to eat?" asks Ariely.

"No," they chorus.

"There must be other things you could have. Of all the instant foods available in the UK, the only option you like is cheese-and-tomato spaghetti value meals?"

There is no reply, only nervous giggling. It's evidently hard work being grilled by an intense academic in red trainers.

"They have habits!" he beams. Orthodox economists don't recognise habits. "They assume ordinary people do a constant cost-benefit analysis on everything they do. But actually, after you reach a decision, you say, 'That's the end of it!' - and just continue." Which is one reason why more competitors entering an industry does not immediately prompt customers to swap.

Jams, ready meals and a large dollop of human psychology: no wonder these behaviouralist upstarts have come in for some sniffiness from the academic old guard. Most British students still graduate without ever coming across the field. Which is a shame, because a lot of behaviouralism sounds like an extended freshers' week: in the name of research, Ariely has served beer laced with vinegar, and persuaded Berkeley undergraduates to masturbate while filling in questionnaires on laptops (wrapped in protective film, naturally).

Despite such antics, behavioural economics has already produced one Nobel prize winner (Daniel Kahneman in 2002), and is winning over politicians too. While he was a professor at the University of Chicago, Barack Obama befriended two of that campus's most eminent behaviouralists, Cass Sunstein and Richard Thaler. They are now informal advisers to the Democrat hopeful - and their fingerprints are all over his policies. Britain is also starting to cotton on. Just before Christmas, the UK's chief consumer watchdog, the Office of Fair Trading, launched a behavioural economics unit - and in March the National Audit Office called for more government agencies to apply the discipline's findings.

If these are the field's pioneers and politicos, Ariely is its populariser. A professor at the Massachusetts Institute of Technology, his book, Predictably Irrational, clearly sets out the behaviouralists' argument that average people are all far more irrational and more human than economists allow. His lectures are one of the wonders of modern academia; coming on like Jackie Mason with a PowerPoint presentation, he machine-guns through a stream of jokes, optical illusions and a video of a child in a gorilla suit. A few weeks ago, he did the routine at the London School of Economics, and prompted that rare thing in a lecture hall, a stage invasion. Impartial observers swore that they had never seen such excitement among the normally mild-mannered students of portfolio theory.

Back in the supermarket, Ariely is explaining the complications in that most elementary of tasks: evaluating how much to pay for something. "In standard economics, that should be your most immediate thought: how much pleasure is this giving me, and what would I pay for it? Setting that price should come automatically. But in reality it's very hard to do." As if on cue, a blonde woman sails by with a trolley-load of Actimel.

"How much would you pay for this?" he asks.

"Well, I object to paying £4.50. It used to be £4," she says.

"In the States it costs $20. Would you pay that much there?"

"I suppose so. I'm rather hooked."

Another point scored against conventional economics. "First the woman moaned about paying more than £4, then she agreed to pay $20 - £10!" Doesn't that just prove she's very fond of yoghurt? "We all do it ... A standard economist would say that, by definition, every shopper is making the best decision they can. But consumers don't set their own prices from scratch, they're guided by the recommended retail price, or the manufacturer's suggested price - even discounts are set by the shops."

Take the word beloved by shoppers everywhere: free. The very term would normally set an economist's teeth on edge; Milton Friedman, after all, always denied there was such a thing as a free lunch. But like the marketing departments, behavioural economists acknowledge the power of the word. "When you see something for free - for FREE! - it's like pressing a magic button. You forget about any possible downside and just think: Oh my God, it's all good!"

Ariely has been suckered himself. Not so long ago, he went to buy a people carrier to ferry around his family. Next to the dealership was an Audi showroom, offering free - sorry, FREE! - oil changes for new customers. You can guess the next bit: he came home with a sports car too small for anything but posing. "I chose the wrong car because of an oil change worth $150." The Audi cost $30,000.

Do mainstream economists really approach shopping and life with such clear and cold eyes? Listen to the story of Oz Brownlee, late professor at the University of Minnesota. One Friday, he and a colleague stopped to buy some steaks. Finding a long queue, they offered cash to the person in front to swap places. The shopper was dumbstruck - which the academics took as a bargaining ploy, so they raised the price. As news spread down the line, other customers turned hostile. The Minnesota department of economics alumni newsletter goes on: "Attempts to explain that they were ... trying to ascertain whether there was a mutually beneficial trade of time for money that might improve their welfare and that of the next person in line without disadvantaging others met with little success." The economists left without any steak.

Brownlee's mistake was to put into practice something that worked only in theory. That, Ariely and other critics say, is the point: conventional economics tries to make the man fit the model, rather than the other way around.

It wasn't always like this: Adam Smith had his theory of moral sentiments, and Keynes discussed the "animal spirits" of share investors. But as neoclassical economics gained ground, maths replaced psychology. And the human quirks exhibited in everyday practices, from buying chocolate to taking out insurance, were lost.

Neoclassical economists even use their own ideal human in their modelling, called Homo economicus, or rational man. Not so much a straw man as an Excel man, he is a basic creature, says Pete Lunn, a Dublin-based behaviouralist. "Economists treat all people as selfish; as independent; as rational. Those are the assumptions that underpin economics."

Dons debate them at high table, but in Whitehall these beliefs might as well be carved in stone. And the result is some daft policies. Take the opening-up of the directory inquiries market in 2003, when the 192 service was replaced with hundreds of 118-numbers. This was conventional economics in action: give the customer plenty of options and information, and let them make their own way. The result was a jump in prices, a spike in complaints - and what regulators now admit was a shambles. How might behavioural economics have helped? Well, the jams experiment would have proved to officials that too many options only bamboozle people. And civil servants meeting the Actimel guzzler would have seen first-hand our wobbly grip on what constitutes a fair price.

Yet mainstream economics often works. The British tax petrol far more than the Americans, for instance, which is an incentive not to consume as much petrol. And indeed, the cars on our motorways are more fuel-efficient than the ones on their interstates.

Isn't Ariely a bit harsh on orthodox economists? They are after the bigger picture, the Olympian perspective, and from up there, individual quirks blend into one smoothly functioning economy. "That would be true if we all made mistakes in lots of different ways, which would cancel each other out," he says. "But if we all make similar mistakes, there is something systematically wrong. And that is often what really happens." Which accounts for the turmoil in stock markets? "We all get too afraid of what's happening and we feed on each others' fears. The stock market is exaggerating this widespread fear."

So our irrational behaviour is not confined to the chiller cabinets but spills over into the financial markets. It's time to leave the supermarket and meet Anna Sofat, a seasoned financial adviser. The professor wants guidance on what to do if he had a windfall of £3.5m and moved to London. Hmmm. There is obviously a lot riding on his new book.

Sofat asks about risk. How does the economist feel about it? "Money: I don't know - I've never had much. On holidays, I am very adventurous. Sex: somewhere in the middle." There's a pause. Rather a long one. "OK, how should I invest my lump sum?" asks Ariely.

A plan is sketched out: put some in cash, the rest in bonds and property. Expertly done, but it does not answer the economist's implicit question: how can a neat investment plan suit anyone with the variety of attitudes and appetites displayed by Ariely - and, indeed, most of us? Later, the academic is sceptical: "Money and risk are abstract, complex things. But think how quickly she made plans for me to spend my lifetime income. I'd spend the same amount of time planning for my retirement as I would have done on buying a new digital camera."

So what is his solution? "We must be clear about what people can do and can't do. We should trust them entirely on what they know, such as their priorities. And we should completely discount what they're saying on what they can't know - like how much they need for their retirement income."

People cannot always figure out what is in their best interests, he thinks, and experts should accordingly give them the occasional nudge. Our pensions system has already got there: in a few years' time, nearly every worker will be automatically enrolled in a retirement-savings scheme, like it or not. They can leave, but behavioural research suggests that they won't - out of the same inertia that prevented them taking a pension in the first place.

Push that argument a tiny bit further and you cross the thin line that separates economics from politics. Britain has become a full-blown consumer society - one hellbent on giving people maximum choice, even when it harms them. Industrial quantities of booze are sold as loss-leaders in supermarkets; junk mail is full of offers for credit cards (although they have slackened off since the advent of the financial crisis); and it has never been so cheap to fly. Choice is king, markets know best, and anyone who says different is economically illiterate. That is the orthodoxy, anyway.

A few years ago, Ariely met senior executives at a venerable bank in Manhattan, and urged them to produce a "self-control" credit card, one that blocked excessive spending. Guess what? "They never called me back," he writes. "It might have been that they were worried about losing the $17bn in interest charges."

Evening arrives, and the professor is not going to sing for his supper so much as debate for his dinner. He will argue about the first principles of economics at the aptly named Rules restuarant in central London. David de Meza, head of managerial economics at the LSE, has gamely agreed to play a brutal rationalist. Do they really see the world so differently? Let's start with a nice easy one: what should the government do to stave off recession? Ariely thinks President Bush has already grasped one behavioural truth: "After 9/11, Bush urged Americans to spend. He made it a patriotic duty. Avoiding recessions is partly about holding up confidence, so people still feel like spending." Confidence is hardly the issue, retorts De Meza. "One of the features of the boom was overoptimism in spending and markets."

So how would they have avoided the bubble that landed us in this mess in the first place? "Even for economists such as me, it's incredibly difficult to compute how much borrowing is in your best interest," says Ariely. "So you could tell people how much they can borrow."

"But what about entrepreneurs?" asks De Meza. "They borrow a lot. Do you seriously want potential entrepreneurs to sit an exam before they get funding?" Academics evidently do not watch Dragons' Den. "What's the big deal? Let's limit people's ability to hurt themselves in borrowing like we do with seatbelts in driving," says Ariely.

Finally, the biggie: is tipping irrational? Yes, thinks Ariely - it's just convention. No, argues De Meza: "What you're often paying for is nice treatment next time around." The manager, John, is called on to provide expert testimony. Poor man, there is a lot at stake: his own profession's interests, as well as serious academic inquiry. Still, after a bit of squirming, he sides with the behaviouralists: "It is slightly irrational, I suppose," he says. But, of course, we leave a tip.

· This article was amended on Friday May 23 2008. We originally referred to the University of Chicago as Chicago University. This has been corrected.