Open the business pages at this time of year, and a whole bunch of telephone numbers come tumbling out. An average payout of £233,000 for the investment bankers at JP Morgan. A £9.7bn pot for the swots at Goldman Sachs. And a £2m kiss goodbye for the boss of Lloyds, Eric Daniels, presumably as thanks for bungling the high-street bank's affairs so badly that it now relies on cash from the British taxpayer.

Ask a City executive to justify such huge bonuses, especially as the rest of the country braces itself for the biggest spending cuts since 1945, and you get a brush-off. The financiers have two main justifications for their mammoth handouts. First, bonuses spur on staff to perform better, which in the end is better for both the business and the wider economy. The second defence is the one given by the boss of Barclays, Bob Diamond, to MPs on the Treasury select committee last week. Unless banking chiefs paid out these vast sums, he said, they would lose their best and brightest and oh-so-rare employees: "The other option is that we don't have investment banks located in the UK." (Diamond, by the way, is himself in line for an £8m bonus.)

Well, guess what? I've been through stacks of the research on bonuses, and there is something in what Diamond, Daniels and those other tanned boys in their corner offices say. All the evidence suggests that where bonuses would be most useful is not in finance – but in jobs such as fruit picking and working on supermarket checkouts. The people who should be getting bonuses aren't in the glass and steel office blocks of Canary Wharf, but are further east, getting caked in mud in the fields of Kent.

Take the financiers' first line of defence, about the relationship between pay, targets and performance. In 2005, the Federal Reserve Bank of Boston, a regional branch of the US central bank, published a piece of research called Large Stakes and Big Mistakes. In it, a team of eminent behavioural economists reported on experiments they had conducted with American undergraduates. The students were offered money to tap a keyboard as fast as they possibly could, and also to add up some numbers.

When it came to the simple chore of hitting computer keys, bonuses worked a treat: the more cash on offer, the faster the undergraduates tapped. On the more complex task of doing maths, however, incentives served to worsen performance. "Tasks that involve only effort are likely to benefit from increased incentives," wrote the economists. "While for tasks that include a cognitive component, there seems to be a level of incentive beyond which further increases can have detrimental effects on performance."

In other words, bonuses can spur workers on to do basic mechanical tasks faster and better – clearing a field of fruit before it goes rotten, say, or scanning in multi-packs of Andrex in busy supermarkets. But on more complex tasks, any sum beyond a paltry one is counter-productive. The same results have been shown in other studies. When investment bankers argue that their work is so complex they need bonuses, they are contradicting the research. And of course, they are ignoring the history of the past few years, which shows that bonuses drove an entire industry to pull stupid gambles – with disastrous consequences.

Now for the other argument, that financial institutions need incentives to keep these superstars. That fallacy is neatly quashed by Boris Groysberg in his recent book, Chasing Stars. An academic at Harvard Business School, Groysberg studied 366 Wall Street equity analysts who changed employers between 1988 and 1996. He chose these 366 because they had been rated number one in their field before moving. He found that once these stars swapped banks they were no longer so super: "Their job performance plunged sharply and continued to suffer for at least five years after moving to a new firm."

"Moving employer on Wall Street is no big deal," Groysberg tells me. "You hand in your BlackBerry, you pick up your coat, you cross the street and in 45 seconds you can be back in business." But what you leave behind are your colleagues, your bosses, your knowledge of how your company functions – in other words, all the institutional and collective factors that made you a success, but which usually get forgotten in the acclaim for individual achievement.

The mirage of talent is how management writer David Bolchover labels this. He argues that those skills aren't so rare nowadays when China and India are churning out tens of thousands of maths and engineering graduates.

Besides, rewards in financial services are already so high that the City lures in bright young graduates like Camden market draws goths. On the fruit fields of Kent, however, the shortage of talent is real. Employers have to hire migrants from eastern Europe to do the work that others won't do. The same goes for supermarkets, where bored teens sit at tills. Yet most of us would prefer to have fruit and grocery shopping than bad advice on mergers and acquisitions. The solution, I would say, is simple: award the bonuses to the fruit- pickers, the checkout staff and assembly-line workers. We can discuss what to give the bankers later.