Gender inequality has been an issue in the City for years, but now the new science of 'neuroeconomics' is proving the point beyond doubt: hormonally-driven young men should not be left alone in charge of our finances…

For the past few weeks I've had two books by my bed, both of which offer a first draft of what history may well judge the most significant event of our times: the 2008 financial crash. Read together, they are about as close as we might come to a closing chapter of The Rise and Fall of the American Empire. As literature, one of them – the final report of the Financial Crisis Inquiry Commission of the US Treasury – doesn't always make for easy reading: there are far too many nameless villains for a start. And, quite pointedly, there is not a heroine in sight. Reading the report I became preoccupied by, among other things – the fairy steps from millions to billions to trillions, say – the overwhelming maleness of the world described. The words "she", "woman" or "her" do not appear once in its 662 pages. It is a book, like most historical tragedies, written about the follies and hubris of men.

The other book, an entirely compulsive companion volume, is Michael Lewis's best-selling The Big Short, which Google Earths you into the crisis. Rather than looking at a global picture, it lets you into the bedrooms and boardrooms of the individual corporate men who catastrophically lost billions of dollars and, on the other side of those bets, the extraordinary ragtag of obsessive individuals who saw what was coming and made eye-watering fortunes. It gives the crash a human face, and once again that face is universally male.

The books are linked by more than subject matter, though. Lewis, a one-time bond trader himself – he left, 20-odd years ago, in incredulity and disgust to write his insider's account, Liar's Poker – gave evidence to the Crisis Inquiry Commission over the course of its 18-month sitting. In the end, however, he refused to sign off the report; and not only did he refuse to sign it, he also refused to put his name to the dissenters' addenda to the report, which three of the committee insisted upon. And not only that, he did not add his name to that of the single individual who insisted on a further addendum stating that he dissented from the dissenters' view. Lewis was not a fan of the report.

The reason for this was simple, he suggested. He felt that the committee, for all its considered judgment, had not understood, from the outset, a single, pivotal word. That word was "unprecedented". Though the inquiry had set out in the belief that the crash was an event different in kind to anything that had gone before, it nevertheless proceeded to judge it in the terms of previous crashes. What it failed to do, in Lewis's eyes, was this: it neglected to look for the things that might have changed in Wall Street or the City, the things that might have made individuals on the trading floors act in ways that were seen to be entirely, unprecedentedly, reckless. When he came to consider these things himself, Lewis felt that perhaps chief among the unprecedented novelties was this one: women.

"Of course," he observed, with tongue firmly in cheek, "the women who flooded into Wall Street firms before the crisis weren't typically permitted to take big financial risks. As a rule they remained in the background, as 'helpmates'. But their presence clearly distorted the judgment of male bond traders – though the mechanics of their influence remains unexplored by the commission. They may have compelled the male risk-takers to 'show off for the ladies', for instance, or perhaps they merely asked annoying questions and undermined the risk-takers' confidence. At any rate, one sure sign of the importance of women in the crisis is the market's subsequent response: to purge women from senior Wall Street roles…"

When I first read those remarks it was not clear how much in earnest Lewis had been when he made them. Subsequently, though, I heard him speak at the London School of Economics, and he took this idea in a slightly different direction. When asked what single thing he would do to reform the markets and prevent such a catastrophe happening again, he said: "I would take steps to have 50% of women in risk positions in banks." Pressed on this, he went on to suggest how science reveals that women in general make smarter decisions regarding investment than men, that when it comes to money, women in couples are demonstrably better at evaluating risk than their partners, and single women much better still.

Though those of us males who have an uncanny sense of money always slipping through our fingers might anecdotally believe this to be true, I was surprised to hear it stated as a fact. It seemed to beg a number of questions. First, if women really are better at making these judgments, why is it always men, still, without exception, who troop out before select committees to explain where it all went wrong, and how they weren't really to blame. And second, would it really be different if women were in charge?

You don't have to look too far into the science to realise that Lewis's claim, in broad terms, stands up. The first definitive study in this area appeared in 2001 in a celebrated paper that broke down the investment decisions made with a brokerage firm by 35,000 households in America. The study, called, inevitably, "Boys will be Boys" found that while men were confident in making multiple changes to investments, their annual returns were, on average, a full percentage point below those of women who invested the family finances, and nearly half as much again inferior to single women.

A more recent study of 2.7 million personal investors found that during the financial crisis of 2008 and 2009, men were much more likely than women to sell any shares they owned at stock market lows. Male investors, as a group, appeared to be overconfident, the author of this study suggested. "There's been a lot of academic research suggesting that men think they know what they're doing, even when they really don't know what they're doing." A fact that will come as a surprise to few of us. Men, it seemed, typically believed they could make sense of every piece of short-term financial news. Women, never embarrassed to ask directions, were on the whole far more likely to acknowledge when they didn't know something. As a consequence, women shifted their positions far less frequently, and made significantly more money as a result.

Naturally, if these findings were widely applicable, then it would be hard not to agree with Lewis's suggestion for reforming the sharpest end of capitalism. Rather than ring-fencing casino investment banks or demanding that high street banks hold vastly greater capital, as we heard at the Mansion House last week, wouldn't a safer model just be to hire more women?

To argue this case, you would probably need more than just behavioural evidence; you might need to understand some of the mechanisms which produced the trillion-dollar bad decision-making that led to what happened in 2008. In recent years, and particularly since the crash, a new science of such decision-making – neuroeconomics – has become fashionable in universities and beyond. It proposes the idea that you will create a better understanding of how people make economic choices if you bring to bear advances in neurobiology and brain chemistry and behavioural psychology alongside traditional economic maths models. Not surprisingly, neuroeconomics has plenty to say about the question of whether decision-making, in high-pressure situations, divides on gender lines.

The problem is that most of the scenarios used to investigate this divide are artificial. It is one thing attaching someone to an MRI scanner and telling him or her that a million pounds rests on their decision in a game; it is another when that person actually stands to lose a million pounds. Only one study, as far as I could discover, has had access to the brain chemistry, the neural biology, of young men actually working on trading floors. But the results it produced were nonetheless startling.

The study was led by a pair of Cambridge researchers. One, Joe Herbert, is a professor of neuroscience, and the other, John Coates, a research fellow in neuroscience and finance. Herbert, a specialist in the effect of hormones on depression, was fascinated to put some of his theories about the role of chemicals on decision making into practice. The curious thing about banks, he told me, "was that they know all about computers and systems and markets but they know next to nothing about the human machine sitting in the chair in front of screens making decisions. Nothing. We aimed to correct that just slightly."

It was Coates, though, who made the experiment possible. Having met Herbert at his lab in Cambridge, I met Coates in a pub in west London. He had a special advantage in gaining access to bond traders' brains, he explained: he used to possess one himself. Sharp-eyed and fit-looking, Coates retains the intensity of a man who used to run a trading desk on Wall Street during the dotcom bubble. He started off at Goldman Sachs and went on to Deutsche Bank. After some years trading, and making a lot of money out of a lot of money, he became increasingly fascinated by the way, during the dotcom years, the traders he worked alongside radically changed behaviour. They became, he says, "euphoric and delusional. They were taking far more risks, and were putting up trades with terrible risk-reward profiles". The dotcom was fun, in a way, he suggests; it was like the roaring 20s. "But I don't think anyone looks back on the housing bubble and laughs."

Coates was a relatively cautious trader himself, but there had been times when he too felt this surge, this euphoria: "When I had been making a lot of money myself, I felt unbelievably powerful," he recalls. "You carry yourself like a strutting rooster, and you can't help it. Michael Lewis talked about 'Big Swinging Dicks', Tom Wolfe talked about 'Masters of the Universe' – they were right. A trader on a winning streak acts exactly that way."

The second thing that Coates noticed was even more revelatory to him. "I noticed that women did not buy into the dotcom bubble at all," he says. "You couldn't find one who did, hardly. And that seemed like a pretty cool fact to me."

With this cool fact in mind, Coates began splitting his time between his trading desk and the Rockefeller University in Manhattan, which is perhaps the world's leading institute for the study of brain chemicals. There he started to become interested in steroids, and in particular something called "the winner effect". This occurs when two males enter a competition and their testosterone levels rise, increasing their muscle mass and the ability of the blood to carry oxygen. It also enhances their appetite for risk. Much of this testosterone stays in the system of the winner of a competition, while the loser's testosterone melts away fast; in evolutionary terms, the loser retires to the woods to lick his wounds. In the next round of competition, though, the winner already has high levels of testosterone, so he starts with an advantage, and this continues to reinforce itself.

"Steroids," Coates explains, "like most chemicals in your body, display what is called an inverted U-shaped response curve." That is to say, when you have low levels of them you lack vitality, and do very poorly at mental and physical tasks. But as the levels rise you get sharper and more focused until you reach an optimum. The key thing is this, however: "If you keep winning, your testosterone level goes past that peak and sliding down the other side. You start doing stupid things. When that happens to animals, they go out in the open too much. They pick too many fights. They neglect parenting duties. And they patrol areas that are too large." In short, they behave like traders on a roll; they get cocky.

Coates became convinced that this winner effect was what he observed in bullish trading markets, and what ended up dramatically distorting them. It also explained why women were mostly immune to the euphoria, because they had only 10% of the testosterone of men. What struck him most, though, was that, for all the literature about financial instability, economics, psychology, game theory, no one had ever clinically looked at a trader who was caught up in a bubble.

Coates wrote a research proposal. He came back to Cambridge where he had done his first degree, and because of his background eventually gained access, with Herbert, to a major City bond-dealing floor in London. They tested the traders for two hormones in particular, testosterone and cortisol (the anxiety induced, depressive "stress hormone"), and mapped their levels over a period of weeks against the success or failure of trades, individual profit and loss. Coates had imagined the experiment to be a preliminary study but the correlations he found – for evidence of irrationality produced by the winner effect and its converse – was "an absolute dream". They not only discovered that a trader's morning testosterone level could be used to predict his day's profitability. They also found that a trader's cortisol rose with both the variance of his trading results and the volatility of the market. The results pointed to a further possibility: as volatility increased, the hormones seemed to shift risk preferences and even affect a trader's ability to engage in rational choice.

Though the sample was limited, and suitable caution was needed in claiming too much, the correlations suggested that over a certain peak, testosterone impaired the risk assessment of traders. "And cortisol," he suggests, "was in some ways even more interesting than testosterone. We thought cortisol would rise when traders lost money," Coates says, making individuals more than usually cautious, "but actually it was going up incredibly when they were faced with just uncertainty. The stress hormones were switching over to emergency states all the time. There was an optimal level but these stress hormones can linger for months. Then you get all sorts of really pathological behaviours. If you are constantly prepared for high tension it affects your brain, and it causes you to recall stressful memories and become exaggeratedly risk-averse and kind of helpless."

Unfortunately this particular study ended in June 2007, before the full effect of the crisis, but its implications account, Coates believes, for some of what he subsequently heard from the trading floor. "If cortisol goes beyond a certain point, then it may become very difficult for traders to assess any risk at all. These guys are not built to handle adversity that well. There is an observable condition called 'learned helplessness', which if you are submitting to great stress over a long period of time makes you give up suddenly. Lab animals develop it: you open the cage and they won't escape. Traders have it too. They just slump in their chairs. In the crisis there were classic arbitrage opportunities as the markets were falling. Free money. But traders would sit there staring at the numbers and not touching it."

Since then, Coates has partly been working on the other strand of his original hypothesis, looking at the brain chemistry of women working in the markets. Because of the small sample sizes he has to work with – there were only three women out of 250 traders on the floor he first tested – the detail of that is far from complete, and he is properly reluctant to draw conclusions. What he will go so far as to say, though, is this. "Central bankers, often brilliant people, spend their life trying to stop a bubble or prevent a crash, and they are spectacularly unsuccessful at it. And I think it is because, at the centre of the market, you have these guys either ripped on testosterone or overwhelmed by cortisol so that they become completely price insensitive." Coates wrote a couple of articles after that research was published, suggesting that, if the winner effect was right, it was possible that bubbles were an entirely young male phenomenon. And if that were the case, then the best way of preventing boom and bust was to have more women and more older men – less in thrall to hormones – in the markets. "We know that opinion diversity is crucial to stable markets. What no one talks about is endocrine diversity, a diversity of hormones. The billion-dollar question is how to achieve it."

To most experienced, male, investment bankers, of course, this sounds like fighting talk. An old friend of mine, who traded his Cambridge English degree for an extremely lucrative life as a bond dealer, offered this, when I presented Coates's evidence to him. "It would be nice to think that having more female traders on the floor would make for less volatility," he said, "but that's wishful thinking. Financial markets are now global, so while we in the west might decide not to chase trends or react instinctively to breaking news because there are mature mothering types in boardrooms and sitting on risk committees, the rest of the world will, and our banks would lose out." And that's not all. "Many of the women I know who have managed money or have put capital at risk for banks have tended to be even more aggressive with risk than their male counterparts, as if perhaps to compensate for their supposed diffidence. Fighting their way through a male-dominated environment to a position in which they can invest/punt/ risk-manage, many women develop an ultra-masculine persona so as to be thought of as ballsy…"

Just a cursory glance through some of the recent spate of books and blogs written by young women who have worked in the City and lived to tell the tale would certainly seem to support this observation. Melanie Berliet, who worked as one of the only female traders in Wall Street, set the tone in her confessional blog: "If anything," she observed, "my token status gave me an extra thrill. I enjoyed being called a 'fucking dullard' or being instructed, patronisingly, to 'remove head from ass', because my reaction – to grin rather than cry – impressed the guys. I loved their attention and the daily opportunities to prove that I fitted in. What separated me from my colleagues was physical: my 5ft 9in, 120lb frame, my long, blondish hair – and my vagina. I had two options with my boss: trade sexual banter or resist. Typically, I chose the former. Like most traders, my base salary wasn't terribly high—$75,000 at the start of my third year. The bonus was all, and getting the right number rested on one thing, as I saw it: my willingness to promote my boss's fantasy of fucking me…"

John Coates doesn't believe the caricature, or at least he believes that in the upper reaches of banks, things have moved on. "A lot of my former colleagues are running divisions, or whole banks," he says. "I don't buy the sexist macho argument. The big investment banks desperately want women traders. But when they interview women who are qualified, the women don't want to do it…"

Neuroeconomics also starts to provide the answers to some of the reasons for that. Muriel Niederle is a professor at Stanford University, looking at gender differences in risk decisions. Over a period of years Niederle has developed clear evidence for the theory that though in non-competitive situations women demonstrate an advantage over men in making investment decisions, they either shy away completely from making those decisions in intensely competitive environments, or they respond less well than men to competition with very short-term high intensity and results-driven focus. This pattern is set, Niederle proves, from a very young age (and no doubt has a good deal to do with the differential presence of troublesome testosterone). Joe Herbert told me at his lab in Cambridge: "What is clear is that there are neurological differences between the sexes. Women, in very general terms, are less competitive, and less concerned with the status of being successful. If you want to make women more present, you have to remember two things: the world they are coming into is a man-made world. The financial world. So, either they become surrogate men… or you change the world."

Ah, changing the world. In the wake of 2008, there was a good deal of talk about that heady idea. Much of this talk concerned the creation of more gender balance in the city. The Economist coined the phrase "Womenomics" and argued that excluding nearly 50% of talent from crucial positions in business and finance was not only discriminatory but caused serious harm to stability and growth. Iceland's banks brought in women to clear up the mess that men had left. A good deal was made of the fact that the extraordinary success of microfinance in the developing world was because 97% of the loans were granted to women (men were – biologically? culturally? – not to be trusted). Science, neuroeconomics, was harnessed to develop some of those themes. And then, well, nothing. The commissions and the select committees decided that a return to something like the status quo, with all its implicit risks and inequalities, was the only option.

Womenomics still persists in a few places, however. The 30% Club was an initiative set up last November by executive women, and some senior men in FTSE 100 companies and accountancy and legal practices, to increase the number of women in decision-making and boardroom positions to that figure. It goes a little further than Lord Davies's recent report on the subject. But 30% is not an arbitrary number; it is thought – by neuroeconomists again, and through observation – to be the minimum proportion of women at the top of an organisation required to begin to change the culture; below that number, women tend to behave "like surrogate men"; above it, the subtle differences produced by gender might begin to influence the way decisions are made. In Britain there is still a good way to go: only 5.5% of executive directors in FTSE 100 companies are women (yet evidence shows that companies with women leaders have a 35% higher return on equity, and companies with more than three women on their corporate board have an 80% higher return on equity). On city trading floors, the percentage remains, for some of the reasons outlined above, at around 3% or 4%. Testosterone rules.

The country that has attempted most radically to change this balance is Norway, where a Conservative minister imposed a quota of 40% female directors in every boardroom. Most of the data suggests the initiative has been a great success, both culturally and commercially (though some, male, commentators argue that the turnaround is better explained by the spike in oil prices).

It would be hard to find many people in the city, even among women, who would favour quotas, though that argument can be made. John Coates, wearing his dealmaker's hat, suggests a practical solution. "The question is not whether men are risk takers and women are risk-averse. It is more what kind of risk do they want to take? My hunch is that women don't like high-frequency trading, so what you have to do is change the accounting period over which they are judged."

He then gives me a potted description of how things remain: "Say you have two traders. One trader makes $20m a year for five years, of which she might typically pocket a couple of million a year herself. At the end of five years she has made the bank the best part of $90m. Another trader makes $100m a year for four years. They don't want that guy to go off to a hedge fund so they let him take home $20m a year. But then in the fifth year – because of the winner effect – he loses $500m. That is essentially what happened in the financial crash. The bank has lost $100m and the trader has gained $80m. If you were judging these things over a five-year period, then you can see which person you would hire."

But, of course, that would require a very different idea of markets, and of money, to the one that is currently desperately being defended and remade. It would certainly require a greater degree of "endocrinal diversity". Still, the next time you hear someone suggest that things are getting back to "normal" in the city, and that we should at all costs start believing in exponential growth again, at least you can look him in the eye and state that you think his hormones might be playing up.

Neuroeconomics: Six things that the science of decision-making reveals

■ If groups of young men are shown pornographic pictures of women and then asked to choose between safe and risky investments, compared with men shown non-pornographic pictures they choose far riskier portfolios.

■ Our brains are designed to seek out novelty, but too much information can overwhelm them; we are generally better at assessing risk when listening to Bach than with the chatter of TV news.

■ Men's brains tend to shut down after they have proposed a deal, waiting for the response. Scans show that women brains continue to be active, analysing whether they have done the right thing.

■ Humans are the only animals that can delay gratification, a function of the prefrontal cortex. However, the prefrontal cortex only matures after the age of 30, and later in men than women. Before that, we are more likely to seek immediate gratification.

■ Our brains reward social interaction with the release of a chemical called oxytocin. It makes us feel good when we follow the herd. Stock market bubbles are one likely result of this.

■ Our brains are wired for human oxytocin-mediated empathy (or HOME). We are biologically stimulated to love (or hate) what is most familiar to us. We are built to form attachments, to value what we own more than what we do not own. This fact skews the rationality of all our investment decisions.