I can't really put my finger on the precise moment I embraced my own male mediocrity. It was more a dawning realisation.

Be it striding confidently onto stage to deliver a lecture with minimal preparation, having the sheer audacity to think I can pull off live TV or radio, or even sharing my thoughts with you here, its warm embrace is always with me.

In fact, I wear my mediocrity — or at least the confidence that comes with it — like armour. Most blokes do. It's almost a defining male characteristic.

And yet a growing body of research shows my masculine mettle is actually costing men, women and organisations dearly.

Thanks in large part to their brazen over-confidence, mediocre men are being promoted to senior roles — in science and other fields — ahead of vastly more qualified women, damaging productivity, research excellence and stunting everyone's performance as a result.

The good news, however, is that a reckoning is coming: evidence suggests the days of the mediocre man running the show are numbered.

Confidence as a proxy for competence

Journalists Katty Kay and Claire Shipman have coined this measurable effect — that is, the notion that men are more self-assured than women — the confidence gap.

Success, they write in their book, The Confidence Code, "correlates just as closely with confidence as it does with competence".

In other words, confidence has somehow become a proxy for competence as a basis for success.

Indeed, numerous studies support their thesis. A 2011 study on The Emergence of Male Leadership in Competitive Environments found that men have a natural tendency to overrate their past performance on maths tasks by 30 per cent.

Ernesto Reuben, one of the study authors, almost apologetically described this behaviour as honest overconfidence.

Men aren't deliberately trying to fool anyone, he says, they honestly believe their performance is better than it really is.

Similarly, a 2016 study of over 1,500 undergraduate biology students showed that being male was a prerequisite for "celebrity student" status.

The most renowned students in every class studied were male, and males tended to overestimate the performance of their male classmates.

So we don't just rate our own performance better, we're also biased to the performance of other men.

The problem with honest overconfidence

Importantly, this honest overconfidence has a huge influence on how we identify and promote leaders.

The same 2011 study showed that male overconfidence is a key reason why qualified women are not selected as leaders as often as they should be.

There's ample empirical evidence of the absence of women in senior leadership across many fields, and the negative impacts it has on decision-making, financial performance and other key outcomes.

For example, fewer large Australian companies are run by women than men named John — or Peter, or David.

If you needed any evidence of skewed leadership selection in science, this jaw-dropping picture from the White House speaks for itself.

Loading

Notions of equity aside, this is also problematic from the perspective of ensuring good leadership.

A meta-analysis of 45 studies of leadership styles showed that women tend to exhibit many of the character traits associated with effective leadership — such as effective communication, a tendency to empower subordinates, and creative problem solving — and are more likely to adopt effective leadership styles than men.

Are we picking the wrong kinds of leaders?

But our counterproductive tendency to exclude the most effective leaders raises a broader question. What if the traits on which we select leaders are not the ones that actually make them effective at the job?

This plays out in politics, when effective opposition leaders sometimes struggle to transition into successful prime ministers.

In sport, where strategy, motivation, courage, and vision are just as important for leadership as skill or talent, the most successful player on a team isn't always the most effective captain.

A confident gesture from a confident President, Donald Trump. ( AP: Patrick Semansky )

The satirical Peter Principle crystallises this problem, describing a situation where people get promoted in a hierarchy until they reach their level of incompetence.

Some have suggested women and minorities are exempt from this outcome because they don't get the chance to reach their level of incompetence, an effect Tom Schuller has dubbed the Paula Principle, whereby, he argues, "Most women work below their level of competence."

I see examples of this in science, where success in leadership — and increasingly career survival — tends to be defined by a very narrow set of demonstrably flawed metrics based on blunt measures of publication output and funding input.

This narrowly focused meritocracy, which prioritises productivity over research excellence, is damaging the fabric of modern science. It is also drives a research culture that sees a huge gender imbalance at senior levels.

Of course, research excellence is only one (albeit important) aspect of scientific leadership, and by definition brings with it other important qualities like tenacity and resilience.

But in placing so much emphasis on selecting the most productive people, are we unintentionally selecting against empathy, ethics, courage, and other traits of successful leaders?

Clearly, in science and other fields, performance metrics need to be broadened and aligned with desired leadership qualities to recognise and promote all aspects of good leadership.

So what's the best way to achieve it?

Tomas Chamorro-Premuzic of Columbia University has suggested that instead of focusing on the obstacles impeding women aspiring to leadership, we should instead be addressing the lack of obstacles for incompetent men.

The result of not doing so, he writes, "is a pathological system that rewards men for their incompetence while punishing women for their competence, to everybody's detriment."

A more direct approach that should shake the confidence of blokes everywhere (but probably won't) is the example of the crisis of the mediocre man.

The adoption of gender quotas in Swedish politics in 1994 — and the consequent displacement of mediocre male leaders — saw an overall increase in the competence of politicians.

Analysis of this effect identified a 'virtuous cycle of higher competence', breaking a trend for mediocre leaders to maintain power by surrounding themselves with mediocre followers.

And so maybe it's time we tested the merits of gender quotas — after all, it's hard to argue with an approach that has the potential to lift everyone's game by knocking out average blokes while simultaneously promoting competent women.

The Swedish experience suggests it might be more effective than waiting for cultural change by trying to shift deeply entrenched values.

In the end, I wonder if our armour of mediocre male confidence will prove to be a blind spot, and leave us woefully unprepared for the coming change.

Maybe that's the root of the testerical abuse that would be directed at a female colleague making the very same arguments?

Dr Darren Saunders is a Senior Lecturer in Medicine at the University of New South Wales.