Tomas Chamorro-Premuzic, a psychologist and chief talent scientist at ManpowerGroup, says we’re not picking leaders in the right way. While we should be promoting people based on their competence and potential, it’s often the incompetent, overconfident candidates — most of them men — who get ahead. Studies show that, by many measures, women are actually better equipped to become strong, successful managers. But the solution to getting more of them into the executive ranks isn’t quotas or other initiatives that mandate gender diversity. To improve leadership across the board, we need to focus on the metrics proven to enhance performance and set higher standards for everyone. Chamorro-Premuzic is also a professor of business psychology at University College London and Columbia University, and the author of the book Download this podcast

ALISON BEARD: Welcome to the HBR IdeaCast from Harvard Business Review. I’m Alison Beard.

We’d like to think that most organizations choose leaders for their competence, or even potential. But too often hiring managers, HR departments and even corporate boards are swayed by a candidate’s confidence or charisma instead. Those can be useful traits in business, but they’re not always the best predictors of performance.

Research shows that’s because many executives – and especially men – are overconfident. They believe they’re better employees and managers than they actually are and persuade others that they deserve to rise through the ranks. The flip side is that women who are equally competent, but not overconfident, are bypassed.

Tomas Chamorro-Premuzic, our guest today, is a psychologist and an organizational consultant who has studied this problem. His new book is provocatively titled Why Do So Many Incompetent Men Become Leaders?: (And How to Fix It). Tomas, thanks so much for being here.

TOMAS CHAMORRO-PREMUZIC: Thank you for having me.

ALISON BEARD: So, I have to start by asking, are there really so many incompetent men in leadership roles, especially in the corporate world? I’m sure there are many successful male CEOs and managers who would bristle at that idea.

TOMAS CHAMORRO-PREMUZIC: That’s true, but it is also true that the average experience that people have of their bosses, managers, and leaders is far from positive. You only need to Google “my boss is” to see what options Google turns out: crazy, abusive, unbearable, toxic and some other things that are just too rude to repeat here.

So, I think it’s almost unnecessary to look for scientific data quantifying the number of incompetent leaders or how dismal the baseline of managerial incompetence is. People see it in their everyday lives and as the saying goes, that’s one of the main reasons why people leave their jobs and companies, they “join jobs, but quit their bosses.” What’s interesting and depressing at the same time is that there is a very well-established science to selecting and developing good leaders, but they seem to be the exception rather than the norm.

ALISON BEARD: How do you know that this problem is men? Because there are female bosses too.

TOMAS CHAMORRO-PREMUZIC: That’s true. So, there’s a couple of important data points that we look for. The first is if you look at scientific studies that evaluate the key predictors of good leadership, women tend to outperform men. They lead more transformationally. They’re also better at the transactional elements or components of leadership such as monitoring others’ performance, giving them unbiased critical constructive feedback and empathizing with others. And they’re also less likely to lead in a passive or absentee way, which is the worse type of leadership. So, men are more likely to kind of commit that sin when they lead or manage.

On the other hand, we also know – this is just sheer volume of numbers – that on average men outnumber female leaders by 65 or 70 percent to 30 or 35 percent. And then all you have to do is how the majority of those leaders actually impacts their subordinates, teams and organizations.

We know that most people are disengaged. Most people dislike their bosses. That most people actually are considering self-employment or even entrepreneurial ventures because they have been traumatized by their boss.

So, everybody all the time says, “Oh most leaders are male, but they rarely actually look at how these leaders perform. And it is true that there are some very good male leaders, but the majority, the average male leader is actually doing pretty poorly.

ALISON BEARD: You talk a lot about men being overconfident and not as self-aware as women. What evidence do you have that that’s the case?

TOMAS CHAMORRO-PREMUZIC: Well, the first thing to remember is that people – so humans in general – are overconfident. We suffer from an optimistic bias. We tend to distort reality in our favor. On top of that we know that males are even more likely to suffer from this bias.

The most significant data point here is looking at 360-degree feedback, or multisource feedback evaluations, which compare how leaders rate themselves with how they are rated by others. And this data tells us two things. Firstly, that most managers see themselves in a more favorable way than they are seen by their subordinates and secondly, that this bias is particularly prominent or more pronounced in male than female leaders.

And we also incidentally know that managers perform better when their self-views are in sync or aligned with other people’s views on them, and that they even do better when they see themselves in a more critical way than others see them. And that’s one area where women again, have an advantage vis-à-vis men because they’re more likely to be self-critical even if it is a sort of harsh and perfectionistic self-criticism that is harsher than what others see them.

ALISON BEARD: Aren’t optimism, confidence, charisma – aren’t all of those attributes that we want in leaders?

TOMAS CHAMORRO-PREMUZIC: We want them and that’s one of the main problems really, that we want them, but we don’t necessarily need them. So, the traits or qualities that seduce us in leaders have very little to do with the qualities or attributes that are needed to be an effective leader.

If we want to upgrade the quality of our leaders, we need to stop falling for people who are overconfident, charismatic and even narcissistic, and select people on traits such as humility, integrity and competence, rather than confidence.

Too often the literature and discussions on leadership focus on the factors that help people rise to the top. Even if you look at much of the popular advice that is focused on helping people become leaders, it promotes and nurtures a narcissistic mindset: Love yourself, no matter what. Don’t worry about what people think of you. If you think you’re great, you are.

And all those tips and similar advices actually lead to a surplus of leaders who are unaware of their limitations, unjustifiably pleased with themselves and end up seeing leadership as an entitlement. So, they, when they get there they don’t have the necessary levels of self-control and integrity to inhibit risks or act in an ethical way. So, yeah, these things get you there, but they shouldn’t.

ALISON BEARD: And how do you know that these soft side skills translate not just into teams feeling better, but also better corporate performance?

TOMAS CHAMORRO-PREMUZIC: Because we correlate the degree of the scores or the degree of emotional intelligence, self-awareness, integrity. You find for example 2,000 leaders with different levels of these traits, of these qualities, and then examine how their teams perform. What the revenues, profits, productivity levels of their teams are; also how the teams rate them in 360’s or upward feedback; what the level of engagement, trust of their teams is. And you will see that about 30 percent of the variability in their teams’ performance across all these markers and KPI’s that I just mentioned, is directly attributable to the leader’s soft-skills.

ALISON BEARD: What are some specific tools that I as an individual hiring manager or my company can use to make sure that I’m adequately evaluating emotional intelligence and adequately evaluating humility?

TOMAS CHAMORRO-PREMUZIC: So, broadly you want to select leaders on three main factors. The first is technical expertise. In order to judge that it helps if you have technical expertise in that area yourself. But otherwise you consult with experts who can look at the leaders track record, their resume, their CV and see whether they are an expert or not. In most cases that’s the easiest part to evaluate.

Then you want to see whether they have soft skills or people skills relating to emotional intelligence. There are several and psychometric assessments, mostly personality assessments, based on what we call the Big Five model which actually provides you with a pretty predictive and reliable measure of whether somebody is more or less capable of empathy, integrity, altruism and so forth. And typically, these tools will predict how a leader behaves beyond interview ratings or other views of that person.

And then the third one, and one of the most obvious, and yet undervalued kind of markers or predictors of leadership effectiveness that we need to look for is intelligence. Whether the leader has high learning ability, including their curiosity, whether they’re able to adapt to complex cognitive intellectual problems and whether they actually can synthesize information and see patterns and make decisions better than their followers.

So, broadly you want leaders who are smart, who have technical expertise and who are rewarding to deal with, likable and able to connect with others and who have people skills. I think as leadership becomes more and more dependent on tasks that can be partly outsourced to computers, AI and automation, there will be an even bigger premium on the soft side of leadership which includes these EQ-like traits.

ALISON BEARD: So, I get that you’re saying it’s not about necessarily elevating all women. It’s more about just looking across the board at whether women or men are competent enough to be leaders. But how do we change that status quo if the incompetent people are already at the top?

TOMAS CHAMORRO-PREMUZIC: Well, the top is made up of not only of incompetent people, I would argue that the majority of leaders are incompetent or at least not competent because of how they affect their organization. But we also have not just enough leaders who are competent, but some that are trying to make a difference.

One of the main differences leaders can make if they are effective is to promote, develop, select and reward others who are talented. And the main point that I’m making which is often criticized on both sides of the debate, the feminist side and the, well if we don’t want to call it the chauvinistic side, at least the non-feminist side, is that the best gender diversity intervention is to focus on talent rather than on gender.

Because if we were truly able to look at talent, we would not only have a more balanced gender ratio at the top, but we would find that women actually outnumber men at the top. And that’s not a reflection of my not wanting to level or balance the gender ratio, but I think if we focus explicitly on gender, it often backfires because people think that it’s some kind of positive discrimination intervention that we’re orchestrating because women are not capable of getting to the top by themselves, because they don’t have the same leadership potential as men have. When in fact, men don’t have much leadership potential. Women have more and focusing on talent would not just level the number of women and men who are in charge, but also improve the quality of our leaders.

ALISON BEARD: So, you’re not a fan of quotas or affirmative action?

TOMAS CHAMORRO-PREMUZIC: Well, I think if my solution is too utopian, too idealistic and it’s maybe too extreme for businesses or organizations to implement, having quotas might be the second-best option. But we should be ready for the backlash that we have experienced so far and understand that if we select on gender, rather than talent we may not improve the quality of leaders. Whereas if we select on talent rather than gender, we will improve the quality of leaders while also increasing the number of female leaders. So, one solves both problems and the other solves one without the other and creates a backlash.

ALISON BEARD: Right, because there are outliers. They’re very competent, modest men and very overconfident, incompetent women.

TOMAS CHAMORRO-PREMUZIC: Correct. And this is an important point because the two other kind of side issues that we need to preempt or prepare for – and actually we’re seeing these occurring already today – is that under the current system, regime or with the current criteria that are in place, some organizations are very tempted to promote to power women who out-male males in masculinity.

And when they get there they seem to be showing even higher levels of aggressiveness, selfishness, competitiveness, greed and recklessness than men do. And that doesn’t do women a service and it also does a disservice to men who are potentially better leaders, but don’t display the traits – the masculine traits – that match our leadership archetypes.

Secondly, what I’m arguing for is not that we need to have more women leaders, but that we need to have more feminine leaders. And this point also sometimes is dismissed as some form of kind of benevolent sexism because I’m claiming not that women and men are equal in their personality, attitudes and values, but that actually women do display more feminine features and traits than males.

But what I’m saying is that we need more of that. After decades of selecting people on the basis of their kick-ass aggressiveness, greed, selfishness and determination, what we need is not for women to lean in like incompetent men, but for women and men to come who are capable of questioning themselves, capable of empathy, capable of some degree of self-doubt and who understand that they might not be the center of the universe. And all those traits are inherently feminine.

ALISON BEARD: Can men or any leader work to become less overconfident and more competent in all of the skills that you’re talking about?

TOMAS CHAMORRO-PREMUZIC: Absolutely. So, not just men, but also women; not just leaders, but also employees – humans have the capacity to change their behavior, to improve on any domain of competence. Obviously even if you accept the fact that things such as leadership talent or potential can be partly predicted from an early age, because people have different predispositions towards social skills, intelligence and influencing others.

Nobody would want a two-year-old running a company and that’s because we learn things as we develop, we grow older and we have experiences. And so, people can change. They can get better and at the same time we also know that it requires a lot of effort and that most people, they are less interested in changing than in having changed.

This is why the vast majority of New Year’s resolutions are broken even within the first two or three months and they typically are recycled from previous years and referred to things that we really want to change. So, imagine if I’m not able to quit smoking, drinking or start exercising more or eating less and that’s something that I really want to do, you can imagine putting things into perspective it’s going to be a lot harder for me to stop interrupting others, listening to others more, stop patronizing others, or mansplaining things, or behaving in a better way especially if my behaviors have been the product of years and years of inertia and I’m adjusted to them.

And by the way, any coaching intervention will be less effective if you are trying to change somebody who is older and more senior. Not just because things change less as we grow older, but when people have more status and they’re more important they’re less convinced that they actually need to change because their bad habits have mostly got them where they are and they’re less likely to listen.

ALISON BEARD: So, what are some specific, practical things that companies can do?

TOMAS CHAMORRO-PREMUZIC: I think definitely decrease or de-emphasize the importance that is given to job interview. Ensure that if job interviews are considered they are very structured, that no questions are asked without a predefined template to interpret the answers. That all the answers are interpreted in a way that links responses to a past benchmark or baseline that actually predicts performance. That scientific defensible assessments like metric assessments – mostly personality and intelligence – are included and rightly scored in a way that actually identifies the key traits or attributes that predict performance in a culture.

And that even when companies look for things such as values they don’t rely on their own perception of whether somebody’s values might match the culture, but they actually map the individual’s profile so their assessment results to their engagement data, or their climate surveys, so that they actually see whether they resemble a high-performing individual, or high-performing leader, or not.

Typically, there’s no big innovation in this field. I mean the things that work have been known for about 60 or 70 years. The quickest way an organization can get a sense of who they should hire is to get proper performance on their existing leaders and benchmark, identify the exact range of attributes that high performing leaders have, and then look for people who are similar.

ALISON BEARD: So, what if an organization does all this, becomes more disciplined about their hiring processes and promotions, and discovers that they’re just aren’t enough people suited for leadership in their ranks? Is the problem that we just don’t have enough truly competent people to manage large teams?

TOMAS CHAMORRO-PREMUZIC: Well, and it’s a great question and I think if an organization finds itself in that situation it would have made a lot of progress. That’s what you’re mentioning is almost like a First World problem. And because it would have the awareness that maybe people need to be skilled, reskilled or upskilled to develop expertise or self-skills to be better leaders, it would signal progress as well. It’s a little bit like if you don’t have a serious, reliable, self-diagnosis of anything, how are you going to improve? Is it better to be deluded and think that things are great when they’re not? I don’t think so.

At the same time, I think it is unlikely that any organization finds itself in that problem because this isn’t about finding the best leaders in history or people who are perfect leaders, but actually choosing the people who are best suited for leadership. And any large organization will have people who are better suited to be leaders. We also know, by the way, that they tend to, most organizations tend to go and shop for leadership from the external world. They go outside and then end up paying more money to have people that will take longer to adjust and on average perform worse than if they promote from within.

ALISON BEARD: So, obviously the title of this book is very controversial. You wrote a post for HBR.org that received many comments – some very critical. When my husband saw me reading the book, he rolled his eyes. So, how do you respond to that criticism?

TOMAS CHAMORRO-PREMUZIC: Well the reaction personally, I mean from what, I mean I am not paying attention here to the people that I would put in the kind of “friends and family” category because even if they don’t like what I’m saying they’re probably going to lie to me and be nice.

But I think if I read what the reaction has been online and from the press, media, et cetera – look, I think there’s clearly a strong correlation between gender and whether people liked it or not, where it’s been received much more positively by women than by men. That is to be expected.

I think at the same time there are a lot of men who like it because they might not realize that the book is, could be about themselves because of, even if they are in the incompetent category by definition for being incompetent, they’re probably not self-aware, so they might not realize that.

And I think the more interesting one is always as an author is the people who might initially disagree with you, but who you could persuade. I think sadly book readers are a bit like anything inside our filter bubble today. Most people read things because they agreed with you before they read it and they don’t change their minds. And most people see what they want to see and hear what they want to hear.

In that sense my hope is not that it actually can change people’s opinions, views, or at least force them to rethink the issue of gender and leadership.

ALISON BEARD: Well, I really enjoyed the book and learned a lot. Thank you so much for joining us today.

TOMAS CHAMORRO-PREMUZIC: Thank you for having me.

ALISON BEARD: That was Tomas Chamorro-Premuzic. He’s the author of the book, Why Do So Many Incompetent Men Become Leaders?: (And How to Fix It).

This episode was produced by Mary Dooe. We get technical help from Rob Eckhardt. Adam Buchholz is our audio product manager.

Thanks for listening to the HBR IdeaCast. I’m Alison Beard.