There's a jaded view of humanity according to which we're all manipulating each other, all the time: a planet of confidence tricksters, differing mainly in how good we are at it. After all, we can't know the inside of anyone else's mind. So when deciding whom to trust, date, buy things from or vote for, we're forced to rely on "proxies" – outward indicators of expertise or trustworthiness. But there's a loophole: to triumph at this game, you don't need to be well-qualified or trustable. You just need to master the gestures, the expressions, the lingo. You might not be doing this consciously. You might even really be all you claim to be. But until they release a mind-reading app for Google Glass, how could anyone tell?

Perhaps the worst example of this is confidence. Stride into a meeting and just repeat your point the most insistently, and you've a good chance of winning the day, some studies suggest. New research, led by Bryan Bonner at the University of Utah, underlines the point. He asked small groups to answer factual questions, such as the driving time from Salt Lake City to Manhattan, or the weight of the world's heaviest person (1,400lb). The most confident people wielded outsized influence, regardless of accuracy. But when the groups were first asked consciously to consider reasons why certain members might truly know their stuff, or other reasons for preferring certain answers, the influence of the merely confident fell, while accurate members did better. Confidence is what Bonner calls a "messy proxy" for expertise. To short-circuit its effects in the context of meetings, reframe them as fact-finding exercises. Keep a running list of conclusions on a whiteboard, or do anything else to switch the focus from who is being convincing to what they're saying.

In a new book, Confidence, the psychologist Tomas Chamorro-Premuzic goes further, arguing that confidence is virtually worthless. On average, he reports, it's correlated with competence at about 0.30, which means the probability of the most confident person in the room also being the most competent is a paltry 15 percentage points better than chance. Partly, that's down to the famous "better-than-average bias": a majority of us believe we're better than most at all sorts of things – driving, keeping healthy – even though that can't be true. (Amusingly, we also think we're less guilty of self-serving biases than most.) "Feeling good," Chamorro-Premuzic writes, "does not increase the probability of being good." His more radical claim is that confidence doesn't even serve those who have it, and that there's no point trying to increase it. Humble-but-competent leaders, he shows, are both better-liked and more successful than braggarts. And "just low enough" confidence is a boon: it frees you from the dangers of overoptimism, stops you ignoring negative feedback and keeps you motivated to acquire skills.

This might be overstated. Studies such as Bonner's suggest confidence can be helpful in short-term, selfish ways. Then there's the question of assertiveness and gender bias: might office humility work best when you already benefit from other structural privileges, such as being male? Still, some general truths emerge: there are benefits to having less confidence, and we should stay sceptical of those who have it in buckets. Seeming as if you know what the hell you're doing, or feeling that way, are terrible proxies for whether you actually do.

oliver.burkeman@theguardian.com

Follow Oliver on Twitter