Impostor syndrome – the feeling that you're a fraud, and any day now you'll be exposed – is presumably even more common than surveys suggest: after all, it's not the kind of thing to which people like to admit. Indeed, it can be hard to tell when you've got it: those others might have a syndrome, your reasoning goes, but I'm genuinely out of my depth. It's a classic case of "comparing your insides with other people's outsides": you have access only to your own self-doubt, so you mistakenly conclude it's more justified than anyone else's. This is a strange kind of self-doubt, when you think about it, since it's premised on the idea that you're highly talented at something, namely deception. Still, it's no fun, and if new research is anything to go by, it might be harder to cure than anyone thought.

Two US sociologists, Jessica Collett and Jade Avelis, wanted to know why so many female academics opt for "downshifting": setting out towards a high-status tenured post, then switching to something less ambitious. Contrary to received wisdom, their survey of 460 doctoral students revealed that it wasn't to do with wanting a "family-friendly" lifestyle. Instead, impostorism was to blame. They also uncovered a nasty irony. It's long been known that impostorism afflicts more women than men – one of many reasons that institutions match younger women academics with high-ranking female mentors. But some survey responses suggested those mentors might make things worse, because students felt like impostors compared with them. "One said she suspected her mentor was secretly Superwoman," Science Careers magazine reported. "How could she ever live up to that example?"

This is only one of impostorism's frustrating ironies. Another is that true frauds and idiots rarely seem to experience it. ("The trouble with the world is that the stupid are cocksure and the intelligent are full of doubt," said Bertrand Russell.) Arguably the worst one, though, is that getting better at your job won't fix it. Achieve promotions, or win accolades, and you'll just have more cause to feel like a fake. Enhance your knowledge, and as you expand the perimeter of what you know, you'll be exposed to more and more of what you don't. Impostorism, as Pacific Standard magazine put it recently, "is, for many people, a natural symptom of gaining expertise". Move up the ranks and if your field's even vaguely meritocratic, you'll encounter more talented people to compare yourself negatively against. It never stops. "I have written 11 books, but each time I think, 'Uh-oh, they're going to find [me] out now,'" as some low-profile underachiever named Maya Angelou once said.

The only solution, many experts say, is for higher-ups to talk about their own insecurities much more. ("When people see those they respect struggling, or admitting they didn't know everything when they started, it makes it easier to have realistic opinions of their own work," says the Ada Initiative, which supports women in technology.) Few of us instinctively want to do that – least of all those with impostorism, or women in fields where stereotypes hold that they can't hack it. But if you're a higher-up yourself, you should try. One day, in an insecurity-friendly utopia, perhaps we'll look with scepticism on those remaining few who claim never to have felt as if they're faking it. Because they're probably faking that.

oliver.burkeman@theguardian.com

Follow Oliver on Twitter