‘Do you have a nemesis?” I asked the two men, friends of my flatmate, to whom I had just been introduced. We were at a gig, waiting for the band to come on. My flatmate wordlessly assumed an apologetic air, but one of his friends seized on my question like he thought I would never ask.

“Yes,” he said, without any hesitation. I could almost see the face of his nemesis in his eyes, reflected in the stage lights. “He’s a social worker.”

The other friend ummed, and ahhed, and cast his mind back to everyone he had ever met – including me, just now – before deciding that the closest he had had to a nemesis was a former colleague with whom he hadn’t got on very well, but even they had turned out all right in the end, so the answer was no, not really.

For some months, I had been asking the same question of everyone with whom I had come into contact. I was writing a short book about the unexpected upsides of having a nemesis, and had come to recognise these as representative responses. Either your nemesis leapt to mind with an immediacy that suggested they were never far from there in the first place – or you tried to conjure one out of politeness to appease your friend’s intense flatmate.

As is no doubt obvious from my question, I belong firmly in the first camp. As an ambitious, occasionally petty person, I have long understood that the power to motivate myself lies in an imagined competition with people who probably don’t think of me at all.

At the time of writing I have four nemeses. Three were formed at old jobs, over professional slights or disagreements that are intensely felt but tedious to explain; they themselves are oblivious. The fourth knows what she did.

Yes, it is juvenile, and petty, and revealing of other character traits that are collectively and euphemistically described as “complex”. Owning up to having nemeses certainly doesn’t win you any friends – but it’s a suspiciously anodyne person who really likes everyone they meet. In February this year, a Totaljobs survey of more than 7,000 British workers found that 62% said they had an “office enemy”, running the gamut from minor irritant to bully. Horrifyingly, nearly 300 people said that their nemesis clipped their toenails at work.

As I wrote at the time, I see my embrace of nemeses as fairly harmless. Picking a colleague to serve as your antagonist makes your life more like a sitcom, and less like wage slavery on the slow march towards death. And if they really are your antagonist, so much the better (so long as it is kept in check). In embracing the rivalry, you can unlock its potential benefits, and possibly, actually win.

In sport, harnessing the power of competition to improve performance is well known. A 2014 study of long-distance runners found that those who recalled a rival reported significantly higher motivation than those who did not. Competing against a rival led to significantly faster times: an improvement of nearly five seconds a kilometre. Runners could be expected to run a five-kilometre race roughly 25 seconds faster if they were competing against one of their rivals than if they weren’t.

Prof Brian Uzzi, an expert on leadership and social networks at the Kellogg School of Management at Northwestern University in Illinois, says there are parallels with work and creativity. “The good kind of nemesis can really drive motivation, help you reach further, let you see the possibility to do things that you couldn’t before – things that might have been impossible. And all of that, to me, means opportunity in life.”

Moreover, says Uzzi, a nemesis is somewhat inevitable once you reach a certain level in your career. “As you progress, your reputation builds and the probability of having someone that will be a nemesis also grows – either because your success is a threat to theirs or because, as you grow in authority, people become more critical of your behaviour.”

In June he published the results of his two-year study of day traders, which showed that not only did they gravitate towards having a balance of friends and enemies in their professional networks, but that their performance improved when they achieved it. People look to be friends with their friend’s friend and the enemy of their enemy, and to oppose their friend’s enemy and their enemy’s ally.

When those conditions are met, according to a framework called structural-balance theory reinforced by Uzzi’s study, a network is considered balanced – not, I note, when there are no enemies at all. It can be hard to push yourself in a vacuum; it is quite a bit easier to want to do better than the guy clipping his toenails.

A nemesis may even fulfil an important existential function. In 2010 researchers at the University of Kansas found that, “although superficially disagreeable”, seeing themselves as having powerful enemies allowed people to maintain a sense of personal control in the face of many more numerous and diffuse hazards.

The paper’s authors also found that people tended to attribute greater power to an enemy at times when the government, law enforcement and the wider social system appeared to be chaotic and disordered. When you might feel you are losing control over your environment or destiny, an enemy “can be effectively controlled, managed or understood”.

The key thing, says Uzzi, is to keep active conflict to a minimum. Friendly competition, not properly managed, can easily turn toxic, leaving the relationship beyond repair. “The bad kind of nemesis is something you really don’t want to develop, and if it does, you want to fix that or develop an exit strategy.” But the good kind is “someone who’s going to raise your game … They have to be people you want to compete with and do better than.”

The Totaljobs survey found that most people identified office nemeses of the same gender and the same level of seniority (or higher) as them, which is revealing of the tendency to benchmark ourselves against broadly equivalent individuals. Comparing yourself with others is of course fraught, and counter to the typical advice for achieving contentment – but it can be helpful in clarifying your own ambition.

My emotional response to my nemeses’ activities and achievements – usually never even voiced – is often most revealing about my own desires and insecurities. If their news makes me churn internally, I realise that it is something I want for myself, encouraging me to think about taking steps to make it happen. Equally, when it stirs in me disproportionate emotion, I have learned that it means there are probably more complicated factors at play.

Prof Barbara Louise Gray – a former director of the Center for Research in Conflict and Negotiation at Penn State University – says it is worth paying attention to people who push your buttons as they can teach you about yourself.

According to Jungian psychology, everyone has a “shadow self”, informed by our upbringing and cultural training, that we subconsciously strive to keep hidden; when we have a strong emotional response to someone, we might be projecting on them characteristics that we struggle to accept in ourselves.

Your nemesis, then, can be a prompt for useful introspection. “Try as you might to change someone else, as most people in marriages know, that doesn’t work very well,” says Gray. “So you have to turn the light around and shine it on yourself: ‘Why am I so teed off by this person? Why does this person rub me the wrong way all the time?’”

Identifying traits you are intolerant of in others may reveal those that you work to conceal in yourself, says Gray, and thereby present an opportunity for personal growth. Learning to accept our shadow selves as they are reflected in other people can be a humbling experience, one that can make us more confident and effective in our relationships and happier in ourselves.

The cliche that we are our own worst enemy may be true, but an external one can help that process of introspection along, says Gray. She adds that it is important to be attuned to your behaviour, and any patterns in the things that provoke you in others. “The key for an individual is to know when their buttons are pushed, and when it evokes a really strong reaction in them – that’s the time to say: ‘OK, I have a nemesis operating in here that I probably ought to meet.’”

Why Everyone Needs a Nemesis: Harnessing Pettiness For Greatness by Elle Hunt is available in audiobook and eBook, published by Hodder & Stoughton, £3.99