On 25 May 1832, John Constable was busy adding the final touches to his masterpiece, The Opening of Waterloo Bridge. One of England’s greatest 19th-century landscape artists, he had been working on the painting for more than 10 years and was finally set to reveal it to the world the next day, at the opening of the Royal Academy of Arts’ 64th annual exhibition. Next to his piece hung Helvoetsluys by J M W Turner, an artistic genius in his own right. Watching Constable’s last-minute efforts, Turner decided to add an extra brushstroke of his own: a red buoy floating on the water.

That single daub of red paint against a background of grey sky and sea was so arresting that visitors couldn’t take their eyes off it, certainly not to look at Constable’s painting. It was yet another landmark in the bitter rivalry between the two artists. A year earlier, Constable had used his position in an exhibition committee to have a Turner painting taken down and hung in a side room, replacing it with a painting of his own.

Turner and Constable are not alone in the pantheon of epic rivalries between creative giants. Isaac Newton and Gottfried Leibniz, two of the most brilliant mathematicians and thinkers of the 17th century, laid claim to the development of calculus, the mathematical study of change. Thomas Edison and Nikola Tesla both invented electrical systems in the 1880s. Steve Jobs and Bill Gates went head-to-head as pioneers of the computer age. If you Google almost any famous figure along with ‘rivalry’, you’ll find some interesting results.

Think of rivalry as a type of über competition driven by mutual obsession, with the rivals propelling each other to spiralling achievement, and investing more mental and emotional resources in each other than circumstances would ever dictate on their own. In 2014, across two sets of studies involving undergraduate students and runners, Gavin Kilduff, a psychologist at New York University, found that rivals tend to be the same age, gender and social status. True rivals know each other and, indeed, often have long, enmeshed histories. Rivals are, by definition, evenly matched – but the higher the level of their attainment, the more they propel each other on.

Rivalry can be double-edged: it motivates not just heightened accomplishment but, sometimes, unethical behaviour such as lying, cheating or stealing. In a series of studies, Kilduff found that those primed for rivalry were more open to Machiavellian acts and more likely to exaggerate positive results in a cognitive task. Rivalry could account for scandals and malfeasance at the highest levels of industry, and might even explain some of the risky behaviour behind the economic collapses of the recent past.

The social drama of rivalry, with its hostility and aggression, masks a deeper subconscious dynamic. We might think of our nemesis as the polar opposite of ourselves, but as Kilduff’s research suggests, our rivals are much more like us than we dare admit. While this might seem counter-intuitive, it follows that rivalry can actually be good for us: acknowledging that our rivals share our most essential traits, good and bad, can help us up our game and gain some of the insight we need for greater success.

Orson Welles summed up this idea in his movie The Third Man (1949): ‘In Italy, for 30 years under the Borgias, they had warfare, terror, murder and bloodshed – but they produced Michelangelo, Leonardo da Vinci and the Renaissance. In Switzerland, they had brotherly love – they had 500 years of democracy and peace, and what did that produce? The cuckoo clock.’

Although this might seem cynical, art historians tend to agree: the birth of the Renaissance is attributed to the rivalry between two artists over who would design the bronze doors of the Florence Baptistery. In 1401, the cloth importers’ guild declared a competition to design a set of doors for this building – one of the oldest in Florence, where the poet Dante and members of the prominent Medici family were baptised. Lorenzo Ghiberti, aged 23, won the commission, ousting his more established opponent, Filippo Brunelleschi. Ghiberti’s victorious design ushered in a new style of art, more naturalistic and with greater emphasis on perspective and idealisation of the subject. While it took him another 21 years to complete the assignment, the episode began a competitive frenzy that became a trademark of the Renaissance.

In fact, the most important artistic achievements of the Renaissance occurred in the small area between Rome, Florence and Venice, home to just a couple of hundred thousand people at the time. One of the largest cathedral domes in the Christian world, the Duomo in Florence; the realistic representation of the human body; and linear perspective in painting all came into existence thanks to the rivalry between Renaissance giants such as Brunelleschi (1377-1446), Da Vinci (1452-1519), Michelangelo (1475-1564), and Raphael (1483-1520).

Raphael’s results were applauded by all – all, that is, except Michelangelo

According to their contemporary, the art historian Giorgio Vasari, rivalry was common among elite artists of the period. Renaissance Rome was home to any skilled artist aspiring to work for the Vatican – the biggest and almost single employer of the time. The natural intensity of competition in such a restricted setting yielded works of art that still hang in the world’s elite museums. The practice of exhibiting paintings by different artists side by side in order to compare technique and style naturally heightened the pressure on each artist. Raphael achieved new heights in his work when he designed 10 tapestries, commissioned by Pope Leo X to hang in the Sistine Chapel under Michelangelo’s divine ceiling. The results were applauded by all – all, that is, except Michelangelo.

That should come as no surprise. The famous sculptor and painter was also renowned for his temper. When the handsome young Raphael first arrived on the Rome scene and was quickly commissioned by Pope Julius II, Michelangelo labelled him a bitter rival and proceeded to repeatedly accuse him of plagiarism. At one point, Michelangelo worked on his ceiling masterpiece behind a partition in order to hide it from Raphael. The latter, no shrinking wallflower himself, managed to arrange a view of it and later, in his fresco The School of Athens, incorporated a seated figure taken straight from Michelangelo’s work. Thanks to these machinations, the rivalry between the two giants became one of the most famous in the annals of art.

It wasn’t until the establishment of science societies in the late 16th century that major scientific rivalries reared their head. Perhaps the most notable early outbreak was the fierce war between Newton and Leibniz, each of whom claimed to be the first to invent calculus – today widely considered to have been developed independently by each of them. The feud caused such a rift between the English and European mathematics communities that, for more than a century, almost no scientific knowledge was exchanged between them.

In the early 18th century, Newton balked at nothing in his campaign for priority over the invention of calculus: in 1712, the Royal Society of London published a document granting Newton ownership of the invention and discrediting Leibniz. The paper, however, should be taken with an exceptionally large grain of salt, since Newton, who was president of the society at the time, personally appointed all the committee members and even wrote large parts of the document himself. The two colossi of mathematics never met in person and it is not clear that Leibniz was ever exposed to Newton’s work. One can only imagine how a productive exchange of their ideas, disputed over a public platform, could have enhanced the introduction of calculus and the scientific developments that followed.

‘The aim of argument, or of discussion, should not be victory but progress,’ said the French 19th-century essayist Joseph Joubert. Once the new societies and their publications made information more accessible, rivalry between scientists, research institutes and even states began to drive new discoveries. Journalistic interest in the drama enabled more public exposure to science. In one notable case, the dispute between Thomas Huxley and Richard Owen, two of the leading biologists in 19th‑century Britain, shined an important spotlight on Charles Darwin’s theory of evolution, little‑known to the general public at the time.

One of the stormiest scientific rivalries of recent years raged between the paleoanthropologists Donald Johanson and Richard Leakey over the discovery of some of the oldest fossils from pre‑human species. Johanson discovered the skeleton ‘Lucy’, thought to be around 3.2 million years old, while Leakey discovered ‘the Turkana boy’, believed to be more than 1.5 million years younger than Lucy – each cited by its discoverer as the proverbial ‘missing link’ between humans and apes. Their public falling‑out was remarkable even for science. The researchers had refused to share a platform since 1981, but finally met on stage in May 2011, explaining their positions and giving interviews at a highly publicised event at the American Museum of Natural History in New York, where their vocal discord first erupted 30 years before.

Thirty years later, older and wiser, they expressed a genuine desire to integrate their findings with many of the dramatic discoveries that took place since their feud first erupted. It also became clear how those two men complemented each other: while Leakey generated an abundance of fossils, it was Johanson who was better at interpreting his findings.

The qualities in our rival that arouse our hostility are exactly the ones we prefer to repress in ourselves: weakness, anxiety, greed, aggression, lust

Entire societies and social groups can rival each other, too. ‘Cruel is the snow that sweeps Glencoe and covers the graves o’Donald,’ begins a ballad by Jim McLean about one of the most brutal events in the bloody history of Scotland. The massacre of Glencoe took place one early morning in February 1692, conceived by the British authorities as a punishment for the failing of the MacDonald clan of Glencoe to swear allegiance to William and Mary, the new co-regents over England, Scotland and Ireland. Thirty-eight men were killed by British soldiers who lived among them, and 40 women and children were killed when their homes were torched or died later from starvation. The mass murder was presented to the MacDonald clan as a revenge spree by the Campbell clan – a claim that fell on willing ears given the long history of clashes between the two groups. This bitter tribal rivalry, which began in the 14th century, continues in different forms to this day.

The rivalry between Sunnis and Shiites started after the prophet Muhammad died in 632, when both groups vied to succeed him as leader of the Muslim community. The ancient divide between these major denominations of Islam is a major source of unrest in the Middle East to this day. In other cases, tribes have fallen out much more recently: the divide between Hutu and Tutsi that resulted in the Rwandan genocide of the 1990s was less than 100 years old and partially created under colonial rule in the 1920s.

Competitive sport is rife with rivalry. Glaswegian football fans can back either the Rangers or Celtic, a late sublimation of the warring Scottish clans mentioned above, and there is an endless array of favourites from boxers to racing car drivers. Nothing can match the fervour that caused El Salvador to declare war on Honduras following the ‘Football War’ of 1969. While the true causes were economic, emotions first flared when fans of both teams clashed violently at a FIFA World Cup qualifier. The third, decisive game was held in Mexico City on 26 June 1969. El Salvador won 3‑2 after extra time. The same day, El Salvador dissolved all diplomatic ties with Honduras and the two countries were at war less than three weeks later.

When people get so worked up over a rival, isn’t something deeper going on? The fiercest rivals are often firstborn, says the American science historian Frank Sulloway in Born to Rebel: Birth Order, Family Dynamics, and Creative Lives (1996). Sulloway cites evolution as the basis for his claim that the finite resource of parental attention is a source of sibling rivalry. Firstborn children use their size and strength advantage to uphold their status, and are more likely to compete over physical or intellectual territory. Younger siblings tend to undermine the status quo and develop a rebellious personality. In a particularly meticulous study, Sulloway analysed the biographies of almost 4,000 researchers and scientists from the 18th and 19th centuries, including 83 pairs of siblings. He found a younger sibling was 7.3 times more likely than a firstborn to support an innovative theory. But a firstborn’s chances of engaging in rivalry were 3.2 times greater than those of younger siblings. You guessed it: Newton and Leibniz were the eldest sons in their families. Turner was an older brother, and Constable’s older brother was intellectually disabled, so the onus of success fell on him as if he were the eldest.

The prototype, of course, is Cain, who committed the first envy-driven murder in the Bible. A comprehensive study of sibling relations by the Dublin Institute of Technology in 2012 found that, although most people support their siblings, some exhibit signs of rivalry verging on outright hostility. Given our achievement-oriented culture, it should come as no surprise that a third of siblings report rivalry and emotional distance, with 15 per cent not even talking to each other. Sibling rivalry is greater when there is a small age gap, no gender difference, or when one sibling is intellectually gifted.

An especially profound exploration of rivalry comes from the psychologist Carl Jung, the founder of analytical psychology, who said that we have more in common with our rivals than we would like to admit. The qualities in our rival that arouse our hostility are exactly the ones we prefer to repress in ourselves: weakness, anxiety, greed, aggression, lust and rudeness are a few common examples. Jung called this panoply of traits ‘the shadow’.

In Freudian theory, we defend ourselves from urges we don’t want to acknowledge by denying their existence and ‘projecting’ them onto others. This makes us attribute qualities, intentions and desires to others that are actually our own. According to Jung, such urges are buried deep within the ‘shadow’ part of our mind. The less cognisant we are of the shadow inside us, the darker and denser it becomes.

If we project qualities from our own ‘shadow’ onto a potential rival, we can easily find ourselves spiralling into a heated conflict when our rival behaves like us. Even worse, without our rival, we might feel that we lack an independent existence and wallow in the darkness of our ‘shadow’ with no one to project it upon.

Look to your shadow to identify your lifelong rival – the source of your creativity and, perhaps, your rage

Jung’s notion of the ‘shadow’ adds dimension to the relationship between our rival and ourself. Jung calls that conscious self the ‘ego’. The ‘shadow’, meanwhile, is the dark part of our personality hiding behind our social mask – the ‘persona’. As soon as we are old enough to comprehend the cultural mores around us, we select those parts of the self that are socially acceptable and classify them as ‘ego’, while repressing socially undesirable traits – transporting them to the shadow, where they continue to exist unbeknown to us. Jung claimed that the ‘ego’ and the ‘shadow’ have the same origin and maintain a perfect balance: the clearer the conscious part of our personality, the more well-defined our ‘shadow’ self. The opposite is true, too: a ‘shadow’ that is not contained can wreak mental havoc.

Look to your shadow to identify your lifelong rival – the source of your creativity and, perhaps, your rage. If you have a particularly strong negative response to someone and think he or she is a real jerk, think again. That might be a reflection of your ‘shadow’ in action.

Edward Bennett, a friend of Jung’s, elaborates on this in What Jung Really Said (1967). He describes the phenomenon as a gut reaction that projects the source of our emotion onto another, usually by means of sharp criticism or outright accusation. When we hate someone, we hate something in them that is part of us; if we do not subconsciously recognise our own traits in the other person, then we will not be too bothered by them.

Projecting our shadow onto someone else is always easier than acknowledging and containing it. When someone else projects their shadow onto us, it encourages us to project our shadow back onto them, unless we are aware of what is happening. But withstanding that dynamic takes an unusual level of self-awareness – even for brilliant minds. And why would we want to resist? The shadow is the seat of creativity, as far as Jung was concerned. In Owning Your Own Shadow (1991), Robert Johnson, a popular American Jungian author and analyst, explains why rivalries tend to erupt between especially creative people: ‘Narrow creativity always brings a narrow shadow with it, while broader talents call up a greater portion of the dark.’ The more creative you are, the greater your chances for rivalry. And the fiercer your rivalry – the higher your chances of remarkable progress.