It’s a movie classic. The lovers are out for a walk when a villain dashes out of his house and starts fighting the man. The woman takes refuge in the house; having seen off his rival, the villain re-enters and chases after her. Yet the hero returns, pulling open the door so that the heroine can escape. The villain chases the lovers, until they finally flee, and he smashes his own home apart in fury.

Who are these characters? None of them ever made another movie, and you won’t find them in any directories of famous actors. They are, in order of appearance, a large triangle (villain), a small triangle (hero), and a circle (heroine). The animated film was made in 1944 by the psychologists Fritz Heider and Marianne Simmel of Smith College in Massachusetts, whose paper ‘An Experimental Study of Apparent Behaviour’ is a milestone in understanding the human impulse to construct narratives.

At one level, their movie is just a series of geometric shapes moving around on a white background. It appears to lack any formal elements of story at all. Yet study groups (of undergraduate women) who saw the film in 1944 were remarkably consistent in their judgment of what it was ‘about’. Thirty-five out of 36 decided that the big triangle was a mean, irritable bully, and half identified the small triangle as valiant and spirited.

That’s a striking result: near unanimity on the emotional journey of a bunch of shapes. Then again, how surprising were these findings? Abstract animation existed as early as the 1920s, and experimental animators such as the Hungarian Jules Engel had already shown in sequences such as the Mushroom Dance in Walt Disney’s Fantasia (1940) that very little visual information is needed to create characters and story. So perhaps research was just catching up with what the empiricism of art had already discovered.

Maybe so. But the question remains: where does this narrative impulse come from? Why and how do we construct these stories? What kinds of stories do we impose on events? And should we?

As ancient myths attest, the question ‘What is a story?’ was already complicated long before postmodernism. Nevertheless, there are a few general characteristics. Narrative traditionally concerns itself with historically coherent events, even if they are not told in strictly chronological order: there are causative reasons why one thing leads to another, and characters respond in ways we recognise. A story also tends to have an arc of tension: relatively low at the start and the end, higher at some point in between. It is, in the clichéd yet apt phrase, an emotional journey, and it is navigated with a sense of purpose.

Making stories around geometrical shapes is one thing, but we do it with instrumental music, too. When Richard Wagner listened to a passage of Beethoven’s Third Symphony (Eroica), he found that it conjured the image of ‘the loveable glad man who paces hale and hearty through the fields of Nature, looks laughingly across the meadows, and winds his merry hunting-horn from woodland heights’. These days, only the ominous foreshadowing of Nazi Völkisch sentiment makes this little vignette seem anything other than risible. In Wagner’s time it was unremarkable to construct a narrative around music. It was, however, a relatively new habit: for the contemporaries of Bach and Mozart, it would have seemed absurd to tell so explicit a story about an instrumental piece.

Where do these narratives come from? In some cases they might be nothing more than projections of our own knowledge or ideas about the composer. One of the most notorious examples was the assertion by the American musicologist Susan McClary in 1987 that Beethoven’s Ninth is a rapist’s murderous fantasy; but her interpretation of Tchaikovsky’s Fourth Symphony as the story of a homosexual man suffering under the weight of his father’s conventional expectations, who becomes trapped in a manipulative relationship with a woman, is just as dubious. This, after all, seems to have been precisely the situation of Tchaikovsky himself.

Yet it’s not surprising that music seems to elicit narrative interpretations. Both music and story unfold over time in non-random events that relate to one another. The formal musical structures that became established in the 18th and 19th centuries seem to mimic the dynamics of archetypal stories: we are introduced briskly to the characters (themes) before a slow, lyrical interlude that is followed by something wild and exciting (the scherzo), leading eventually to a rousing climax and conclusion. The British musicologist Donald Tovey suggested in 1944 that the sonata form consists of two tragedies separated by a vision. We feel after listening to almost any extended composition, whether it’s Mozart’s Jupiter symphony or Pink Floyd’s Shine On You Crazy Diamond, that we’ve been taken on a journey, often an emotional rollercoaster, before – like Bilbo Baggins or Peer Gynt – arriving back home.

This might be precisely what makes listening to these compositions a coherent experience: we can construe it as a story. As the musicologist Fred Maus at the University of Virginia put it in 1990:

Musical events can be regarded as characters, or as gestures, assertions, responses, resolutions, goal-directed motions, references, and so on. Once they are so regarded, it is easy to regard successions of musical events as forming something like a story, in which these characters and actions go together to form something like a plot.

But because there are not really characters and plot (opera aside), the stories proposed for specific pieces of music are usually both vague and banal. Schumann’s Second Symphony is, we’re told, a story of suffering leading to healing and redemption. Well, I can read Cinderella for that. Shostakovich’s Tenth Symphony is ‘a progression from dark to light or struggle to victory’. You don’t say? So is Star Wars. Narrative readings here might help us convey the sonic trajectory, but they don’t in themselves deliver much sustenance.

For this and other reasons, some music theorists deny that narrative adds anything to our understanding. Lawrence Kramer of Fordham University in New York quotes Byron’s view that narrative offers the illusion of truth while distorting form and aesthetics to fit a preconception imposed by everyday experience. For Jean-Jacques Nattiez of the University of Montreal, imposing narrative on music is ‘superfluous metaphor’.

A complex musical piece might wander far while observing only small movements from one chordal centre to another nearby; by the end, we have truly ‘gone somewhere’

Yet there’s a more profound way of understanding the narrative illusion of music. When we listen, we really do go on a journey: through the abstract space of musical harmony.

Music theorists recognise that the various major and minor chords of Western tonal music are related to one another. Some are closer and some are further away, in the simple sense that the respective scales share more or fewer notes in common. C major is close to G major, and many nursery rhymes simply skip to and fro between them, but it’s also close to its so-called relative minor, A minor. We can venture further: through G major to D major, say, or via C minor to E flat major. These chord ‘progressions’ sound relatively smooth and logical in a way that a sudden big jump in harmonic space – say from C major to G flat major – doesn’t. A harmonically complex musical piece might wander far afield in this space while all along observing only small movements from one chordal centre to another nearby. By the end, we feel that we have truly ‘gone somewhere’.

And in a sense we have. One of the most striking discoveries in the field of musical neuroscience is that we have a mental map of harmonic space literally imprinted on our brain. Using magnetic resonance imaging to watch which parts of the brain become active during music listening, Petr Janata and colleagues at the University of California, Davis found in 2002 that an area of the prefrontal cortex – a part of the brain that’s a junction for pitch, emotion and memory processing – holds a spatial map of harmonic space in which different groups of neurons are assigned to each chord. Their test subjects were musically trained, but it seems likely that these harmonic relationships will be intuited more strongly with increasing musical exposure. They aren’t hardwired: as if called up from some hard drive in the long-term memory, the map seems to be elicited each time we hear music. But it explains why music seems like a narrative, full of surprises and recollections, that takes us there and back again.

These musical ‘stories’ might be weak on plot and characterisation, but they do have identifiable arcs and events. Prokofiev leaves us asking: ‘How did I get here?’ Schoenberg might confound us from the start in the manner of the experimental modernist novel. Schubert (forgive me, but it’s my experience) can elicit a: ‘Yeah, yeah, heard it before.’ All of which makes one wonder how many different stories there are to tell.

To answer that, we might turn to that genius and scientist-manqué, Kurt Vonnegut. Vonnegut’s (rejected) masters thesis in anthropology at the University of Chicago in 1947 theorised that all literary narratives can be represented as a graph of the hero’s fortunes against time on a happy-sad axis. So there’s the Boy‑Meets‑Girl arc (up-down-up: Jane Eyre), the From‑Bad‑To‑Worse plot (down and down: Metamorphosis), and so on. This, Vonnegut claimed with characteristic deadpan, reveals the New Testament to have basically the same arc as Cinderella.

While the idea of filing Goldilocks with Gone With the Wind, or Nicholas Nickelby with Shrek, doesn’t hold much promise for literary criticism, there might be something to be said for the approach more generally. We do, after all, need some explanation of how abstract human activities nevertheless acquire significance and cohesion from the narratives they elicit.

That has been demonstrated recently for sporting competitions. If Maus is right that ‘whenever there is an interesting action, there are stories that can be told about it’, then narrative and sport are made for each other. The games we remember are often not those that surround isolated events (what happened in the tennis match when Roger Federer smashed his racquet?) but those with a memorable storyline: victory snatched from the jaws of defeat (Jimmy Connors clawing his way back against Mikael Pernfors at Wimbledon in 1987), the triumph of experience over youth (Connors losing to Arthur Ashe in 1975), the neck‑and‑neck tussle (Rafael Nadal and Federer, 2008 final).

The mathematician and social scientist Peter Sheridan Dodds of the University of Vermont and colleagues claim to identify distinctive motifs in the ‘unfolding narratives’ of Australian rules football. This rather niche sport resembles Gaelic football in having an oval ball that is kicked or handled through sets of posts at either end of the pitch. Because these are high-scoring games, the pattern of point-scoring during a match offers a dynamic ebb and flow analogous to Vonnegut’s storygraphs. The extent to which they encode ‘plots’ can be seen by comparison with random sequences of scoring – a game in which the narrative is just one damned thing after another has no meaningful trajectory.

The potential for a sporting game to create a narrative is surely part of its appeal

By looking at more than 1,300 such game histories over the 2008-14 seasons, Dodds and colleagues could pick out different storylines – from blowouts (runaway victories) to comebacks and nailbiters. Unsurprisingly, they don’t split neatly into the various categories. The number of types of game is therefore rather arbitrary, depending on how many fine grades of blowout and so on you want to distinguish. But there are nevertheless clearly distinct ‘motifs’, corresponding to narratives such as ‘losing early on, coming back, then pulling away’, or ‘one team taking an early lead and then holding on for the rest of the game’.

The potential for a sporting game to create a narrative is surely part of its appeal. We all like our team to win, but how much more glorious if they do that and offer a satisfying story: where one thing leads to another to create an arc with tension and resolution. What Maus said about music applies equally to sport: it is easy to regard sporting events as a ‘narrative’, in which characters and actions form a ‘plot’. If Manchester United in their heyday won every game with a blowout, even fans might have got bored; that they so often did with a last-minute goal made them exciting (to fans) – or infuriating (to rival fans). Are we, perhaps, drawn not so much to the teams that win more often but to those that offer the best stories?

The current fad for statistical analysis in sport, described for football in The Numbers Game (2013) by Chris Anderson and David Sally, suggests that the analysis has more than academic interest. It also shows that we have long been addicted to stories – they just might have been the wrong ones. Managers and coaches might be tempted to construct a narrative (‘that referee’s decision made all the difference’, ‘we conceded too many corner kicks’) which the statistics undermine. And therein lies the whole seductive allure of storytelling. When the plotline isn’t self-evident from the outset, we might simply invent the story we like – and thereby fall prey to delusions of comprehension.

On the football pitch, who cares? But there is far more at stake in the human predilection for narrative. Take history, where storytelling has always been a primary tool. These days, even schoolchildren are discouraged from simplistic narratives that explain the First World War as the outcome of a political assassination. Even so, history would be barely worthy of the name if it didn’t seek some kind of causality – if it contented itself with a blow-by-blow account of ‘what happened’ without worrying about why. That’s what storytelling depends on: narrative is about consequences. Things happen for a reason, whether that is the caprice of gods or the domino effect of the farce. It is causality that lies at the root of our interpretation of Heider and Simmel’s abstract animation: the bad triangle has beaten the good one, the circle moves to escape the bad triangle.

There is no better illustration of the pitfalls of a good story than economics. The shortcomings of this discipline as a predictive or even explanatory science have been well advertised, but one rarely acknowledged reason for those failures is the penchant of analysts to mistake narrative for causal understanding. How often do we see charts of fluctuations in, say, share prices or economic indices decorated with arrows indicating the political or financial events that ‘caused’ them? Even when these are not transparently spurious, with spikes that are evidently mere noise commandeered to serve a story, the narrative is often an empty posture of omniscience.

Some economic movements are of course predictable in the broadest terms – no one expects markets to look the same if the Eurozone collapses – but there might be no logical reason to expect some events to have a big effect and not others. Even so major an event as the 2008 crash lacks a consensus ‘explanation’ with anything like the force of the explanation for, say, the explosion of the Space Shuttle Challenger in 1986; we can agree that subprime mortgages played a part, but if we knew exactly how then the effects would have been predictable.

This economic storytelling isn’t just the art of charlatans and false prophets. It is endemic to the discipline. The standard theory of market fluctuations attributes them to ‘exogenous’ causes, meaning that they are induced in a potentially stable market by external political events. This makes it permissible to tell a story about what ‘caused’ the market to behave a certain way. The truth, however, is that the real consequences of such disturbances are generally unpredictable because the internal dynamics of the economic system are inscrutable: everything depends on feedbacks that might or might not amplify a particular modest disturbance into a major disruption. History typically works that way too, as does traffic flow, crowd behaviour and many other human social systems. The corollary is that narratives do more to mask true understanding with a superficial veneer of authority than they do to offer a useful explanatory framework.

In such cases, it’s not necessarily that we have the wrong narrative. Rather, we might need to rethink what a narrative can and should involve. Economic markets are a classic example of what scientists call a complex system, meaning one in which many agents interact with one another simultaneously, creating the possibility for knock-on effects and feedbacks that might amplify tiny causes into big effects. In such a situation, causation doesn’t have a simple meaning: do we ‘blame’ the random fluctuation or the incipient instability of the whole system? When traffic is such that any tiny, chance disturbance will trigger a jam, does it really matter what that disturbance was? Seismologists can tell if a big earthquake might occur in a particular location – but they know better than to talk as if that local rupture will ‘cause’ the quake.

In other words, it might be quite wrong to tell such a story in terms of this small-scale action leading to that large-scale effect. And in fact the aptness (or otherwise) of such a narrative can now be quantified. In 2013, the psychiatrist Giulio Tononi and colleagues at the University of Wisconsin-Madison described a very general abstract model of a complex system in which the ‘micro’ behaviour underpinning the ‘macro’ properties could be completely specified, and yet causation didn’t flow from the small to the large. Think again of the traffic analogy: even if we know the detailed trajectory of each car, it’s meaningless to blame a jam on this or that vehicle.

The virtue of the model devised by Tononi and colleagues is that they could put numbers to the causative influences. They defined a quantity called the ‘effective information’ for each size scale (for individual cars, let’s say, groups of five, 50, 500…). This measured the degree to which the system’s states at that scale could be considered to ‘cause’ the behaviour at the largest scale (in the traffic analogy, a massive jam of the whole highway). They found that in certain cases this effective information was greater at the larger scales than at the smaller: the big picture was, in a formal sense, where the true ‘cause’ was located. Tononi and colleagues suggested that the brain might be such a complex system, too – in which case the apparent belief, evinced in some current international projects, that we will understand it by mapping and computer-modelling it down to the last neuron and synapse is misconceived.

These considerations surely apply in several other areas of science and technology, where they should commend caution about the kinds of narrative we spin. Some of the recent debate about the role of genes in biological behaviour probably stems from this ambiguity. Because of the complex networks of gene interactions, it can be hard to assign high-level properties (personality traits or intelligence, say) to the activity of specific genes, even if the inheritability of the trait suggests a strong genetic influence. We like to tell the story in terms of a gene ‘for’ the trait, but work such as Tononi’s might imply that this is wrong – not simply because it’s hard to trace the trait back to the genes ‘responsible’ but because that’s simply the wrong way to look at causality in this case.

We don’t know which story to tell – but we know that we need a story of some kind

What is leading us astray? I would argue that it is our instinct for story, albeit in a very abstract form. The putative ‘gay gene’ or ‘criminality gene’ or whatever becomes a character whose motive is to make the organism gay or criminal – even though, when such candidate genes are examined, they might turn out to encode digestive enzymes or something. This storifying crops up in Richard Dawkins’ notion of genes as autonomous replicators struggling (just like us!) for reproductive success: it’s a narrative we understand, even though no gene is (or probably ever has been) a replicator in that sense at all.

It seems likely, says the literary scholar Richard Walsh of the University of York, that complexity in general ‘resists the tendentiousness of narrative representation’ – and that in such cases ‘there is an important gap between our narrative talk of what a system does and how the system actually does it’. Here stories surely help us make sense of a complicated universe, but sometimes they are just that: comforting tales, not genuine accounts of why things are the way they are.

But how else, if not with narrative, are you going to make scientific ideas comprehensible? This is not an argument about how to explain science at a popular level, but about how to do science at all. Quantum physics has made that particularly apparent. The primary literature is full of narrative devices to frame the phenomena. Which slit did the photon go through? Was the electron behaving as a wave or a particle? Did the act of viewing the experiment disturb it? How did this particle communicate instantaneously with that one? Did this event happen differently in a parallel universe? All these questions refer to stories placed on top of the maths: it’s not the equations that demand them, but us. We don’t know which story to tell – but we know that we need a story of some kind. These narratives of which particle did what to which raise more questions than they answer, but without them the science somehow seems incomplete.

And that’s the real point. We need narrative not because it is a valid epistemological description of the world but because of its cognitive role. It’s how we make sense of things. An inability to render life experiences into a coherent narrative is characteristic of psychotic disorders such as schizophrenia. Text that fails to deliver narrative coherence, for example in terms of relating cause to effect and honouring the expectations of readers, is harder to understand.

So identifying narratives in abstract activities such as music and sport seems inevitable: if they lacked the properties that make this possible, they wouldn’t catch on, because they would seem pointless and unintelligible. Looked at this way, we might wonder if the ultimate intelligibility of the universe will be determined not so much by the capacity of our minds to formulate the appropriate concepts and equations, but by whether we can find a meaningful story to tell about it.