Not long ago, I was bingeing on the second series of Jill Soloway’s award-winning television series Transparent when I was, completely unexpectedly, brought to tears. The show tracks the ramifications of a family patriarch’s decision to transition to a woman, from Mort to Maura. The scene in question revolved around Maura’s son, Josh Pfefferman, a jovial, wry, essentially decent record company executive who is thoroughly modern, and always wanting to be supportive of Maura’s journey.

But as the series progresses, things start slipping for Josh. In one scene, he’s driving with band members and starts uncharacteristically ranting. “Look at this traffic!” he says. “They time it out so you can’t get anywhere. It’s a fucking conspiracy.” Josh honks his horn at other drivers. “Fucking go you piece of shit!” He’s losing control. The woman beside him insists he pulls over. Josh is hyperventilating.

Sometime afterwards, he calls in to see his mother, Shelly, only to find she’s out. Shelly’s new boyfriend Buzz lets him in. “Nothing’s adding up,” Josh confides to Buzz. “I thought stuff would add up by now, but everything’s slipping through.” Buzz – grey ponytail, hippy shirt – is of a different generation. His perception of reality, and Josh’s predicament within it, comes from an earlier time. He suggests Josh is in “shock” about the “loss” of his father. Josh pushes back. Buzz doesn’t get it: nobody has died. “You think I miss Mort?” he asks, irritated.

“What do you think?” says Buzz.

“Well, it’s like politically incorrect to say that you miss someone who has transitioned, so…”

“This isn’t about correct, Joshua, this is… this is about grieving. Mourning. Have you grieved and mourned the loss of your father?” There is a moment of silence, and then Josh crumbles into the arms of the older man and sobs.

‘I was continually fascinated by the Pfeffermans. And yet they were entirely made-up people’: bingeing on Transparent. Photograph: Amazon/Everett/REX Shutterstock

And there I was, sobbing, too, utterly gripped. I was, by turns, made to laugh, gasp, plead and moan in frustration, continually fascinated by the Pfeffermans and the characters that orbited around them. And yet they were entirely made-up people. They don’t exist. Nor, of course, do any other fictional characters. So why do we care so much about them? Hour after hour, millions of us obsess over the fantasy selves that feature in film, television and literature, transfixed, emotional, addicted. What are they going to do next? How are they going to get away with that? Are they going to get what they want? How are they going to get it? Why did they do that ridiculous, unpredictable, self-destructive thing?

If we have a natural obsession with the behaviour of other humans, real or made-up, it’s partly because of the unique realm in which Homo sapiens have come to exist. We’ve been a social animal whose survival has depended upon human co-operation for hundreds of thousands of years. But over the past roughly 1,000 generations, it’s been argued that our social instincts have been rapidly honed. As we began to form settled communities, the people who were better at understanding and getting along with others, rather than the physically dominant, became more and more successful. They were less interpersonally aggressive, but more adept at the kind of psychological manipulation necessary for negotiation, trading and diplomacy. This “sharp acceleration” of selection for social traits, the developmental psychologist Professor Bruce Hood writes in The Domesticated Brain, has left us with brains that are “exquisitely engineered to interact with other brains”. We’d become brilliantly expert at controlling an environment of other human minds.

Stories are often a deep investigations into human character.

And that’s who we are today. If dogs live principally in a world of smell, and moles in a world of touch, then the human realm is that of other minds. In order to survive and reproduce, we’ve had to become masters at manipulating our environment of people. And the importance and complexity of human behaviour means we have developed an insatiable curiosity about the ever-fascinating “whys” of what people do. Storytellers exploit both these mechanisms and this curiosity; the stories they tell are often deep investigations into human character.

In order to really investigate who people are, we need to test them. Many of the most memorable and gripping scenes in stories are the ones that truly push its characters. A major purpose of plot is to apply enormous pressures on to these fantasy selves, and force them to reveal who they truly are. Some of the most moving moments in a story arrive when characters suddenly reveal previously hidden parts of themselves under strenuous circumstances. This is precisely what happened in that moment of high drama involving Josh Pfefferman in Transparent.

Of course, different stories, characters and scenes connect with everyone differently. We all bring ourselves to the art that we love. It’s because of a peculiarity in my own real-life backstory that any scenes involving themes around “loss of the father” are liable to make me suddenly weepy and helpless. But, even allowing for that, the scene worked. It was powerful. It was dramatic. It was built out of story-stuff.

Such scenes move and fascinate partly because they ask fundamentally dramatic questions: who is this person really? What are they all about? Deep underneath the surface, what’s the true nature of their character? These are the questions many of our most compelling stories ask. If there’s a single secret to storytelling, then I believe it’s this. “Who is this person?” Or, from the perspective of the character, “Who am I? Who am I going to be?” It’s the definition of drama – its electricity, its heartbeat, its fire.

We can’t jump into a screen and throttle the villain, but the urge to act compels us to keep watching, hour after hour

In fact, it seems that the dramatic question probably goes back to the very roots of human storytelling. We’ve been telling stories about the nature of each other’s characters for tens of thousands of years, starting back when we were still living in large, co-operative hunter-gatherer tribes. Recent research suggests human language evolved principally so that members of a tribe could swap “social information” about each other. In other words, we’d gossip. We’d keep track of our tribe-mates, closely tallying their behaviour and telling tales of their moral rights and wrongs. When these gossipy stories concerned a person behaving selflessly – when they put the tribe’s needs before their own – listeners would experience a wash of positive emotions and an urge to celebrate and reward them.

But when they were told tales of someone being selfish, listeners would experience moral outrage. They’d be motivated to act – to punish them, whether by mocking and humiliating, violently attacking or ostracising them from the group, which would have been a death sentence. This is how we policed our tribes and kept them functioning as highly co-operative groups. Tales of people being selflessly heroic or selfishly villainous, and the joy and outrage they triggered, were crucial to human survival. We’re wired to enjoy them. Such stories are designed to ask and answer the dramatic question: who is this person really?

These Stone Age values and moral questions remain strongly evident in modern stories, not least in box-set television series. When a character behaves selflessly, we often experience a deep primal craving to see them recognised as a hero and rewarded. When a character behaves selfishly, putting their own interests before the tribe’s, we tend to feel a monstrous urge to see them punished.

“Stories arose out of our intense interest in social monitoring,” the psychologist Professor Brian Boyd writes in On the Origin of Stories. They work by “riveting our attention to social information” and feature “heightened versions of the behaviours we naturally monitor”. Today, like then, the social emotions aroused by the story motivate us to act. But because we can’t jump into a television screen and throttle the villain, the urge to act compels us to keep watching, for hour after hour, until our tribal appetites have been satisfied. These emotional responses exist as neural networks that can be activated whenever they detect anything in the environment with the rough shape of tribal unfairness. In many of our archetypal stories, moral outrage on behalf of a protagonist is triggered in the earliest scenes. Watching a selfless character being treated selfishly is a drug of enchantment for the storytelling brain. We can’t help but care.

‘It might seem odd that we root for anti-heroes such as Tony Soprano’: James Gandolfini as a mafioso in The Sopranos. Photograph: HBO/Rex/Shutterstock

There’s another quality to ancient storytelling that’s still massively evident in tales we enjoy for pleasure. Studies of gossip in contemporary hunter-gatherer tribes find that, just like stories in our most advanced cities, they are dominated by tales of moral infractions by high status people. We don’t like people who are more powerful. We fear and resent them. We’re much more likely to extend our empathy to characters who are low in the hierarchy – vulnerable, reluctant, trembling in the shadow of evil Goliath.

The opening minutes of the television classic Breaking Bad are powered by these ancient tribal forces. Walter White is deserving of high status – we see in his home an award for being the “contributor to Research Awarded the Nobel Prize” – but when we meet him, he’s just a suburban chemistry teacher who’s passionate about his subject. He has a pregnant wife and a teenage son with cerebral palsy and, to make ends meet, works after hours at a car wash. There, we see him humiliated by a wealthy student whose sports car tyres he has to polish as he’s mocked by the smirking boy and his giggling girlfriend. He returns home to find his wife has organised a surprise party for his 50th birthday. Walt’s boorish brother-in-law, DEA agent Hank, humiliates Walt repeatedly, allowing his son to play with his firearm against his wishes, mocking him and even taking his drink off him during his own birthday toast. Next we discover Walt has lung cancer. And it’s inoperable.

It’s not fair that selfless, low-status Walt should suffer in this way. It goes against our wired-in sense of justice. It’s outrageous that this hardworking and under-rewarded man, with his family to support, has to endure the tribal punishment of humiliation and is then told he’s going to die. We begin to root for him, just as we crave that awful Hank gets his comeuppance. Breaking Bad screenwriter Vince Gilligan is so successful at manipulating our primal tribal emotions that we’re still rooting for Walt when he’s dissolving the bodies of his enemies in acid.

Similar psychological manipulations take place on behalf of other antiheroes, not least the protagonist of the pioneering long-form television series The Sopranos. It might seem odd that we root for antiheroes such as central character Tony Soprano. But, when we do, it’s often because our tribal emotions are being subtly manipulated by clever writers. Our first meeting with the mafioso occurs in a psychotherapist’s waiting room. We learn he’s developed an emotional bond with some ducks and ducklings that regularly land in his pool, and that he suffered a panic attack when they finally left. He weeps when he speaks of them. Not only is Soprano sensitive and in pain, he’s relatively low in status. Far from being some all-powerful John Gotti, he’s the capo of a marginal New Jersey gang and, anyway, as he tells his therapist: “I came in at the end, the best is over.”

When we do see Soprano beating a man, the victim is just a “degenerate fucking gambler” who owes him money and insulted him: “You’ve been telling people I’m nothing compared to the people who used to run things.” As the episode unfolds, Soprano secretly tries to help a non-mob friend in whose restaurant his much more horrible uncle has planned a hit. Soprano cares for his mother. He visits her and brings her gifts. When he takes her to a prospective nursing home and she becomes distressed, he suffers another anxiety attack. We then discover she’s plotting with his uncle to have him killed. It’s not fair! Tony might do bad things, but the dramatic question is pointing to something much more nuanced about his essential nature. Despite ourselves, we begin to root for him. If we do, it’s because, even in our 21st-century golden age of box-set television, we still experience the world with Stone Age brains.

The Science of Storytelling by Will Storr is published by William Collins at £12.99. To order a copy for £9.99, go to guardianbookshop.com