Once upon a time, in such and such a place, something happened." In basic terms that's about it – the very best definition of a story. What an archetypal story does is introduce you to a central character – the protagonist – and invite you to identify with them; effectively they become your avatar in the drama. So you have a central character, you empathise with them, and something then happens to them, and that something is the genesis of the story. Jack discovers a beanstalk; Bond learns Blofeld plans to take over the world. The "something" is almost always a problem, sometimes a problem disguised as an opportunity. It's usually something that throws your protagonist's world out of kilter – an explosion in the normal steady pace of their lives: Alice falls down a rabbit hole; spooks learn of a radical terrorist plot; Godot doesn't turn up.

Your character has a problem that he or she must solve: Alice has to get back to the real world; our spooks have to stop a bomb going off in central London; Vladimir and Estragon have to wait. The story is the journey they go on to sort out the problem presented. On the way they may learn something new about themselves; they'll certainly be faced with a series of obstacles to overcome; there will be a moment near the end where all hope seems lost, and this will almost certainly be followed by a last-minute resurrection of hope, a final battle against the odds, and victory snatched from the jaws of defeat.

You'll see this shape (or its tragic counterpart) working at some level in every story. It might be big and pronounced, as in Alien or Jaws, it might be subtler, as in Ordinary People, or it might represent a reaction against it (Jean-Luc Godard's Weekend) – but it will be there. It reveals itself most clearly in the framework of the classic crime or hospital drama. A murder is committed or someone gets sick; the detective or doctor must find the killer or make their patient well. That's why detective fiction is so popular; the unifying factors that appear at some level in all stories are at their most accessible here.

The protagonist

Normally the protagonist is obvious. It's Batman, it's James Bond, it's Indiana Jones. If it's difficult to identify a protagonist then perhaps the story is about more than one person (say EastEnders, or Robert Altman's Short Cuts) but it will always be the person the audience cares about most.

But already we encounter difficulties. "Care" is often translated as "like", which is why so many screenwriters are given the note (often by non-writing executives) "Can you make them nice?" Frank Cottrell Boyce, a graduate of Brookside and one of Britain's most successful screenwriters, puts it more forcibly than most: "Sympathy is like crack cocaine to industry execs. I've had at least one wonderful screenplay of mine maimed by a sympathy-skank. Yes, of course the audience has to relate to your characters, but they don't need to approve of them. If characters are going to do something bad, Hollywood wants you to build in an excuse note."

We don't like Satan in Paradise Lost – we love him. And we love him because he's the perfect gleeful embodiment of evil. Niceness tends to kill characters. Much more interesting are the rough edges, the darkness – and we love these things because, though we may not want to admit it, they touch something deep inside us. If you play video games such as Grand Theft Auto or Call of Duty: Modern Warfare (and millions do), then you occupy literal avatars that do little but kill, maim, destroy, or sleep with the obstacles in your path. David Edgar justified his play about Nazi architect Albert Speer by saying: "The awful truth – and it is awful, in both senses of the word – is that the response most great drama asks of us is neither 'yes please' nor 'no thanks' but 'you, too?' Or, in the cold light of dawn, 'there but for the grace of God go I.'"

The key to empathy, then, does not lie in manners or good behaviour. Nor does it lie, as is often claimed, in the understanding of motive. It's certainly true that if we know why characters do what they do, we will love them more. However, that's a symptom of empathy, not its root cause. It lies in its ability to access and bond with our unconscious.

Why are so many fictional policeman and doctors mavericks? Laziness on the writers' behalf possibly, but can that really account for the widespread prevalence of one particular character trait? Why did so many find themselves drawn to Sarah Lund in The Killing? Like her pulp-fiction counterparts, she broke the rules, ignored her bosses and went behind their backs; like them she was told by her bosses: "You've got 24 hours or I'm taking you off the case." Why did she – and why do all mavericks – prove so popular? Largely because that's how many of us feel at times, too. When we watch Sarah Lund rejecting her bosses, we think, "I wish I could do that"; when we watch Miranda Hart's Chummy in Call the Midwife, we bleed for her clumsiness. There is something immensely attractive in living through a character who does obtain revenge, who is proved to have value or, like Lund, is finally proved right. The attraction of wish fulfilment, benevolent or masochistic, can't be underestimated – what else can explain the ubiquity of Cinderella or the current global dominance of the Marvel franchise? Isn't there a Peter Parker in most of us, longing to turn into Spider-Man? We may recoil at the idea of empathising with Adolf Hitler but, as Downfall attests, we can and do.

The antagonist

So something happens to a central character that throws them off the beaten track and forces them into a world they've never seen. A beanstalk grows; a patient collapses, a murder is committed. All of these actions have consequences; which in turn provoke obstacles that are commonly dubbed forces of antagonism – the sum total of all the obstacles that obstruct a character in the pursuit of their desires.

The detective and "monster" templates illustrate this well, but antagonism can manifest itself in many different ways, most interestingly when it lies within the protagonist. Cowardice, drunkenness, lack of self-esteem – all will serve as internal obstacles that prevent a character reaching fulfilment; all make the person more real. While antagonists can be external (James Bond), internal (The Diving Bell and the Butterfly) or both (Jaws), all have one thing in common, which Hitchcock summarised succinctly: "The more successful the villain, the more successful the picture."

What do Bond and Blofeld, Sarah Connor and the Terminator, Sam Tyler and Gene Hunt (Life on Mars), Fiona and Frank Gallagher (Shameless) have in common? They're all opposites. "We're not so very different, you and I," says Karla to Smiley in Tinker Tailor Soldier Spy. "We both spend our lives looking for the weaknesses in one and another's systems."

As the Joker, displaying an uncharacteristic grasp of story structure, says to Batman in The Dark Knight, "You complete me." All forces of antagonism embody the qualities missing in their protagonist's lives.

The desire

Geena Davis and Susan Sarandon in Thelma and Louise. The women want to escape the police and get to Mexico. Photograph: Allstar/Cinetext/MGM

If a character doesn't want something, they're passive. Aaron Sorkin, writer of The West Wing put it succinctly: "Somebody's got to want something, something's got to be standing in their way of getting it. You do that and you'll have a scene."

The Russian actor, director and theoretician Konstantin Stanislavski first articulated the idea that characters are motivated by desire. To find Nemo, to put out the Towering Inferno, to clear their name, to catch a thief, purpose must be bestowed and actively sought. Why do characters in EastEnders offer up the mantra, "It's all about family"? Because it gives them something to fight for; it gives them a goal – it brings them to life. "Tell me what you want," said Anton Chekhov, "and I will tell you what manner of man you are."

Whether simple (kill the shark) or profound (return the key in Channel 4's The Promise), the underlying "grail quest" structure is ever present. Cops want to catch the killer, doctors want to heal their patient. In North by Northwest, everyone is simply chasing microfilm of an unspecified variety. Again, Hitchcock says it best: "[We] have a name in the studio, and we call it the 'MacGuffin'. It is the mechanical element that usually crops up in any story. In crook stories it is almost always the necklace and in spy stories it is most always the papers."

When "something happens" to a hero at the beginning of a drama, that something, at some level, is a disruption to their perceived security. Duly alarmed, they seek to rectify their situation; their "want" is to find that security once again. They may often, however, choose to find that security in the wrong place. What a character thinks is good for them is often bad. This conflict is one of the fundamental tenets of structure, because it embodies the battle between external and internal desire.

External and internal desire

Blockbusters, with one or two exceptions, are two–dimensional. It's a world where desire is simple: the hero wants something - to kill Bill or find the secret of the Unicorn. In pursuit of that goal the multiplex hero doesn't change.

The cynic might say that's because of the demands of the franchise – we want James Bond to be the same in every film. But Bond is the refined, simplified bastardisation of a deeper archetype. He is white bread: impurities removed, digestion eased; a product of the demand for the thrill of story, minus its more troubling and disturbing elements. Bond just wants; he is an embodiment of pure desire. Three-dimensional characters, however, do change.

When we first meet Thelma and Louise they are living in darkness, mortgage-holders in a conservative American society. In The Lives of Others, Hauptmann Wiesler is a Stasi agent, the product of a world where empathy doesn't exist. Here he can flourish – his power and steel are terrifying.

Thelma, Louise and Wiesler are all flawed characters, and it is this concept of "flaw" that is critical in three-dimensional storytelling. Wiesler cannot care; the women are unknowingly repressed.

Flaw or need isn't the same as want or desire. Wiesler wants to punish the dissident couple he has been sent to spy on; Thelma and Louise want to escape the police and get to Mexico. Both sets of characters go on a journey to recognise that what they want stands in direct opposition to what they need. Going to Mexico or imprisoning dissidents will not make them complete.

The Russian formalist Vladimir Propp coined the beautiful term "lack" for what a protagonist is missing in the initial stages of any story, and it's this lack that three-dimensional stories exploit.

While it's possible for characters to get what they want and what they need (certainly that's what happens in Aliens or Star Wars), the true, more universal and powerful archetype occurs when the initial, ego-driven goal is abandoned for something more important, more nourishing, more essential. In Rocky, Cars, Saving Private Ryan, Little Miss Sunshine, Midnight Run and Tootsie, the heroes find a goal they weren't aware they were looking for.

The inciting incident

All stories have a premise – "What if?" This is almost always the inciting incident, or the "something" that happens. In The Long Good Friday Harold Shand is a gangster, planning to develop London's derelict docklands. He's invited the mafia to London to secure their investment when, without warning, one of his gang, charged with taking Harold's mother to church, is blown up in his car. That's the inciting incident – or part of it, because what the inciting incident must also do is awaken a desire. We go back to our story shape: a problem occurs; a solution is sought. Harold's solution is to track down the perpetrators and destroy them: "I'll have his carcasses dripping blood by midnight," he mutters. That's his "want", and that's the film.

Hollywood tends to insist that inciting incidents are massive explosions. But as Fawlty Towers demonstrates, they may just be the arrival of a guest followed by an ever-growing complication AW Schlegel first codified the structural point in 1808, calling them "first determinations". In many ways, it remains the perfect term.

The journey

In Terminator 2,Arnold Schwarzenegger was turned from villain into hero, arguably helping position him as a family-friendly star, but the far more significant adjustment was the upgrade the character underwent. The new model Terminator, the T2, was programmed to learn from his surroundings and experience. Cunningly, his ability to undergo internal change was actually built into the script.

Compare From Russia with Love with Casino Royale, and The Terminator with Terminator 2: the former in each case is a brilliantly slick product, but the latter has a far greater depth and resonance. As the heroes pursue their goals, their journeys in the latter films move us beyond visceral thrill to touch not just our senses but something deeper. In both sequels, the protagonists' superficial wants remain unsated; they're rejected in favour of the more profound unconscious hunger inside. The characters get what they need. Expecting one thing on our quest, we find ourselves confronted with another; traditional worldviews aren't reinforced, prejudices aren't reaffirmed; instead the protagonists' worldviews – and ours too – are realigned.

The quest is an integral ingredient of all archetypal stories, internal or external, and, perhaps most rewardingly, both. Change of some kind is at the heart of this quest, and so too is choice, because finally the protagonist must choose how to change. Nowhere is this more clearly embodied than in the crisis.

The crisis

In The Wire, the death of Omar Little (Michael K Williams) at the hands of a complete stranger works because it is narratively wrong. Photograph: PR

The crisis is a kind of death: someone close to the hero dies (The Godfather), the heroes themselves appear to die (ET) but more commonly all hope passes away. Some US TV drama series refer to it as the "worst case," and in BBC continuing drama, "worst point" has become an almost ubiquitous term. Not for nothing; it's the point of maximum jeopardy in any script, the moment the viewer should be shouting "Oh no!" at the screen, the moment where it seems impossible for the hero to "get out of that". The crisis is also, in self-contained stories, almost always the cliffhanger before the last commercial break and the ending of every episode of EastEnders, of the 1960s Batman TV series and every American serial film of the 1940s from Superman to Flash Gordon.

The crisis occurs when the hero's final dilemma is crystallised, the moment they are faced with the most important question of the story; just what kind of person are they? This choice is the final test of character, because it's the moment where the hero is forced to face up to their dramatic need or flaw. In the Pilgrim's Progress-type structure that underlies Star Wars, Luke's choice is between that of being a boy and a man; in Casablanca Rick has to confront and overcome his selfishness ("I stick my neck out for no man"), and in Aliens Ripley learns, by choosing to save Newt, that she can be a mother once again. In all you can see the cleverness of the structural design, where the external antagonists are the embodiments of what each protagonist fears most. To overcome that which lies without, they must overcome the chasm within.

Hence the stench of death – every crisis is the protagonists' opportunity to kill off their old selves and live anew. Their choice is to deny change and return to their former selves, or confront their innermost fears, overcome them and be rewarded. When Gary sings, "Am I a man or a Muppet?" at his crisis point in 2011's The Muppets, he's articulating the quintessential dilemma all protagonists face at this crucial structural point. Being a "man" is the road less travelled, and it's the much harder choice.

The climax

The climax is the stage at which the protagonist finds release from their seemingly inescapable predicament. It's the final showdown with their antagonist, the battle in which the hero engages with their dramatic need and overcomes their flaw. Historically it is sometimes referred to as the "obligatory scene" (a term coined in the 19th century by French drama critic Francisque Sarcey).

When Thelma and Louise shoot the rapist and decide to run from the law, there's one essential sequence that has to happen: they must do battle with the law. Once Elliot has adopted ET and saved him from the faceless hordes of government, he has to face the "villains" he's hidden him from.

During each film we watch as Thelma, Louise and Elliot develop the skills they need to overcome their flaws; the two women need to believe in themselves and each other; Elliot needs to find the tenacity and selflessness within. And here, in the climax, they apply them. Both are classically structured films, where the flaws of the protagonists are embodied in the characterisation of the antagonists, so that in ET, when Elliot overcomes his external obstacle, his internal need is liberated, and when the women renounce society they become (we are led to believe) emancipated and whole.

A climax can be subverted (the Coen brothers' No Country For Old Men kills its protagonist at the crisis point, but it's very much an exception) but the effect is akin to Bond running from Blofeld. Unless it's part of a wider schematic plan it feels wrong – the writer has set up something and then refused to pay it off.

The inciting incident provokes the question "What will happen?", and the climax (or obligatory act) declares, "this". It is the peak of the drama. Protagonist faces antagonist – all come together to fight it out and be resolved.

The resolution

The word "denouement" is a derivation of dénouer, meaning "to untie", and that's what it is – the knots of plot are undone and complications unravelled. But it is also a tying up of loose ends. In a classically structured work there must be a payoff for every set-up, no strand left forgotten.

Traditionally, stories always ended happily ever after, with all action resolved. Either the tragic hero died or the romantic couple got married. As the journalist and author Christopher Booker has observed, a number of significant changes took place as a result of the industrial revolution in the way we tell stories. "Open endings" have become more commonplace, partly to add an air of uncertainty and partly because, in a godless universe, death doesn't mean what it once did. As Shakespearean scholar Jan Kott noted: "Ancient tragedy is loss of life, modern tragedy is loss of purpose." Characters nowadays are just as likely to drift into meaningless oblivion as to die (The Godfather: Part II); just as likely not to marry as to find themselves at the altar (Four Weddings and a Funeral). Archetypal endings can also be twisted to great effect. The Wire found an extremely clever way of subverting the normal character arc, by brutally cutting it off at an arbitrary point. The death of Omar Little at the hands of a complete stranger works precisely because it's so narratively wrong; it undercuts the classic hero's journey by employing all its conventions up to the point of sudden, tawdry and unexpected death. In effect, saying this is a world where such codes don't operate, such subversion also has the added bonus of telling us just how the cruel and godless world of Baltimore drug-dealing really works.

Putting it all together

These building blocks are the primary colours of storytelling. To a greater or lesser extent they either occur in all stories, or else their absence (the missing bit of Omar's arc in The Wire; the early death of the hero in No Country for Old Men) has an implied narrative effect. In archetypal form these are the elements that come together to shape the skeleton of almost every story we see, read or hear. If you put them all together, that skeleton structure looks like this:

Once upon a time a young friendless boy called Elliot discovered an alien in his backyard. Realising that unless he helped the creature home it would die, he took it on himself to outwit the authorities, win over sceptics and in a race against time, in a true act of courage, set his friend free.

It sounds very simplistic, and in some senses it is, but like the alphabet or the notes on a musical stave, it is an endlessly adaptable form. Just how adaptable starts to become clear when we see how it lends itself to conveying a tragic tale.

Tragedy

When we first meet Michael Corleone in The Godfather he's in an army uniform. Every inch the war hero, he explains the nefarious deeds of his father and his brothers to his fiancee, before mollifying her: "That's my family, Kay, that's not me." Macbeth bears an uncanny resemblance. As he emerges from the mists of battle, Duncan cannot help but be impressed: "So well thy words become thee, as thy wounds: They smack of honour both."

Michael Corleone and Macbeth are both flawed, but their faults are not what are traditionally described as tragic flaws or blind spots. They are, instead, good qualities: selflessness and bravery, and it is this that provides the key to how tragic story shape really works.

Tragedies follow exactly the same principles as Jaws or ET but in reverse. In tragedy a character's flaw is what conventional society might term normal or good – a goodness that characters overturn to become evil in their own way. Historically, critics have focused on the Aristotelian definition of a fatal malignant flaw to describe tragic heroes, but it is just as instructive, I would argue, to chart how their goodness rots. It's a common trope of liberal American movies (in both The Good Shepherd and The Ides of March idealistic patriots find their morals slowly eaten away) but it's equally apparent in Hilary Mantel's Wolf Hall, where Thomas Cromwell undergoes a similar corruption. It is Cromwell's goodness that corrodes him, his loyalty to Cardinal Wolsey that fixes him on the same tragic trajectory as both Macbeth and Michael Corleone. Furthermore, it's a goodness that is corroded according to an absolutely archetypal pattern. From Line of Duty to Moby-Dick, Dr Faustus to Lolita ("good" is a relative concept), there's a clear pathway the characters follow as, in pursuit of their goal, their moral centre collapses. The initial goals can be good (The Godfather or Line of Duty), seemingly innocuous (Carmen, Dr Faustus), but the end-result is the same: the characters are consumed by overwhelming egotistical desire.

It seems impossible to understand how, with only eight notes in an octave, we don't simply run out of music. But just as tones give rise to semi-tones and time signatures, tempo and style alter content, so we start to see that a simple pattern contains within it the possibility of endless permutations. Feed in a different kind of flaw; reward or punish the characters in a variety of ways; and you create a different kind of story.

What's more fascinating perhaps is just why the underlying pattern exists, and why we reproduce it whether we've studied narrative or not. Every act of perception is an attempt to lasso the outside world and render it into meaning. Elliot's journey to maturity, just like the Terminator's journey to human understanding, are interpretations of that basic act. Both metaphorically (and literally in the case of ET) every story can therefore be seen as a journey into the woods to find the secret that lies outside the self. It's in that journey that narrative shape is forged.

• Into the Woods: A Five Act Journey into Story by John Yorke is published by Particular Books on 4 April (£16.99).

• This article was amended on 18 March 2013. The original referred to Thomas Cromwell's loyalty to Cardinal Newman, instead of Cardinal Wolsey.