People change, don’t they? That’s what everyone says. I once met a man who changed more than you could imagine. His name is John Pridmore. To see him was to know him, or so you’d think. Shaven head, goatee beard, heavy jewellery, black T-shirt, torso that looked more fat than muscle but no less intimidating for that. If you believe in evil, then that’s what John became. As a kid he was a happy little Sea Scout who once asked for a Bible for his birthday. He enjoyed holidays in Hastings on the south coast of England. He liked watching John Wayne films with his dad. But when he grew up, he was an enforcer for the gang that supplied London’s West End with most of its drugs.

This is what John became. One day, he spotted a man he’d been looking for. The man was collecting his six-year-old son from school and, as the boy watched, John punched the man to the floor, knelt on his throat, took him by the ears and smashed his head against the curb. Parents screamed. Blood ran. Days later, in a Leyton pub in east London, the beaten man’s sixtysomething father complained to John that his grandson had been traumatised. John punched a pint glass into the old man’s face, then returned to the pub with two associates and a set of golf clubs and destroyed everything that he could see.

Say what you like about him, but John was a success. He drove a BMW 7 Series and had three homes, including a penthouse flat in St John’s Wood, overlooking Lord’s Cricket Ground. He attended Champagne and crack parties in Notting Hill. As well as being an adept stand-over man, he worked the door at clubs where his firm controlled the narcotics trade. But he was losing control over himself. One night, when he was on duty at a gig venue just off Charing Cross Road, two men tried to enter without paying. The woman at the counter asked them to leave. One of the men insisted they wouldn’t be stopped, even by John. ‘What they were doing, in my eyes, was belittling me in front of this girl,’ John told me. ‘I’m known as someone who’s got a certain amount of authority. And she’s seeing these two muppets aren’t respecting me. They’re taking away my reputation. So I get a bat from behind the bar and I beat them. And I can honestly say that I nearly killed them.’

John lives in Ireland these days, so we’d arranged to meet at his mother’s home in east London. It was a small flat in a retirement complex, neat and crowded with thoughts and shadows. There were pictures of Mary and Jesus, and crucifixes on walls and shelves. His mum, having brought in tea and biscuits, perched on a sofa to the side of John and me, while we sat opposite each other. I’d come because I wanted to find out how John had changed from little boy to big monster to the man he is today. It seemed extraordinary. Most people don’t change, not really. So what was special about John? What was it that drove his two metamorphoses?

His most dramatic transformation came after yet another violent outburst. Thinking that he’d killed a man and worried that prison was becoming inevitable, he decided to take some time off to relax. He drove to his old flat on the Beaumont Estate in Leyton with a spliff and some cans of Special Brew. There were swords on the wall, pizza boxes and pornography on the floor. John didn’t know it but, a few streets away, his despairing Catholic mother was praying to St Jude, the patron saint of hopeless causes, for his death.

As he watched TV, John heard a strange voice in the room with him. It was listing all the terrible acts he’d carried out. ‘This is your life,’ it said. ‘This is what you’ve done.’ At first he thought it was the television. But how did the television know all his secrets? John sat up. ‘Hang on,’ he thought. ‘There’s something wrong here.’ He switched off the TV. The voice was still there. Then suddenly he understood. ‘I felt the life draining out of me,’ he said. He ran outside his flat in a panic and wailed at the sky: ‘Help me!’ As he let out his cry, he felt bathed in a glow of ecstasy.

By now, the evening was filling up the room. We sat in the half-dark, wreathed in the violence and religious magic of John’s memories. ‘It was for less than a minute, but I felt this incredible sense of the love of God,’ he said. ‘The greatest buzz I’d ever felt.’ Even better than crack? ‘Not even in the same ballpark.’ It was the early hours when he arrived on his mother’s doorstep. ‘Mum,’ he said. ‘Something’s happened. I’ve found God.’

One of the most extraordinary facts that I picked up while researching a book about belief and the brain came from the American psychologist Timothy D Wilson. In Strangers to Ourselves (2002), he describes a study that suggests that, at any given moment, the brain is capable of receiving more than 11 million bits of information. And yet we are consciously aware of fewer than 40. How does the brain parse this maelstrom of input? It does it by telling stories.

The brain generates a narrative to make sense of the world around us, but also to make sense of ourselves. We think we’re captained by the part of us that’s self-conscious – the bit that we experience as our own living ‘me’, that collision of sense, memory and internal monologue at the centre of which sits the ‘I’. Yet there’s a silent, unconscious ‘I’ to which we have no access. It communicates with emotions, wordlessly coaxing us this way and that with its ceaseless blooms of disgust and fear and desire. It influences everything we think and do.

Exactly how much influence does this self have over our behaviour? Experts disagree. Some say its control is total: that the voice that speaks in the privacy of our heads might seem like it’s in charge, but really it’s just a babbling spin doctor, making excuses for the misdeeds of its boss. Others claim that our rational selves can play an executive role under certain, limited circumstances – but not much more than that. Either way, most of the time we feel that we’re autonomous only because the voice in our heads narrates all our actions, explaining why we’re doing what we’re doing at any given moment, even though it actually has no idea.

Our brains conjure the illusion of order; they wrench a plot from the chaos and then place us heroically at its centre

This story-making has a name. It’s called ‘confabulation’. In an ordinary, healthy brain, it tends to make us feel good about ourselves and the reality that we feel we’re at the centre of. Religion is a kind of confabulation. It’s a story that explains the origin of life and the universe while answering profound questions about death, loneliness and morality. Another story is the notion that your family is special, or that your political ideology is important. None of this is true. To insist we matter is to conduct a circular argument: people matter because they matter to people (in which case, if you got rid of all the people, it wouldn’t really matter). But we must believe these tales – and we do.

We live, moment to moment, in an emotional reality of love, hate, feuds, sorrows and dreams. We spin seductive, reductive narratives of heroism and villainy, struggle and victory, to parse reality and give ourselves esteem and our lives meaning. Rituals help. We use them to place ourselves at particular plot-points in the story of our lives. They reinforce our tales, making us seem important and our journeys comprehensible. In the chaos of the daily world and our irrational behaviour within it, our brains conjure the illusion of order; they wrench a plot from the chaos and then place us heroically at its centre. And what heroes we are! A symphony of optimism biases soothe us into believing we’re smarter, better looking, more morally upright than we are. Primitive tribal instincts turn our enemies into ruthless ignorant baddies while our allies are crowned with undeserved haloes. As we push through the minutes of our lives, we’re all Davids fighting our own personal Goliaths. We’re seduced into believing in the autonomy and moral behaviour of our coherent and comprehensible selves.

But the self is a lie.

John’s first metamorphosis was driven by anger. One evening when he was 10, he came home from Sea Scouts and his parents told him they were divorcing. Which one of them did he want to live with? He thought they were joking.

After the break-up, his mother was committed to a psychiatric hospital called Claybury, in Essex. John remembers the screams down distant corridors and the vacant eyes staring at him. Claybury scared him so much that he often felt like vomiting. But he missed his mother, so he visited regularly, even though she mostly didn’t recognise him. Once, she told him she was Joan of Arc. Another time she said he was ‘the devil’s son’. John broke out in a nervous rash that covered his body. His doctor forbade him from seeing her.

Living with his gravely unpredictable father, John’s fury began to gather. He found friends who were loyal and kind to him, but also criminal. At 19, he spent time in a young offender institution, in solitary confinement. He would stare out of his window towards the distant sea and think about killing himself. He wrote to his mother, apologising for letting her down.

‘I did think he’d let me down,’ she told me.

‘Well, you didn’t come visit, did you?’ John said, not directly looking at her. ‘I was very annoyed about that.’

It was a strange moment. I looked at his mother.

‘I would’ve gone, because it’s not so far, really, by train,’ she said. Then she blamed her boyfriend. ‘He put me off. He used to say: “Oh, it’s such a long way”.’

I asked John if it hurt, hearing this.

‘No,’ he said.

‘It hurts me because I feel bad for John,’ his mum said tearfully. ‘I feel I’ve let him down, really.’

I found it difficult to blame John for what he became. Knowing the emotional physics of a life makes it hard to judge it. But I was fascinated to discover what story John told himself that brought him a sense of pride. If it’s true that our brains tell us tales that make us feel good about we do, no matter what we’re doing, then what about his? What was John’s confabulation of self, his hero narrative?

‘When I look back at that person, I don’t even know who he was. He’s just so removed from who I am now’

‘To us,’ he explained, ‘the general public were complete idiots. Why would you go to a nine-to-five job when you could earn so much money doing so little? Our mentality was that we were the only sensible people and everyone else was vermin.’

He felt clever, then. But the self he created had more to it than that. In order for him to do his work, he had to design a version of John Pridmore that people would talk about with dread. This was so important that, when his identity was challenged outside that London nightclub, he nearly killed the men responsible. Of this incident, he remarked: ‘If you’re not looked up to as being the most hard, the most strong, the most vicious, if you lose your name, then you’re nothing.’

Nothing. Nothing! What an observation. John was like his own author, designing a character who would best serve his life’s mission. And if his authoring was to fail – if another self didn’t recognise the character he’d worked so hard to construct – then he would suffer an obliteration. He would be nothing.

And yet listen to him now: ‘When I look back at that person, I don’t even know who he was. I don’t recognise him at all. I don’t know what motivates him. He’s just so removed from who I am now.’

How can we explain such a change? I couldn’t work it out. If the brain is so good at justifying everything we do – and it is! – why would it suddenly destroy both the plot and its hero? Why would it turn itself inside out?

Here’s what I’ve been thinking. It’s a truism that we can be many characters in the space of a day. Because we’re so good at seeing ourselves as consistent and sane, we can underestimate just how many faces we have. Yet our many-faced nature is a fundamental. In Incognito: The Secret Lives of the Brain (2011), the American neuroscientist David Eagleman writes that ‘brains… are machines made of conflicting parts’. The decisions our unconscious makes can be seen as the outcome of a kind of fight between these parts; ‘as a result,’ writes Eagleman, ‘you can accomplish the strange feats of arguing with yourself, cursing at yourself, and cajoling yourself to do something.’

Despite the coherence of his hero illusion, John, like all of us, was always more than one person. But when we behave in ways that conflict with our heroic fantasy of ourselves, we suffer the discomfort of cognitive dissonance. We pick away at what we’ve done, over and over. Our internal monologue hunts for a way to justify our behaviour, to reconcile the regrettable act and the blameless actor.

In the weeks before his conversion, John became angrier and more violent. His constructed self was trying desperately to hold together. Perhaps the old, long-silenced part of himself, the little boy who once asked his auntie to buy him a Bible, was finally beginning to roar back. John began to experience powerful physical sensations. He heard disembodied voices. He would disagree, but some might assume he was having a psychotic breakdown. Yet his brain didn’t stop writing a narrative to explain it all away. To the mess of that night it brought confabulation, order, control… or at least the illusion of it.

To accommodate this new story of reality, his personality had to go through a fresh metamorphosis. A new hero narrative arose

The story of Jesus and the devil had been in John’s head since he was a boy. He was always fascinated by religion. When he heard the voice in the TV set listing all his sins, there could be only one explanation – it must be Satan! But then, to accommodate this new story of reality, his personality had to go through a fresh metamorphosis. A new hero narrative arose. A replacement self, with a new mission.

John is a saved soul, now, preaching the Bible at Christian retreats, conferences and seminars around the world. He’s waging daily war, not against the enemies of his drugs gang, but against Lucifer.

As a journalist with an interest in the mind on the fringes of human experience, I kept noticing this pattern. It seemed as though the brain would do almost anything to maintain its feeling of control, even if that meant turning our selves and the plot of our lives inside out and upside down.

While working on a story about Uganda and Joseph Kony’s Lord’s Resistance Army earlier this year, I learnt that Kony had a warlord predecessor, a charismatic fighter who triggered what came to be known as the Holy Spirit Wars. Her name was Alice Auma. In 1986, Alice was 30 and her life was in ruins. Unable to conceive, she had been abandoned by two husbands and was having a breakdown. She couldn’t move or speak. And then suddenly she was possessed by a spirit, an Italian soldier called Lakwena. She took herself to the forests of Paraa on the banks of the Nile where the animals and the mountains and the waterfalls spoke to her.

Alice believed that the spirits had chosen her because she was a sinner and a failure, at school and as a woman – in her own words, ‘a hard case’. She became a noted mystic and developed a following. In August 1986, the spirit Lakwena instructed her to conduct a war against the government on behalf of her people. Incredibly, she managed to grow her local following into an army of 10,000, naming it the Holy Spirit Mobile Forces. Having convinced her soldiers that they didn’t need weapons because bullets would bounce off their skin, they fought half-naked, slathered in magic oil. The Ugandan forces fled at the sight of these men and women who were wailing hymns, gripping Bibles and throwing blessed stones. Alice’s troops managed to get to within 80 kilometres of the capital, Kampala, when, during a hopeless final battle, she hopped onto a bicycle and pedalled safely away.

On the Isle of Wight, off the south coast of England, I came across a more prosaic example of the same phenomenon: David Icke. Having had careers as a professional footballer and TV presenter, Icke abruptly declared himself the ‘son of the godhead’ and made a humiliating appearance on UK television. He went on to achieve remarkable success as the author of conspiracy theories about alien ‘reptilians’ disguised as world leaders.

What happened to him? It seems significant that his transformation followed hard upon the death of his father, whom he worshipped, and a crisis in his career. David told me that, in the midst of all this, he felt as if someone were ‘pressing too many keys on a computer, and the computer freezes because it can’t process it. It was like sitting in a bubble and then someone bursts the bubble and everything suddenly starts pouring in.’ A psychic had passed on a message that she told him came from Socrates himself: David would be ‘a healer who is here to heal the earth and he will be world-famous’. And so he went on a British chat show and started prophesying a series of terrible natural disasters. A new David Icke was born.

The sheer, magnificent power of their urge to think well of themselves transformed their entire story of reality

There’s Alice, barren and abandoned, a hard case, going through a mental collapse, yet her brain confabulates a story that puts an optimistic spin on the bizarre things that are happening to her. You’re chosen, it tells her, you’re a powerful mystic, you have a mission. And she makes 10,000 people believe it. There’s David Icke, once falling apart on prime-time TV yet today he can sell out New York theatres and Wembley Arena with his lectures about the shape-shifting lizards that control the world. And John Pridmore is the same, except he’s saying that everything is controlled by God. Since his conversion, he has preached to more than a million people worldwide.

These are damaged people. They’ve done amazing things. And they’ve done it all, as far as I can see, as a result of their struggles with madness. None of them were sectioned. None received, or even sought, psychiatric help. Instead, their storytelling brains wove heroic narratives that explained away the collapse of their identities even as they were taking place. Their confabulations were so credible to them that they metamorphosed into different selves. And then, in their own small ways, they went on to change the world.

I’ve spent my professional life meeting and writing about people such as Alice, David and John: people whose sense of self is unusually bright and fierce. They’re often leaders, spokespeople, evangelists for their beliefs. They can be intolerant of other perspectives, with little capacity for nuance and doubt. Their hero narratives are strong – potent enough to adapt and survive, agile enough to pirouette across their psychic crises to emerge more powerful. That’s how they did it, that’s how they changed: the sheer, magnificent power of their urge to think well of themselves transformed their entire story of reality.

But what if you don’t have such a resilient self-narrative? What if, instead of confidence, you have doubt?

I had a personal stake in all this, of course. I became fascinated by the question of change after I made the mistake of reading some diaries that I’d kept, 10 years previously, in the aftermath of a relationship breakdown.

18th April

Finished writing at 20:00 and was really craving beer. I SWORE not to drink, to make a point that I was in control. Went to the pub, had three pints, came home and drank until 04:30. Got into work 12:30, still drunk, slurring words. Had a big row with my editor. Went out, got pissed – attention-seeking, shouting, talking over people. Bad.

28th April

Didn’t sleep until 04:30 last night. Depressed. Called Ella. She came round. Despite my half-arsed decision not to sleep with her, I did.

29th April

Couldn’t sleep. Got paranoid about who Ella was out with before she came round last night. She said it was someone called Karen, but it could’ve been anyone. Went through all the lies she might have told me and all the possible arguments in my head. Called her three times after 1am so I could ask her. No reply. Saw her today and got paranoid about whether she carries a condom with her. I mean, what the fuck?! I carry a condom with me! She said, ‘No, do you?’ I said ‘No’.

That was how it was, page after page. An ugly mind spelt out in cheap biro. Ella and I had been together only six months, but it was a relationship of unbearable passion – obsession, really – and our grim coda was to drag itself out for nearly two years. Do you need me to write the words? I know what I was. From the diaries, it even seems as if I knew it at the time.

John Pridmore achieved his transformation by negotiating an extraordinary pivot in his hero illusion. One story became another. He forgave himself for his reprehensible acts by adopting a narrative in which forgiveness was possible, through regular confession and prayer. That wasn’t an option for me. Knowing what I knew about the lying powers of the brain, there was no story I could tell that would make me feel better about how I’d behaved. I already knew that anything I told myself would just be self-serving confabulation.

In the weeks after reading my diaries, and then meeting John, I became somewhat irrational, my thinking disordered. I was trapped, stuck with a person I couldn’t stand: myself. But then I realised something. In certain objective ways, I have changed. Since writing those diaries, I’ve become sober and drug-free. I’ve been in a happy, stable relationship for 11 years. How can that be possible? My underpowered hero-narrative certainly didn’t save me. So what happened to me?

Should John Pridmore’s victims feel anger at the man he was then or at the man he is now?

People often become offended when they read an admission of self-loathing, but the truth is that, whenever I’ve tried to force change for the better, it’s been as a result of being disgusted with myself. As an addict, giving up drinking and smoking felt like flagellation. I wanted to punish myself. I wanted it to hurt. And I suppose this rather self-abasing impulse probably comes from a desire to think well of myself, just as John, Alice and David did with their more spectacular transformations.

But there are questions I can’t face or don’t know the answer to. Should John Pridmore’s victims feel anger at the man he was then or at the man he is now? Should they resent him and his brain for telling its new, redemptive story – for saving itself? Does he deserve punishment or his victims’ revenge? And what about me? I spent many years as a jealous lover, but I’m not like that anymore. Am ‘I’ the person I am now, or the one I was then? Do I deserve my happy marriage? I can tell you only what my emotions say because, on this, my inner voice runs silent. No, to be honest, I probably don’t.

The American psychologist Roy Baumeister writes: ‘People may claim they are inspired by the Delphic command to “know thyself” but the self they tend to seek is one that is uniquely superior to others and in control of outcomes.’ There is something in that which connects people such as me to those who are more optimistic and confident. I’ve sought this knowledge about change, in the first place, because facts bring order. Once I know how change works, I can decide, I can act, I can attempt to calm the chattering dissonance. In that, I’m just like David Icke, just like Alice Auma, just like John Pridmore, and just like the brain inside you. The only thing I ever wanted was the illusion of control.

In October, Will Storr is teaching a course for writers on the science of storytelling, in London. Details are available here.