How does a book get on The New York Times bestsellers lists? For those outside the publishing industry, the question seems tautological. You get on a bestseller list by being among the top 10 bestselling books in your category. Obviously.

With certain caveats, that’s true. (Among those caveats – you can’t be a perennial bestseller, like the Bible: it would be number one on the list every week.) But that’s like answering the question: ‘How do you get to Carnegie Hall?’ with: ‘Take the Q to 57th street.’ Yeah, sure. But I want to know how you get on stage.

So how does a book achieve a level of sales that grants access to the list? Well, there are certainly tricks that publishers and authors use to boost sales at just the right moment, or in just the right way, to register on the bestseller lists. There are many, many books that were on a list for one week or two weeks and then dropped off, never to return again. They benefited, in all likelihood, from one of those tricks.

But there are other books that climb on to a list and seem to just stay there. How does that happen? That is no publisher’s trick. Neither savvy marketing, nor blanket marketing, nor really any kind of marketing can produce that kind of success. Books stay on a bestseller list for months at a time because people actually like them. They benefit from the exponential power of word of mouth. Albert Einstein reportedly said that compound interest is the most powerful force in the Universe. A publisher would disagree. Word of mouth is the most powerful force in the Universe, because word of mouth also benefits from an exponential model of growth: one person tells three people how good a book is, and they each tell three people, who each tell three more, and pretty soon that book is ensconced so firmly on a bestseller list that the list might as well be etched on stone tablets. That’s how you really get on the bestseller list.

But how do you write a book like that? No one knows.

Well, maybe the airport thriller-writers of the world know; they seem to produce bestseller after bestseller. But they write books that are the literary equivalent of the Candy Crush Saga video game, providing tiny dopamine hits with every swipe of your finger (or turn of the page, if you still read on paper). Just as few people say that they love Candy Crush, very few people say that they love such books. Doing something over and over doesn’t necessarily mean you love it.

For those of us who strive to write novels that are unique, and literary, and still bestsellers – books that people talk about because they love them – well, how to consistently do that is still very much a mystery.

You know what else is a mystery? My daughter. When she was born, she was a cipher. An unreadable blob of so-soft-it’s-almost-edible flesh. When she was about four months old, though, she became a very effective communicator. When she was hungry, she’d shriek like a banshee. When she was tired, she also shrieked. When she was bored or physically uncomfortable, she shrieked. She wasn’t crying. She was shrieking, and it was louder, more high-pitched and more sudden than any other sound I’d ever heard a human make. It was painful to the ears and the heartstrings, which made it a remarkably effective form of communication.

She shrieked to tell us just four things – fatigue, hunger, discomfort, boredom. So I was left wondering a great deal about her inner life. When I hold her in front of a mirror, what does she think? When she is gazing into space, how is she processing all that she has been exposed to? And when she shrieks particularly loudly, is she more upset? Or is she just experimenting with varying her mode of communication?

I don’t think I’ll ever know the answers to these questions. But, luckily, her psyche will become more and more transparent as each month goes by, because in each month she will become more and more interested in books. Our selection of books, from earliest childhood through to the ends of our lives, provides a window into our secret souls. And children, who (necessarily, for survival purposes) are most in touch with their desires and drives, tell us most about their inner lives when they tell us which books they want to read.

Sigmund Freud said a lot of crazy things, but one of his most compelling insights was that the mind is like the city of Rome. Each age has its own architecture, its own monuments, built on top of those from the previous ages. But instead of knocking down those monuments to an older time and replacing them, the mind preserves each landmark. Some, like the Colosseum, are more obvious, while others are hidden in the shadows of Palatine Hill. Even more completely than Rome, each adult keeps the landscape of her childhood intact. If you want to understand that childhood landscape, the foundations on which a person’s life is built, ask her what her favourite books were as a child.

As a young boy, I loved The Carrot Seed (1945), written by Ruth Krauss and illustrated by Crockett Johnson. The text is so simple I can quote it in its entirety:

A little boy planted a carrot seed.

His mother said: ‘I’m afraid it won’t come up.’

His father said: ‘I’m afraid it won’t come up.’

And his big brother said: ‘It won’t come up.’

Every day the little boy pulled up the weeds around the seed and sprinkled the ground with water.

But nothing came up.

And nothing came up.

Everyone kept saying it wouldn’t come up.

But he still pulled up the weeds around it every day.

And sprinkled the ground with water.

And then one day

A carrot came up.

Just as the little boy had known it would.

I was obsessed with this book as a young child. I demanded it over and over, more than any other book. It’s not hard to see why. The writing is simple and subtle. The brother’s curt ‘It won’t come up’ cuts deep, for example. Since I didn’t have a brother until I was nearly five years old, I remember associating that line with my father. In my psyche, he got two putdowns, the second sharper than the first.

The illustrations are no less brilliant than the text. When the carrot finally does sprout, its green fronds are bigger than the boy protagonist, and he carts the carrot away in a wheelbarrow because it is bigger than he is, and (if the laws of physics are obeyed in picture books) must weigh at least 50 pounds. Not only was his family wrong to doubt him, his triumph is supernaturally enormous.

I will skip the Freudian interpretation of the carrot as a penis, though in shape, colour and placement it’s an easy argument to make – especially since the book came out of the culture of publishing in New York City in the same year that Alfred Hitchcock’s Freudian ode Spellbound was released.

I loved this book enough to have kept it at my elbow through college, my first job, my second career, and into full-fledged adulthood

More interesting than the phallic carrot, though, is that the family doesn’t reappear to see the boy’s triumph. Unlike in popular US films, where all the doubters have to watch the hero’s victory and, despite everything, stand and applaud (cf every movie that ends with either a sports event or a prom), the protagonist of The Carrot Seed doesn’t need his family to tell him that the carrot came up. He knows it did. It’s enormous, and vaguely purple, and is way bigger than his dad’s… carrot. If you know what I mean.

So what does it say about me that I loved this book as a child enough to have kept my personal copy at my elbow through college, my first job, my transition to my second career (writing), and now into parenthood?

My father, despite being a wise and kind man, seemed incapable of not competing with me, his eldest child. My mother says this started pretty much from birth, which was the moment that his preeminence in the household was threatened. I loved my father totally, and he loved me too. We played together. He encouraged me in sports – which weren’t one of his strengths. But any new knowledge I acquired and shared had to be either contradicted or upstaged by him. My little brother, once he came along, was always ‘Right, in a way’. My answers, on the other hands, never seemed good enough. Even, to my fury, when I was repeating verbatim facts that my father had taught me earlier.

When my brother was old enough, the three ‘boys’ of the house would play Monopoly. (My mother refused to play with us, because she was always tempted to give my little brother money, which my father and I found unacceptable). The only viable long-term strategy in Monopoly is to buy every property one lands on, to prevent one’s opponents from getting monopolies. Then, once all the properties have been divvied up, if no one has a monopoly, the game will either go on forever, or the players will have to agree to a mutually acceptable trade.

But we three boys were so afraid of losing that we would negotiate over our trade for 30 minutes, 40 minutes, an hour. Often, the negotiations ended with either my brother or me in tears. Soon, the stalemates became so excruciating that we agreed to stop playing Monopoly altogether. Years later, when I was 15 and my father was 56, we were driving in his car. I said to my father: ‘We should all play Monopoly again.’ He responded: ‘What’s the point? It’s impossible for anybody to win.’ I said: ‘The point is to be together.’ He stared through the windshield for a moment. I waited for his response. Finally, he said: ‘You know, that never occurred to me.’

I have a close friend whose favourite book as a child was The Runaway Bunny (1942) by Margaret Wise Brown, illustrated by Clement Hurd. Its first page reads:

Once there was a little bunny who wanted to run away. So he said to his mother: ‘I am running away.’

‘If you run away,’ said his mother, ‘I will run after you. For you are my little bunny.’

On each subsequent page, the little bunny fantasises about different ways in which he could transform himself and escape his mother. But, like a game of rock-paper-scissors, for each transformation the bunny proposes, his mother has a counter. ‘If you run after me… I will become a fish in a trout stream and I will swim away from you.’ ‘If you become a fish in a trout stream,’ says his mother, ‘I will become a fisherman and fish for you.’

On and on this game goes, until ultimately the bunny proposes turning into a little boy and running into a house. ‘If you become a little boy and run into a house… I will become your mother and catch you in my arms and hug you.’ At which point the bunny replies: ‘Aw shucks… I might just as well stay where I am and be your little bunny.’

The book was a map of his relationship with his mother for years to come

I never thought much about the meaning of The Runaway Bunny until I learned that it was my friend’s absolute, bar-none, ask-for-it-every-night favourite book as a child– at which point I burst out laughing. No book could suit him more. Where his mother was concerned, he was always a rebel. Once, when he was sent to his room as a young boy, he stood at the top of the stairs and shouted at his mother: ‘I have a penis and you don’t!’ As a young adolescent, well before he was legally allowed to drive, he ‘borrowed’ his parents’ car in the middle of the night and drove from the suburbs into New York City.

On a recent Mother’s Day, he gave his mother a card that said: ‘I don’t have to give you anything because I know you’ll always love me.’ His mother burst into grateful tears. Whatever you think of that card as a Mother’s Day gift (had I been his mother, I would have been tempted to slap him), the mother-son relationship in The Runaway Bunny describes the dynamic between my friend and his mother pretty well. The book is a map, incomplete of course, of his relationship with his mother for years.

To be clear, I am not claiming that someone’s entire personality can be explained by looking at her childhood library. But often we can, cautiously, gain insight into someone’s personality by analysing the books she loved as a child. With even more caution, we can expand the scope of this analysis to point toward an answer to the question: How does a book get on, and stay on, a bestseller list? What makes thousands of people love a book enough to initiate that magical, process of word-of-mouth, catapulting it onto the list and keeping it there? Perhaps it is the book that speaks to the inner psychic landscapes of large swaths of the American public.

I don’t have the figures to prove it, but I would guess that the most popular children’s story in the world is Cinderella. Adaptations and retellings of it are ubiquitous: if I tried to list its adaptions in film and literature just over the past decade, I might just break the internet. Cinderella has iterations in almost every culture, from Ancient Egypt and China to 18th-century France.

This should come as no surprise. The story of Cinderella is basically that of a child unnoticed and unvalued by peers and parent-figures. Her ‘real’ parent-figure (in the French version it’s her fairy godmother; in Grimm, it’s the spirit of her dead mother) shows up and enables her to unlock her latent worth, proving the naysayers wrong and allowing her to achieve the greatness she deserves.

Most children feel undervalued sometimes. Many feel impotent and unnoticed. And plenty believe that, if only they were seen clearly, or if only they had an opportunity, they could prove that they are more valuable, worthwhile, beautiful, talented or strong than anyone knew. Everyone, at some point in her life, has felt like Cinderella. And many people feel like Cinderella all the time.

So some people, mostly little girls (thanks to the gendered way that the story is usually told), will identify Cinderella as their favourite story. But many people won’t. Instead, they’ll mention Harry Potter, or Star Wars, or any of the dozens and dozens of Cinderella stories that dominate our bestseller lists and box offices.

Let’s look at Harry, from J K Rowling’s Harry Potter and the Sorcerer’s Stone (1997). We know from the outset that he is ‘the boy who lived’, who survived an attack of the darkest magic from the world’s greatest dark wizard and somehow managed, as an infant, to vanquish that wizard. So he’s special. Very special. But no one knows it, because he’s being raised by an ignorant aunt and uncle, along with their brutish son (cf stepmother and stepsisters). But soon, someone comes to rescue him, to take him to the place he’s always meant to be – Hogwarts School of Witchcraft and Wizardry.

There is a wonderful passage in which the half-giant Hagrid, who is rescuing Harry from his horrible aunt Petunia and uncle Vernon Dursley, educates Harry about himself:

‘Do you mean ter tell me,’ [Hagrid] growled at the Dursleys, ‘that this boy – this boy! – knows nothin’ abou’ – about ANYTHING?’

Harry thought this was going a bit far. He had been to school, after all, and his marks weren’t bad.

‘I know some things,’ he said. ‘I can, you know, do maths and stuff.’

But Hagrid simply waved his hand and said: ‘About our world, I mean. Your world. My world. Yer’ parents world.’

‘What world?’

Hagrid looked as though he was about to explode.

‘DURSLEY!’ he boomed.

Uncle Vernon, who had gone very pale, whispered something that sounded like ‘Mimblewimble.’

Hagrid stared wildly at Harry.

‘But yeh must know about yer mum and dad,’ he said. ‘I mean, they’re famous. You’re famous.’

‘What? My – my mum and dad weren’t famous, were they?’

‘Yeh don’ know… yeh don’ know…’ Hagrid ran his fingers through his hair, fixing Harry with a bewildered stare.

‘Yeh don’ know what yeh are?’ he said finally.

Uncle Vernon suddenly found his voice.

‘Stop!’ he commanded, ‘stop right there, sir! I forbid you to tell the boy anything!’

A braver man than Vernon Dursley would have quailed under the furious look Hagrid now gave him; when Hagrid spoke, his every syllable trembled with rage.

‘You never told him? … You kept it from him all these years?’

‘Kept what from me?’ said Harry eagerly.

‘STOP! I FORBID YOU!’ yelled Uncle Vernon in panic.

Aunt Petunia gave a gasp of horror.

‘Ah, go boil yer heads, both of yeh,’ said Hagrid. ‘Harry – yer a wizard.’

When I first read this passage, I was tutoring a third-grade boy in East Harlem, trying to get him interested in books. Harry Potter had yet to become the phenomenon, at least in the United States, that it would become. I picked it from a bookshelf and read him the first chapter. He wasn’t interested. But after he left, I kept reading, and when I got to this passage, I cried. Tears were streaming down my face in the tiny library of the East Harlem tutorial programme.

The passage still makes me cry. It is Rowling at her best, confirming the promise of Cinderella, confirming the unrecognised (but subconsciously felt) greatness inside the child. Rowling is a genius, and her books will one day be in the ‘perennial bestseller’ class with the Bible, because she tells the Cinderella story so well.

When you see an adult who adores Harry Potter, you are likely speaking to someone whose Cinderella fantasy is to transform from a social outsider into a wizard

In the Dursley house, Harry is oppressed by his aunt, uncle and cousin’s cruelty, just as Cinderella is by the cruelty of her stepmother and stepsisters. But in a brilliant adaptation of the Cinderella trope, Harry is also oppressed by the Dursleys’ normality. Harry Potter and the Sorcerer’s Stone opens with the line: ‘Mr and Mrs Dursley, of Number Four Privet Drive, were proud to say that they were perfectly normal, thank you very much.’ It is the Dursleys bloody-minded devotion to all things normal and conventional that makes them hate Harry so much. He is unable to conform because he is special – his magical powers keep manifesting, inadvertently and even unconsciously, driving Vernon and Petunia crazy, and prompting them to punish him with increasingly harsh measures.

Failure to conform is hated. Specialness is hated. Failure to conform and specialness become one. This is the magical adaptation of Harry Potter to the modern world. When you see an adult who adores Harry Potter, who proudly tells you what Hogwarts house she is in, and explains to you the method for determining your own, you are likely speaking to someone who has felt oppressed by the conventionality of her world, and whose Cinderella fantasy is not transforming from an overlooked child into a princess, but rather transforming from a social outsider into a wizard. This is part of the deep psychic appeal of Harry Potter.

Another part of the appeal is the role of parents. In many versions of the Cinderella story, her fairy godmother gives her the clothes she needs to wear to the ball. In the Brothers Grimm version, the magical helpmate is the spirit of Cinderella’s real mother, embodied in a tree growing out of her mother’s grave. In either case, there is a superficial parent figure (the stepmother), who does not see the true value of the child, and a ‘real’ parent figure, who does. Harry’s superficial parent figures are, obviously, the Dursleys. Harry has many ‘real’ parent figures through the course of the series, from his stern stand-in mother Professor McGonagall to his inspiring but distant stand-in father, Albus Dumbledore. But many of the most potent emotional payoffs in the series are when Harry’s real parents return in spirit, like Cinderella’s mother: to protect him, to reveal gifts they gave him long ago or, most movingly of all, to be proud of him. The Harry Potter series is not just for social misfits. It’s for anyone who longs for a parent’s recognition and love.

Cinderella stories appeal to people of all ages. But, as we grow, we need more story types. New psychic pressures come to bear and, to cope with them, we crave new forms of tales. Taking a look at the bestseller list for the past five years reveals that teenagers, or ‘Young Adults’ as they’re called in the biz, gobble up dystopian fiction. Veronica Roth’s Divergent series (2011-13) has done battle with James Dashner’s Maze Runner series (2009-16) at the top of a list which has seen The 5th Wave (2013-16), Legend (2011-13) and others periodically dethrone them. But regardless of which title is on top this week, the bestselling series list has been dominated by dystopian fantasies.

To some degree, dystopian novels have been successful for the better part of the past century. Perhaps the most widely read and beloved is 1984 (1949) by George Orwell. Aldous Huxley predicted the future more accurately in Brave New World (1932) – where an authoritarian control structure is rendered unnecessary by the population’s desire to self-sedate (he predicted a drug called ‘soma’; turns out it was YouTube). But, Orwell has captivated and inspired untold millions, even after the fall of his bogeyman, the Soviet Union. His ideas have infiltrated the language. ‘Orwellian’ has become synonymous with dystopian authoritarianism. I have twice witnessed the effect that 1984 can have on a high-school class: first as a student myself and then again as a teacher. Its impact goes beyond political critique. It hits at the psychic level.

You can see why. 1984 is the story of a man, Winston, who is hemmed in on all sides by an authoritarian power. His life is observed and constrained. His thought is observed and constrained. The power structure is trying to teach him to replace the language of his upbringing and the literature he loves with a language that they have contrived in order to control him. What teenager does not feel each of these things? Parents observe and constrain their lives. Peers observe and constrain their thoughts. Teachers try to replace the language of their youth with ‘appropriate’ and ‘educated’ language that operates by rules which strengthen the power of the teachers themselves.

Depressing, perhaps cynical – but teens see their very lives scribbled on the pages of 1984

Winston’s rebellion is private and introverted – trying to steal moments away from the telescreens to write in his secret diary, like a teenager putting on her headphones and blasting her music behind a locked bedroom door. Until, that is, Winston begins an affair with Julia. It is only through romance that his rebellion comes to vivid, dangerous life. How completely adolescent! Only when a boyfriend or girlfriend shows up can real independence begin. 1984 is effective as a teen novel not because of its political message, but because it dramatises the internal psychic struggle of growing up.

What makes 1984 immortal is that it ends with a real insight into the futility of teenage rebellion. Every teen becomes an adult, joins the machine, and comes to love Big Brother. Teens hate this fact – many of them hate the ending – but it stays with them, because they know it’s true. In 1984’s sibling, Animal Farm (1945), the animals with whom we identify see their ultimate betrayal: the pigs standing like their former human masters in their former masters’ house. But in 1984 it is Winston himself who gives in at the end. He does love Big Brother, because Big Brother, also known as adult society, has manipulated language, power and thought to perpetuate its system. By being born into it, we have already lost. Depressing, perhaps cynical – but teens see their very lives scribbled on the pages of 1984. It’s more fun to read a dystopian novel with a happy ending. But the novel that you pass on to your children is the one that first taught you the truth.

When a child asks for the same book three hundred times, she is telling her parents what she needs to learn, what she needs to come to terms with. Adults do the same thing. Books are psychologists, using imagination therapy to elicit secrets that their readers did not know they kept. A child does not realise what he reveals when he names this doll ‘mummy’, that one ‘daddy’, and then has them shout at one another. Nor do we tend to realise what we are revealing about ourselves when we push a book into the hands of three friends. Maybe the bestseller list, stripped of the fly-by-night entries and dopamine drips, is a snapshot of the national psyche. It might be telling us what we need to learn, what we are coming to terms with. More certainly, I know one thing: I hope my daughter doesn’t love The Carrot Seed as much as I did.