‘Double-fault!’… ‘Ju-ust out!’ Generally a calm man, I throw my tennis racquet against the thick green court curtains with disgust… Akh, the torture of tennis.

I make a dejected approach to shake hands with my supercilious nemesis on the occasion of another easy victory. Maybe he feels sorry for me this time so, while towelling off, he reveals his secret: he has been reading tennis self-help books and ‘hasn’t lost a set since’. I feel outraged: it’s like he was doping, but with ideas.

I never read books that purport to help those who read them to help themselves. Pardon my ivory tower accent, but with a psychology PhD and classes full of (admittedly, glazed-over) undergraduates to teach, there is no way I am going to read a self-help book; those books are, well, ahem… slightly beneath me.

At least that’s what I thought walking home that afternoon but, after yet another loss the following weekend, a slightly different consideration entered my mind: if these tennis books are bestsellers, then is there a chance that all the players who defeat me in my local club league have read them too? Nah, not possible – don’t all self-helpers live in California where they unlock their chi energy and eat chia seeds? The guys at the tennis club don’t ‘talk feelings’ – it’s unlikely they would entertain such drivel.

It was after my fourth consecutive 0-6 loss that I drove directly to a used bookstore and went native. My name is Dr Gabriel, and I used self-help.

How does a person assess that something is not working in his or her life? And how does that person decide on the best way to fix it? In our secular age, there is a plethora of choices, from ancient wisdom (religious) traditions and New Age wish-fulfilment brochures to scientific popular psychology books by bona fide psychologists. On the surface, these three avenues seem very different, but are they all just forms of magical thinking?

When brain scientists and analytical philosophers retreat from the public sphere into their complex and mostly incomprehensible journal literature, there is a danger that the role of wise man falls to the charlatan and the demagogue, in other words, authors of self-help books. The world of ‘knowledge’ is split between an academy that does not acknowledge the soul – or the importance of falling in love – and a publishing industry that offers whatever emotional solace gets books sold. The contrasting knowledge industries offer differing visions of control: academic psychology is Promethean, seeking to explain the human condition by the virtues of rational and empirical truth-seeking, while popular psychology has Narcissus as its model, providing superstitious rituals that depend upon the purportedly limitless power of the self.

I turned to tennis self-help because I was anxious about the efficiency of my game. Ultimately, I wasn’t even sure why I was pursuing this torturous pastime at all. I knew there was something wrong with the way I played (duh!), and I wanted to believe there was a way to fix it. I sought wisdom and, in buying the books, I believed I’d found a way to help myself. Looking back, I think I achieved some control over my game by believing in its controllability. Perhaps this is the clue.

based on the bestsellers list, our aspirations constellate around becoming more efficient, falling in love, and making tons of dough

Self-help ­– including popular psychology – is a genre of books (not to mention seminars and videos) containing practical and ethical strategies for changing our behaviours and mental habits. They deliver principles that – with the requisite hope, practice and belief – promise to deliver control over our minds, which in turn helps us to achieve our aspirations. The self-help movement is, in many ways, a continuation of – and maybe a contemporary substitute for – ancient traditional wisdom. These books inspire and educate millions of people with personal anecdotes that read like profound moral fables. From this literature, we get formulas both prescriptive (‘you ought to stop picking your nose on first dates’) and descriptive (‘look how great Tony Robbins is, he never picks his nose’). In our post-Gladwell age of publishing, they increasingly include smatterings of statistics to keep the narrative grounded.

Recurring themes in popular psychology include: the utopia of therapeutic recovery; the importance of positive thinking; the attainability of happiness; and the possibility of transforming the self (I wonder if Jean-Jacques Rousseau’s estate gets royalties from the industry?). The goal of this literature seems to be a miraculous attainment of one’s sought-after aspirations – and, of course, a blissed-out, sustained, state of happiness. Along with my colleague Sean Victory, I undertook a review of 40 traditional self-help bestsellers in the United States, with a focus on books published since 2001; previous eras are well-covered in Steven Starker’s Oracle at the Supermarket (2002). If one were to make a blanket statement based on consulting the bestsellers list, our aspirations constellate around becoming more efficient, falling in love, and making tons of dough.

Whether critiques of this genre are delivered by scientific psychologists such as me, or jaded adolescent ironists, the usual criticisms of self-help have consistently been that it lacks external validation and casually disregards the importance of individual differences in the application of said panacea strategies. The exploitation of ignorance and hope within the medical and diet end of self-help is a particularly egregious example. Even so, this is an $11 billion industry in the US alone. We should take it seriously.

Looking to the written word for guidance is nothing new, of course: ancient ‘self-help’ such as the Tao Te Ching, the Analects of Confucius, the Bhagavad Gita, the Meditations of Marcus Aurelius, the Holy Quran and the Holy Bible loom large as tried-and-true one-stop wisdom shops. These books provide not only ethical prescriptions but also metaphysical descriptions of the origin and fate of life; in effect, ancient wisdom served as portable moral universes, in book form. Their messages speak to the human condition: our existential, psychological and emotional needs for hope and a sense of purpose and control.

It is no wonder then that popular psychology gained prominence in the US in the mid-20th century. As institutionalised wisdom traditions (including Christianity) began to play a diminished role in our culture, and alternative cosmologies that mixed and matched from indigenous and global traditions became more widely known, popular psychology started to replace the role traditionally held by figures of wisdom in one’s local community.

Self-help shadows the concerns of the people by matching appropriate rituals to prevalent anxieties. Similar to ancient wisdom traditions, it tends to rely on magical thinking; for example, the outsized strength of one’s own thoughts, or believing the universe will hear your wish. Even the newest self-help books do this, despite coming chock-full with empirical claims that purport to leapfrog narcissism and superstition through the unassailability of data. This scientific popular psychology reaches out in seemingly simple book titles that teach us how to quiet our mindset, so that we don’t choke when we feel the nudge to blink. After all, as other self-help titles quip, the paradox of choice is that, after reading books by people with PhDs, we must certainly find ourselves stumbling on happiness and feeling superbetter.

Popular psychology comes with a message about the possibility of surmounting obstacles through the free will of the almighty self, as well as continual exhortations to practise hopeful optimism and disable despair and hopelessness: in all, a reflection of an individualistic, even narcissistic, culture in the contemporary US. Thus, the differences between genres of popular psychology books can be secondary to the inspirational purpose they share: to enable individuals to gain more control over mind and career. In all cases, these books focus exclusively on the role of the individual in achieving self-mastery, while ignoring all other factors in a person’s fate, such as poverty, family conflict or political exclusion.

don’t take the authors’ word for it just because they are extra-spiritual, or manly, or a doctor who knows Oprah – listen because they are scientists and use data

As a scientist, I was taught that ignoring empirical aspects – such as facts, history and statistics – was superstitious behaviour; it meant ignoring the hard-won analytical tools of the modern age and reverting to magical thinking. This could take the form of personifying the weather and praying for rain, or believing in fate or some other abstract power. Fallacious connections entwined with hope and wish-fulfilment might be a part of the human condition, but in academic circles rational justification is the very marrow of quality.

In my study of self-help books, I decided to look more deeply into how sub-genres of popular psychology provide succour to our needs. Each is subtly different. Business self-help has the purpose of mediating professional relationships and fiscal ambitions. Love self-help offers manuals for intimate relationships and, secondarily, salves of hope for an end to loneliness within a national mythology of romantic love. New Age ministers to the need for metaphysical myths to give meaning and hope to daily life. Inspirational self-help curates self-confidence amid personal insecurity caused by social and economic pressures. Scientific popular psychology tends to deliver similar therapeutic advice for love, business and self-esteem purposes – although rather than take the authors’ word for it just because they are extra-spiritual (big ups, Deepak Chopra), or manly (hey-ho, Tony Robbins), or a doctor who knows Oprah (rake it in, Phil McGraw), we should listen because they are scientists and use data.

Each genre satisfies different needs: business books such as Robbins’s Awaken the Giant Within (1991) or David Allen’s Getting Things Done (2001) describe a format for success through organising one’s efforts and career moves efficiently towards attaining fiscal success. Meanwhile, love books such as John Gray’s Men Are from Mars, Women Are from Venus (1992) and McGraw’s Love Smart (2006) address emotions that arise in intimate relationships and the personal insecurities that keep us alone (and miserable to boot). New Age books such as Don Miguel Ruiz’s The Four Agreements (1997) or Eckhart Tolle’s The Power of Now (1999), on the other hand, speak directly to people who have turned their backs on the religious dogma they grew up with but still need both something to believe in and a community of believers. The inspirational genre exemplified by Joel Osteen’s Break Out! (2013) and Bruce Wilkinson’s A Life God Rewards (2002) weds traditional Christian values with the financial ambitions of business books (for some reason, this genre became more prevalent after 2001).

The mystical claims commonly put forward in the New Age and inspirational genres are an amalgamation of Judeo-Christian ethics and Hindu afterlife scenarios; Brian Weiss’s Many Lives, Many Masters (1988) is a clear example. Some claims said to be drawn from the East or Buddhism or Hinduism insist on the efficacy of sending one’s wishes out into the world, for example Rhonda Byrne’s The Secret (2006) and Chopra’s The Book of Secrets (2004). The metaphysical ideas presented in popular psychology – either explicitly in Chopra’s The Seven Spiritual Laws of Success (1994), or implicitly in Ruiz’s The Four Agreements – assume a philosophical position of idealism, namely that mind is stronger than matter.

The wisdom of popular psychology has three main tropes: that there is a power of thought; that there are effective habits for mental organisation; and that there are benefits to be gained from a self-help book. Maybe, as in the ancient Sanskrit Rigveda, rituals or incantations are themselves the embodiment of wisdom. In implementing a system – whether it be ‘the Secret’ or ‘the Tao’– belief becomes embodied in our actions, and so it is within this universal tradition of achieving a sense of control through manifesting beliefs in ritualistic thoughts and actions that popular psychology stakes out its position.

The power of thought is really an idealist claim concerning how proactive thought can change the world through putting the individual in touch with ultimate forces, such as the universe, god, past lives, the great potentiality, love, karma, enchantment, new levels of being or, simply, the true nature of reality. Prescriptions in this vein are generally derived from a syncretic collection of wise men from Buddha to Abraham Lincoln (and more recently, Steve Jobs), or past-life revelations, the wisdom of the ‘East’, miracles, pure faith, and enlightening shifts in the way we perceive the world around us. For example, my tennis book with a Zen Buddhist twist – The Inner Game of Tennis by W Timothy Gallwey (1974) – spent a disproportionate amount of time explaining how to sequester the insidious effects of thought, in particular the tendency to judge one’s own strokes, so as to allow the deeper parts of the self to act in Zen-tennis freedom.

The second message of popular psychology is that effective habits of organisation can help you to empower the self, influence people, work towards achievements, manage your time, improve your life circumstances, make money, and build character. Pseudo-Buddhist and Protestant practical strategies come up frequently, for example in the hardnosed economics of mental energies deployed in the business literature exemplified by early classics such as Dale Carnegie’s How to Win Friends and Influence People (1936) and Stephen Covey’s The Seven Habits of Highly Effective People (1989). There is an emphasis on the individual, and specifically on the existence of an element within each individual, usually referred to as the ‘subconscious’ or the ‘soul’, which is an invisible source of creative power that can be harnessed by using particular contemplative exercises.

All these habits require practice, diligence, trust, cognitive effort and, sometimes, belief in the revelations of an inspirational leader and the requirement to join (at least fiscally) his (it’s usually his) community of devotees. As in the first message, there is an element of wish-fulfillment here: that one’s belief in organising habits and enactment of the rituals of organisation will therein lead to control over the mind. For example, a popular tennis book I read – Brad Gilbert’s Winning Ugly (1993) – advised not only what to think about during important points and in between sets, but also included a chapter meticulously describing the importance of packing your tennis bag with bandages and extra socks.

Believing that there is an answer might be more important than whether the specific practice is actually successful

The third message, on the worth of therapeutic practices, emphasises how belief will ameliorate one’s self-confidence, help manage one’s sense of victimisation, and create a feeling of empowerment. There is a focus on the individual here, and an invitation to a journey of self-revelation that will ‘change your life’. This message aims to engender in the reader a belief that foolproof strategies exist to cure emotional wounds. A range of esoteric methods are presented, including proper meditation skills, learning how to break bad mental habits, doing the ‘primal scream’, remembering past lives, and achieving moments of transcendental revelation about one’s own character and traumas.

Scientific popular psychology relies upon the empirical method, whereby interviews with scientists – as in Malcolm Gladwell’s Blink (2005) – or piles of psychological experiments – as in Daniel Kahneman’s Thinking, Fast and Slow (2011) – buoy the author’s assertions. Above all, these practices require trust in the therapist or scientist and his or her method. The bestseller tennis books I encountered make a case for themselves when the authors detail whom they have coached and how successful they’ve been. Believing that there is an answer might be, at least emotionally, more important than whether or not the specific practice is actually successful. It is no surprise that, in Abraham Maslow’s hierarchy of needs, the need for safety and security (in this case, the belief that there is an answer) is prior to the need for self-actualisation.

Looking across traditional self-help, I found many prescriptions on how to manage one’s mental life to become an adaptable person and align one’s beliefs with new ways of behaving towards some larger purpose. Like the ancient Brahmanic books that dictate ritual processes to put the universe in order, popular psychology coached me into feeling more in control of my thoughts and actions through both therapeutic practices (‘maintain focus no matter what’) and metaphysical narrative (‘when thinking about why you missed a shot, you damage flow by separating mind from body’).

The broad claim is that one’s hopes and dreams – as materially substantiated in the form of job opportunities, romantic relationship partners, spiritual pacification, calming the demons of desire, and just plain making the most of life – can be fulfilled through simply thinking such wishes true. That sounds like magical thinking to me. Still, maybe magical thinking has some value. Could it even work?

Perhaps popular psychology is essentially the same as reading coffee grinds or listening to the advice of a wise minister. If success is conceived as a method to bridge the gap between aspirations and dissatisfactions, then it can be helpful. For many, the practical strategies and principles that constitute the philosophical and ethical core of self-help satisfy the need for hope and empowerment over self-assessed shortcomings. If the purpose of religious ritual is to validate one’s beliefs by satisfying the very emotions that motivate belief, then self-help is a remnant of religious thinking, and belief is truly the engine of magical thinking. Indeed, self-help and religion not only play similar roles as belief and ritual states; they also recruit common mental processes, such as emotions, social needs, and inhibitions.

Magical thinking might be helpful if you have a vague metaphysical issue to deal with, but if your issue is a lack of effort, guidance, training, imagination etc, then in most cases magical thinking will be, at best, useless. Ultimately, the utility of popular psychology might be as a placebo that confirms and provides rituals to enact the reader’s beliefs; in this light, popular psychology is the spiritual self-medication of our times. We know that when animals are placed in helpless positions within a laboratory setting they engage in activities that seem to be associated with a reward or schedule. They engage in this adjunctive behaviour as a way of managing their anxiety.

without putting time into the physical mechanics of tennis, I would have no chance, even if I was the Dalai Lama

The way we assess our lives and the ways we go about finding a cure are in lockstep; the form of the question holds its own answer. While the truth in religion for believers is the word of god, the truth of traditional self-help is locating and then satisfying the desires of the self. Despite its credentials, the new wave of scientific popular psychology is playing the same tune on a different instrument: the power of thought, the importance of organisation, and the reliability of the messenger remain the core messages. The truth now comes in details and data; god has been replaced by the physical universe that science can poke and prod, and the self is now ‘human nature’ to which we are all ineluctably bound. Maybe it is not self-help but rather our hopes of accurate self-assessment that are built around a core of magical thinking: that the will is stronger than the flesh.

Back on the courts, my head filled with tennis wisdom (‘Don’t judge your mistakes’, ‘Make the tennis ball your whole universe’, ‘Exploit his weakness’), and I was down a set and on match point. I wish I could say that my superstitious desire to win my service was successfully broadcast into the universe, or that I was able to watch the game transpire without becoming emotionally involved. But it wouldn’t be true. The fact is, I might have gained some insight into what the other player’s psychology could be, especially if he’d read the same tennis books as me, but my own play was plagued by the same nagging, insurmountable flaws as ever. Frankly, my strokes are not reliable, I have never had a coach, I don’t work with a ball machine. In short: I don’t practise enough.

In truth, without putting time into the physical mechanics of tennis, I would have no chance, even if I was the Dalai Lama. The basics are always more important, and therefore more time-consuming and rigorous, than the esoteric factors. Maybe we in the US need fewer sensational popular psychology books that feed our notions of the power of the self, and more boring, tedious, how-to manuals. While Prometheus steals fire from the gods, Narcissus just stares at his reflection, which unceasingly enchants but, in the end, teaches him nothing.