If I had to describe being shy, I’d say it was like coming late to a party when everyone else is about three glasses in. All human interaction, if it is to develop from small talk into meaningful conversation, draws on shared knowledge and tacit understandings. But if you’re shy, it feels like you just nipped out of the room when they handed out this information. W Compton Leith, a reclusive curator at the British Museum whose book Apologia Diffidentis (1908) is a pioneering anthropology of shy people, wrote that ‘they go through life like persons afflicted with a partial deafness; between them and the happier world there is as it were a crystalline wall which the pleasant low voices of confidence can never traverse’.

Shyness has no logic: it impinges randomly on certain areas of my life and not others. What for most people is the biggest social fear of all, public speaking, I find fairly easy. Lecturing is a performance that allows me simply to impersonate a ‘normal’, working human being. Q&As, however, are another matter: there the performance ends and I will be found out. That left-field question from the audience, followed by brain-freeze and a calamitous attempt at an answer that ties itself up in tortured syntax and dissolves into terrifying silence. Though this rarely happens to me in real life, it has occurred often enough to fuel my catastrophising imagination.

The historian Theodore Zeldin once wondered how different the history of the world might seem if you told it, not through the story of war, politics or economics, but through the development of emotions. ‘One way of tackling it might be to write the history of shyness,’ he mused. ‘Nations may be unable to avoid fighting each other because of the myths and paranoias that separate them: shyness is one of the counterparts to these barriers on an individual level.’ The history of shyness might well make a fascinating research project, but it would be hellishly difficult to write. Shyness is by its nature a subjective, nebulous state that leaves little concrete evidence behind, if only because people are often too uncomfortable with their shyness to speak or write about it.

For Charles Darwin, this ‘odd state of mind’ was one of the great puzzles in his theory of evolution, for it appeared to offer no benefit to our species. However, in research begun in the 1970s, the Harvard psychologist Jerome Kagan suggested that about 10-15 per cent of infants are ‘born shy’. Being easily fearful and less socially responsive, they reacted to mildly stressful situations with a quicker heartbeat and higher blood cortisol levels.

At around the same time, the American animal behaviourist Stephen Suomi, working at an animal centre in Poolesville, Maryland, observed a similar percentage of shyness in monkeys, with the same increased heart rate and rise in blood cortisol. Blood testing, and reassigning shy infant monkeys to outgoing mothers, suggested that this shy trait was hereditary. Suomi’s work might also have inadvertently pointed to the evolutionary usefulness of shyness. When a hole in the chain-link fencing around the centre’s primate range gave the monkeys a chance to get out, the shy ones stayed put while the bolder ones escaped, only to be hit by a truck when they tried to cross the road.

Until a few hundred years ago, life was lived far more in public: whole families would eat, sleep and socialise together in the same room

Higher primates are social creatures, hard-wired to want to meet and mate; but there might also be some value in their being cautious and risk-avoiding, traits that might over-evolve into excessive timidity. Neither Kagan nor Suomi suggest that shyness is fixed at birth. They see it as a case study in the rich interplay between nature and nurture. Similarly, for Antonio Damasio, professor of neuroscience at the University of Southern California, shyness is a ‘secondary emotion’. Unlike primary emotions such as anger, fear and disgust — where there is a large biological and universally felt component — shyness is ‘tuned by experience’, leaving it open to a huge amount of cultural conditioning, historical variation and definitional ambiguity.

If shyness is something that adjusts to different cultural and historical contexts, then it must surely have taken on oppressive new forms with the emergence of modern notions of privacy and private life. Until a few hundred years ago, life was lived far more in public. For example, it was quite normal for people to urinate or defecate in public places. Even in private houses, whole families would eat, sleep and socialise together in the same room. Then, gradually, bodily functions and aggressive language and behaviour were rendered increasingly invisible in polite society, thanks to what the late sociologist Norbert Elias called the ‘civilising process’ that took place in the Western world from the 16th century onwards. As greater physical and psychological boundaries grew up around individuals, particularly among relative strangers in public, there were more opportunities for awkwardness and embarrassment about when these boundaries should be crossed.

More recently, shyness, like other awkward personality traits, has been seen as an affliction to be treated medically rather than as a temperamental quirk. In 1971, the psychologist Philip Zimbardo conducted the Stanford Prison Experiment, with student volunteers acting as prisoners and guards in a pretend prison in the basement of the Stanford University psychology building. The study had to be stopped a week early because the guards were treating the prisoners so brutally, and many of the inmates had adapted by internalising their subordinate positions and sheepishly obeying their tormentors. Zimbardo began thinking of shy people as incarcerating themselves in a silent prison, in which they also acted as their own guards, setting severe constraints on their speech and behaviour that were self-imposed although they felt involuntary.

In 1972, Zimbardo began conducting the Stanford Shyness Survey, starting with his own students and eventually including more than 10,000 interviewees. The odd thing about Zimbardo’s work was that it revealed that feeling shy was very common — more than 80 per cent of those interviewed said they had been shy at some point in their lives, and more than 40 per cent said they were currently shy — but that it also pioneered the modern tendency to see shyness as a remediable pathology. Methods of calibrating shyness were developed, such as the Cheek and Buss Shyness Scale (after its Wellesley College researchers Jonathan Cheek and Arnold Buss) in 1981, and the Social Reticence Scale, formulated by the psychologists Warren Jones and Dan Russell in 1982. Extreme shyness was redefined as ‘social anxiety disorder’, and drugs such as Seroxat (also known as Paxil), which works like Prozac by increasing the brain’s levels of serotonin, were developed to treat it. As Christopher Lane argues forcefully in his book Shyness: How Normal Behaviour Became a Sickness (2007), this was part of a more general biomedical turn in psychiatry, with its ‘growing consensus that traits once attributed to mavericks, sceptics, or mere introverts are psychiatric disorders that drugs should eliminate’.

A small, self-regarding part of me thinks there is something glib about easy articulacy and social skill

In 1999, noting that the number of people identifying as shy in his survey had risen to 60 per cent, Zimbardo told the British Psychological Society that we were on the cusp of ‘a new ice age’ of non-communication. Computers, email and the replacement of cashiers and shop assistants by cashpoint machines and automated checkouts were all contributing to what he called an ‘epidemic’ of shyness as the possibilities for human contact diminished. Shyness, he suggested, was no longer an individual problem; it was now a ‘social disease’.

Today Zimbardo’s prediction of a new ice age created by technology seems wide of the mark. On the contrary, the rise of social networking has made it normal for people to lay bare their private lives without inhibition online, from posting photos of themselves in states of inebriation to updating the world on their changing relationship status, in ways that would have seemed inconceivable a generation ago. The internet, far from cutting us off from each other, has simply provided more fodder for our own era’s fascination with emotional authenticity and therapeutic self-expression — a shift in public attitudes towards personal privacy that Eva Illouz, professor of sociology at the Hebrew University in Jerusalem, has called ‘the transformation of the public sphere into an arena for the exposition of private life’.

In her recent book Quiet: The Power of Introverts in a World That Can’t Stop Talking (2012), Susan Cain worries about a world ruled by what she calls the ‘extrovert ideal’. This, she suggests, found its most malign expression in the excessive risk-taking of those who brought about the banking crisis of 2008. Much of Quiet consists of telling introverts how wonderful they are: how we think more deeply and concentrate better than extroverts, are less bothered about money and status, are more sensitive, moral, altruistic, clear-sighted and persistent. If you’re an extrovert, the book probably isn’t for you.

Yet introversion is not the same as shyness, as Cain is careful to point out, although the two do often overlap. Introverts are people whose brains are overstimulated when in contact with too many other human beings for too long — in which case I am most definitely a shy introvert. If I’m in a noisy group of people for more than about an hour, my brain simply starts to scramble like a computer with a system error, and I end up feeling mentally and physically drained. Introverts such as me need to make frequent strategic withdrawals from social life in order to process and make sense of our experiences.

Shyness is something different: a longing for connection with other people which is foiled by fear and awkwardness. The danger in simply accepting it, as Cain urges us to do with introversion, is that shyness can easily turn into a self-fulfilling persona — the pose becomes part of you, like a mask that melds with your face. There is always something we cling to in an unhappy situation that stops us escaping from it. In my case, it is the belief that lots of voluble people do not really listen to each other, that they simply exchange words as though they were pinging them over a tennis net — conducting their social life entirely on its surface. A small, self-regarding part of me thinks there is something glib about easy articulacy and social skill.

The human brain is the most complex object we know, and the journey from one brain to another is surely the most difficult

My more sensible self realises this is nonsense, and that shyness (or, for that matter, non-shyness) has no inherent meaning. There is nothing specific to shyness that makes you more likely to be a nice person, or a good listener, or a deep thinker. Shyness might have certain accidental compensations — being less susceptible to groupthink and more able to examine the habits and rituals of social life with a certain wry detachment, perhaps. Mostly it is just a pain and a burden.

Yet shyness remains a part of being human, and the world would be a more insipid, less creative place without it. As Cain argues, we live in a culture that values dialogue as an ultimate ideal, an end in itself, unburdening ourselves to each other in ever louder voices without necessarily communicating any better. Shyness reminds us that all human interaction is fraught with ambiguity, and that insecurity and self-doubt are natural, because we are all ultimately inaccessible to one another. The human brain is the most complex object we know, and the journey from one brain to another is surely the most difficult. Every attempt at communication is a leap into the dark, with no guarantee that we will be understood or even heard by anyone else. Given this obdurate fact, a little shyness around each other is understandable.

I have often found myself in a circle of people at a social gathering that has suddenly closed up like a scrum and left me standing outside it, as its constituent parts became animated in conversation, forgot I was there, and absent-mindedly nudged me out of the loop. I have fought all my life this sensation that shyness is a personal affliction that has left me viewing our herd-loving, compulsively communicative species from the edges. Now I am coming to see it more as a collective problem, an inevitable by-product of the thing that separates us from other animals: our unique human cargo of self-consciousness. For all our need for intimacy, we ultimately face the world alone and cannot enter another person’s life or mind without effort and difficulty. Shyness isn’t something that alienates me from everyone else; it’s the common thread that links us all.