I have an unfortunate tendency to falter at crucial moments. For instance, standing at the altar in a church in Vermont, waiting for my wife-to-be to come down the aisle to marry me, I start to feel horribly ill. Not just vaguely queasy, but severely nauseated and shaky – and, most of all, sweaty. The church is hot that day – it's early July – and many people are perspiring in their summer suits and sun-dresses. But not like I am. As the processional plays, sweat begins to bead on my forehead and above my upper lip. In the wedding photos, Susanna is glowing; I am glistening. By the time she joins me in the front of the church, rivulets of sweat are running into my eyes and dripping down my collar. We turn to face the minister. Behind him are the friends we have asked to give readings, and I see them looking at me with concern. What's wrong with him, I imagine they are thinking. Is he going to pass out? Merely imagining these thoughts makes me sweat even more. My best man, standing a few feet behind me, taps me on the shoulder and hands me a tissue to mop my brow.

The wedding readers' facial expressions have gone from registering mild concern to unconcealed horror: Is he going to die? I'm beginning to wonder that myself. For I have started to shake. I don't mean slight trembling – I feel like I'm on the verge of convulsing. I am concentrating on keeping my legs from flying out from under me and am hoping that my pants are baggy enough to keep the trembling from being too visible. I'm now leaning on my almost wife and she is doing her best to hold me up.

The minister is droning on; I have no idea what he's saying. (I am not, as they say, present in the moment.) I'm praying for him to hurry up so I can escape this torment. He pauses and looks down at my betrothed and me. Seeing me – the sheen of sweat, the panic in my eyes – he is alarmed. "Are you OK?" he mouths silently. Helplessly, I nod that I am. (Because what would he do if I said that I wasn't? Clear the church? The mortification would be unbearable.)

As the minister resumes his sermon, here are three things I am actively fighting: the shaking of my limbs; the urge to vomit; and unconsciousness. And I am thinking: Get me out of here. Why? Because there are nearly 300 people – friends and family and colleagues – watching us get married, and I am about to collapse. I have lost control of my body. This is supposed to be one of the happiest, most significant moments of my life, and I am miserable. I worry I will not survive.

Mercifully, the ceremony ends. Drenched in sweat, I walk down the aisle, clinging gratefully to my new wife, and when we get outside the church, the acute physical symptoms recede. I'm not going to have convulsions. I'm not going to pass out. But as I stand in the reception line, and then drink and dance at the reception, I'm pantomiming happiness. I'm smiling for the camera, shaking hands – and wanting to die. I have failed at one of the most elemental of male jobs: getting married. How have I managed to cock this up, too? For the next 72 hours, I endure a brutal, self-lacerating despair.

My wedding was not the first time I'd broken down, nor was it the last. At the birth of our first child, the nurses had to briefly stop ministering to my wife, who was in the throes of labour, to attend to me as I turned pale and keeled over. I've frozen, mortifyingly, onstage at public lectures and presentations, and on several occasions I have been compelled to run off stage. I've abandoned dates, walked out of exams, and had breakdowns during job interviews, on flights, train trips and car rides, and simply walking down the street. On ordinary days, doing ordinary things – reading a book, lying in bed, talking on the phone, sitting in a meeting, playing tennis – I have thousands of times been stricken by a pervasive sense of existential dread and been beset by nausea, vertigo, shaking, and a panoply of other physical symptoms. In these instances, I have sometimes been convinced that death, or something somehow worse, was imminent.

Even when not actively afflicted by such acute episodes, I am buffeted by worry: about my health and my family members' health; about finances; about work; about the rattle in my car and the dripping in my basement; about the encroachment of old age and the inevitability of death; about everything and nothing. At various times, I have developed anxiety-induced difficulties breathing, swallowing, even walking; these difficulties then become obsessions, consuming all of my thinking.

I also suffer from a number of specific fears or phobias. To name a few: enclosed spaces (claustrophobia); heights (acrophobia); fainting (asthenophobia); being trapped far from home (a species of agoraphobia); germs (bacillophobia); cheese (turophobia); speaking in public (a subcategory of social phobia); flying (aerophobia); vomiting (emetophobia); and, naturally, vomiting on airplanes (aeronausiphobia).

Scott Stossel at 10, the age at which he was first taken to a mental hospital for evaluation, then referred to a psychiatrist. Photograph: © Scott Stossel

When I was a child and my mother was attending law school at night, I spent evenings at home with a babysitter, abjectly terrified that my parents had died in a car crash or had abandoned me (the clinical term for this is "separation anxiety"); by age seven I had worn grooves in the carpet of my bedroom with my relentless pacing. During first grade I spent nearly every afternoon for months in the school nurse's office, sick with psychosomatic headaches, begging to go home; by third grade stomach-aches had replaced headaches but my daily trudge to the infirmary remained the same. On the only date I had in high school, when the young lady leaned in for a kiss, I was overcome by anxiety and had to pull away for fear that I would vomit. My embarrassment was such that I stopped returning her phone calls.

In short, I have since the age of about two been a twitchy bundle of phobias, fears and neuroses. And I have, since the age of 10, when I was first taken to a mental hospital for evaluation and then referred to a psychiatrist for treatment, tried in various ways to overcome my anxiety.

Here's what I've tried: individual psychotherapy (three decades of it), family therapy, group therapy, cognitive behavioural therapy (CBT), rational emotive therapy (RET), acceptance and commitment therapy (ACT), hypnosis, meditation, role-playing, interoceptive exposure therapy, in vivo exposure therapy, supportive-expressive therapy, eye movement desensitisation and reprocessing (EMDR), self-help work-books, massage therapy, prayer, acupuncture, yoga, Stoic philosophy, and audiotapes I ordered off a late-night TV infomercial.

And medication. Lots of medication. Thorazine. Imipramine. Desipramine. Chlorpheniramine. Nardil. BuSpar. Prozac. Zoloft. Paxil. Wellbutrin. Effexor. Celexa. Lexapro. Cymbalta. Luvox. Trazodone. Levoxyl. Propranolol. Tranxene. Serax. Centrax. St John's wort. Zolpidem. Valium. Librium. Ativan. Xanax. Klonopin.

Also: beer, wine, gin, bourbon, vodka and scotch. Here's what's worked: nothing. Actually, that's not entirely true. Some drugs have helped a little, for finite periods of time. Thorazine (an antipsychotic, which used to be classified as a major tranquilliser) and imipramine (a tricyclic antidepressant) combined to help keep me out of the psychiatric hospital in the early 1980s, when I was in middle school and ravaged by anxiety. Desipramine, another tricyclic, got me through my early 20s. Paxil (a selective serotonin reuptake inhibitor, or SSRI) gave me about six months of significantly reduced anxiety in my late 20s before the fear broke through again. Ample quantities of Xanax, propranolol, and vodka got me (barely) through a book tour and various public lectures and TV appearances in my early 30s. A double scotch plus a Xanax and a Dramamine can sometimes, when administered before take-off, make flying tolerable – and two double scotches, when administered in quick enough succession, can obscure existential dread, making it seem fuzzier and further away.

But none of these treatments have fundamentally reduced the underlying anxiety that seems woven into my soul and hardwired into my body and that at times makes my life a misery. As the years pass, the hope of being cured of my anxiety has faded into a resigned desire to come to terms with it, to find some redemptive quality or mitigating benefit to my being, too often, a quivering, quaking, neurotic wreck.

Anxiety and its associated disorders represent the most common form of officially classified mental illness in the United States today, more common even than depression and other mood disorders. According to the National Institute of Mental Health, some 40 million Americans adults, about one in six, are suffering from some kind of anxiety disorder at any given time. A study published in the American Journal of Psychiatry in 2006 found that Americans lose a collective 321m days of work because of anxiety and depression each year, costing the economy $50bn annually. In 2012, Americans filled nearly 50m prescriptions for just one anti-anxiety drug: alprazolam, the generic name for Xanax.

It's not just Americans who suffer from it. A report published in 2009 by the Mental Health Foundation in England found that 15% of people living in the United Kingdom are currently suffering from an anxiety disorder, and that rates are increasing: 37% of British people report feeling more frightened than they used to. A comprehensive global review of anxiety studies published in 2006 in the Canadian Journal of Psychiatry concluded that as many as one in six people worldwide will be afflicted with an anxiety disorder for at least a year during some point in their lifetimes.

Of course, these figures refer only to people, like me, who are, according to the somewhat arbitrary diagnostic criteria, technically classifiable as clinically anxious. But anxiety extends far beyond the population of the officially mentally ill. Primary care physicians report that anxiety is one of the most frequent complaints driving patients to their offices – more frequent, by some accounts, than the common cold.

Scott Stossel as a baby with his grandfather and great-grandfather, Chester Hanford (left), who also suffered from acute anxiety. Photograph: © Scott Stossel

Few people today would dispute that chronic stress is a hallmark of our times or that anxiety has become a kind of cultural condition of modernity. We live, as has been said many times since the dawn of the atomic era, in an age of anxiety. And yet, as recently as 30 years ago, anxiety per se did not exist as a clinical category. In 1950, when the psychoanalyst Rollo May published The Meaning of Anxiety, he observed that at that point only two others, Søren Kierkegaard and Sigmund Freud, had undertaken book-length treatments of the idea of anxiety. In 1927, according to the listing in Psychological Abstracts, only three academic papers on anxiety were published, and as late as 1950 there were only 37. The first-ever academic conference dedicated solely to the topic of anxiety didn't take place until June 1949. Only in 1980 – after new drugs designed to treat anxiety had been developed and brought to market – were the anxiety disorders finally introduced into the third edition of the American Psychiatric Association's Diagnostic and Statistical Manual of Mental Disorders, displacing the Freudian neuroses.

Today, thousands of papers about anxiety are published each year; several academic journals are wholly dedicated to it. Anxiety research is constantly yielding new discoveries and insights, not only about the causes of and treatments for anxiety but also about how the mind works –about the relationships between mind and body, genes and behaviour, and molecules and emotion. For instance, we now know, thanks to pioneering research by the neuroscientist Joseph LeDoux in the 1980s, that most fearful emotions and behaviours are produced by, or at least processed through, the amygdala, a tiny almond-shaped organ at the base of the brain that has become the target of much of the neuroscientific research on anxiety over the past 15 years.

We also know far more than Freud or Kierkegaard did about how different neurotransmitters – such as serotonin, dopamine, gamma-aminobutyric acid, norepinephrine and neuropeptide Y – reduce or increase anxiety. And we know there is a strong genetic component to anxiety. In 2002, to cite just one example among many hundreds, researchers at Harvard University identified what the media called the "Woody Allen gene" because it activates a specific group of neurons in the amygdala and elsewhere in the crucial parts of the neural circuit governing fearful behaviour.

And yet for all the advances brought by the study of neurochemistry and neuroanatomy, the psychological field remains riven by disputes over what causes anxiety and how to treat it. The psychopharmacologists and psychiatrists I've consulted tell me that drugs are a treatment for my anxiety; the cognitive behavioural therapists I've consulted sometimes tell me that drugs are partly a cause of it. Is pathological anxiety a medical illness, as Hippocrates and Aristotle and modern pharmacologists would have it? Or is it a philosophical problem, as Plato and Spinoza and the cognitive behavioural therapists would have it? Is it a psychological problem, a product of childhood trauma and sexual inhibition, as Freud and his acolytes would have it? Or is it a spiritual condition, as Søren Kierkegaard and his existentialist descendants claimed? Or, finally, is it – as WH Auden, David Riesman, Erich Fromm, Albert Camus and scores of modern commentators have declared – a cultural condition, a function of the times we live in and the structure of our society?

The truth is that anxiety is at once a function of biology and philosophy, body and mind, instinct and reason, personality and culture. Even as anxiety is experienced at a spiritual and psychological level, it is scientifically measurable at the molecular level and the physiological level. It is produced by nature and it is produced by nurture. It's a psychological phenomenon and a sociological phenomenon. In computer terms, it's both a hardware problem (I'm wired badly) and a software problem (I run faulty logic programs that make me think anxious thoughts).

Stossel with his uncle, John Stossel, an American TV journalist, and his sister, who also struggles with anxiety. Photograph: © Scott Stossel

I don't have to look far to find evidence of anxiety as a family trait. My great-grandfather Chester Hanford, for many years the dean of Harvard, was in the late 1940s admitted to McLean hospital, the famous mental institution in Belmont, Massachusetts, suffering from acute anxiety. The last 30 years of his life were often agony. Though medication and electroshock treatments would occasionally bring about remissions in his suffering, such respites were temporary, and in his darkest moments in the 1960s he was reduced to a foetal ball in his bedroom, producing what my parents recall as an inhuman-sounding moaning. Perhaps wearied by the responsibility of caring for him, his wife, my great-grandmother, a formidable and brilliant woman, died from an overdose of scotch and sleeping pills in 1969.

My mother, Chester's granddaughter, is, like me, an inveterate worrier who suffers from some of the same phobias and neuroses I do. She assiduously avoids heights (glass elevators, chairlifts), and tends to avoid public speaking and risk-taking of most kinds. Like me, she is mortally terrified of vomiting (and has not done so since 1974). My only sibling, a younger sister who is a successful cartoonist and editor, struggles with anxiety that is different from mine but nonetheless intense.

On the evidence of just these four generations on my mother's side (and there is a separate complement of psychopathology coming down to me on the side of my father, who drank himself into unconsciousness five nights out of every seven throughout much of my later childhood), it is not outlandish to conclude that I possess a genetic predisposition to anxiety and depression.

But is it not also possible that the bequeathing of anxiety from one generation to the next on my mother's side had nothing to do with genes and everything to do with the environment? In the 1920s, my great-grandparents had a young child who died of an infection. This was devastating to them. Perhaps this trauma, combined with the later trauma of having many of his students die in the second world war, cracked something in my great-grandfather's psyche. Perhaps my mother, in turn, acquired her own anxieties by witnessing the emotional anguish of her grandfather and by experiencing the anxious ministrations of her worrywart mother; the psychological term for this is "modelling". And perhaps I, observing my mother's phobias, adopted them as my own.

Or do the roots of my anxiety lie deeper and broader than the things I've experienced and the genes I've inherited – that is, in history and in culture? My father's parents were Jews who fled Weimar Germany in the 1930s. My mother's cultural heritage, on the other hand, was heavily Wasp; she is a proud Mayflower descendant who until recently subscribed wholeheartedly to the notion that there is no emotion and no family issue that should not be suppressed.

Thus, me: a mixture of Jewish and Wasp pathology – a neurotic and histrionic Jew suppressed inside a neurotic and repressed Wasp. No wonder I'm anxious: I'm like Woody Allen trapped in John Calvin.

Or is my anxiety, after all, "normal" – a natural response to the times we live in? The threat of cold war missiles that I feared as a child, of course, has now long since receded – but it has been replaced by the threat of hijacked planes, dirty bombs, underwear bombers, chemical attacks, and anthrax, not to mention Sars, swine flu, drug-resistant tuberculosis, the prospect of climate-change-induced global apocalypse, and the abiding stresses of a worldwide economic slowdown and of a global economy undergoing seemingly constant upheaval. In our post-industrial era of economic and social uncertainty is it not normal – adaptive even – to be anxious?

At some level, yes, it is adaptive to be reasonably anxious. According to Charles Darwin (who himself suffered from crippling agoraphobia that left him housebound for years after his voyage on the Beagle), species that "fear rightly" increase their chances of survival. We anxious people are less likely to remove ourselves from the gene pool by, say, frolicking on the edge of cliffs or becoming fighter pilots.

Scott Stossel at home in Washington DC. Photograph: Mike McGregor for the Observer

My life has, thankfully, lacked great tragedy or melodrama. I haven't served any jail time. I haven't been to rehab. I haven't carried out a suicide attempt. I haven't woken up naked in the middle of a field, sojourned in a crack house, or been fired from a job for erratic behaviour. As psychopathologies go, mine has been – so far, most of the time – quiet. Robert Downey Jr will not be starring in the movie of my life. I am, as they say in the clinical literature, "high functioning" for someone with an anxiety disorder or a mental illness; I'm usually quite good at hiding it. More than a few people, some of whom think they know me quite well, have remarked that they are struck that I, who can seem so even-keeled and imperturbable, would choose to write a book about anxiety. I smile gently while churning inside. I may seem calm. But if you could peer beneath the surface, you would see that I'm like a duck – paddling, paddling, paddling.

It has occurred to me that writing a book about anxiety might be a terrible idea: if it's relief from nervous suffering that I crave, then burrowing into the history and science of anxiety, and into my own psyche, is perhaps not the best way to achieve it. What is more, after more than 30 years of endeavouring to conceal my anxiety from people, here I am putting it on protracted exhibition for acquaintances and strangers alike.

My current therapist, Dr W, says there is always the possibility that revealing my anxiety will lift the burden of shame and reduce the isolation of solitary suffering. When I get skittish about airing my psychiatric issues in a book, Dr W says: "You've been keeping your anxiety a secret for years, right? How's that working out for you?"

Point taken. And there is a rich and convincing literature about how hiding or suppressing anxiety actually produces more anxiety. But there is no escaping my concern that this exercise is not only self-absorbed and shameful but risky – that it will prove the Wile E Coyote moment when I look down to discover that there is nothing to stop me from falling a long way.

"Why," Dr W asks, "do you think writing about your anxiety would be so shameful?"

Because stigma still attaches to mental illness. Because anxiety is seen as weakness. Because I worry that my revelations of anxiety and struggle will be a litany of Too Much Information, a violation of basic standards of restraint and decorum.

When I explain this to Dr W, he says that the very act of writing my book, and of publishing it, could be therapeutic. In presenting my anxiety to the world, he says, I will be "coming out". The implication is that this will be liberating, as though I were gay and coming out of the closet. But being gay – we now finally know (homosexuality was classified as a mental disorder by the American Psychiatric Association until 1973) – is not a weakness or a defect or an illness. Being excessively nervous is.

For a long time, governed by reticence and shame, I had told people who inquired about it that I was working on "a cultural and intellectual history of anxiety" – true, as far as it goes –without revealing its personal aspects. But a little while ago, in an effort to test the effects of "coming out" as anxious, I began gingerly to speak more forthrightly on what my book was about: "a cultural and intellectual history of anxiety, woven together by my own experiences with anxiety."

The effect was striking. When I had spoken about it as arid history, people would nod politely. But as I started to acknowledge the personal parts of the book, I found myself surrounded by avid listeners, eager to tell me about their own anxiety.

One night I attended a dinner with a bunch of writers and artists. Someone asked what I was working on, and I delivered my new spiel ("a cultural and intellectual history of anxiety, woven together and animated by my own experiences with anxiety"), talking about some of my experiences with various anti-anxiety and antidepressant medications. To my astonishment, each of the other nine people responded by telling me a story about his or her own experience with anxiety and medication. Around the table we went, sharing our tales of neurotic woe.

I was struck that admitting my own anxiety over dinner had dislodged such an avalanche of personal confessions of anxiety and pharmacotherapy. Granted, I was with a bunch of writers and artists, a population ostensibly more prone, as observers since Aristotle have noted, to various forms of mental illness than other people. So maybe these stories simply provide evidence that writers are crazy. Or maybe the stories are evidence that the pharmaceutical companies have succeeded in medicalising a normal human experience and marketing drugs to "treat" it. But maybe even more people than I thought are struggling with anxiety.

"Yes!" said Dr W when I ventured this proposition at my next session with him. Then he told me a story of his own: "My brother used to host regular salon evenings, where people would be invited in to lecture on various topics. I was asked to give a talk on phobias. After my lecture, every single one of the people there came up to tell me about their phobias. I think the official numbers, as high as they are, underreport."

Not everyone gets overwhelmed by anxiety. My wife, for one, does not. (Thank God.) Barack Obama, by all accounts, does not. Nor, evidently, does David Petraeus, the former commander of US forces in Afghanistan and former director of the CIA: he once told a reporter that despite being in jobs where the day-to-day stakes are a matter of life and death, he "rarely feels stress at all". (Perhaps he would have been better off feeling more stress – more worry about consequences might have prevented the adulterous misadventure that led to his downfall.)

Yet enough of us do suffer from anxiety that perhaps writing about my own ought not to be an occasion for shame but an opportunity to provide solace to some of the millions of others who share this affliction. And maybe, as Dr W often reminds me, the exercise will be therapeutic. "You can write yourself to health," he says.

My anxiety is a reminder that I am governed by my physiology – that what happens in the body may do more to determine what happens in the mind than the other way around. It reminds us that we are, like animals, prisoners of our bodies, which will decline and die and cease to be. (No wonder we're anxious.)

And yet even as anxiety throws us back into our most primitive, fight-or-flight-driven reptilian selves, it is also what makes us more than mere animals. "If man were a beast or an angel," Kierkegaard wrote in 1844, "he would not be able to be in anxiety. Since he is both beast and angel, he can be in anxiety, and the greater the anxiety, the greater the man." The ability to worry about the future goes hand in hand with the ability to plan for the future – and planning for the future (along with remembering the past) is what gives rise to culture and separates us from other animals.

For Kierkegaard, as for Freud, confronting this fear, and risking the dissolution of one's identity, expands the soul and fulfils the self. "Learning to know anxiety is an adventure which every man has to affront," Kierkegaard wrote. "He therefore who has learned rightly to be in anxiety has learned the most important thing."

Learning rightly to be in anxiety. Well, I'm trying.

© Scott Stossel.

Scott Stossel is editor of the Atlantic magazine. The above is extracted from his book My Age of Anxiety, published by Heinemann on 16 January (£20)

The caption to the picture second from bottom was edited on 8 Jan as it had incorrectly identified the man in the photograph as Stossel's father