Great-grandpa Neves – no one can remember his first name – is directing traffic on the Golden Gate Bridge, naked. By 1930, he has been in California, from his native Portugal, long enough – more than 15 years – to perform the hand signals on instinct: palm out front for “stop”; a backward wave for “move along”. To his schizophrenic eyes, the cars are an eyesore: they make the magnificent bridge look as if it crawling with ants. He has come to get them off.

This story is the centrepiece of Jason Tougaw’s chaotic, colourful family lore. Neves is the first of many complex characters that populate his family tree, and his saga on the bridge is the first of many dramatic tales. “Bad blood”, so it goes, flows through the family’s veins: it has made and squandered fortunes, led to drugs, alcoholism, fallen celebrity, mental illnesses, depression, abuse, prison sentences, divorces, and near‑fatal accidents. “My family feels comfortable living on the edge,” says Tougaw. “If things run too smoothly, they tend to sabotage them.”

It would make for a fascinating episode of Who Do You Think You Are?, but what if your life doesn’t resemble that of your family? What if you don’t identify with the folklore? And how, in this case, do you escape it?

A professor of literature at City University of New York, the author of two non-fiction books, in a loving, long-term relationship and with no police record, Tougaw might appear to have dodged his family’s recklessness. “When I was a kid, I thought I was from a different gene pool – the straight one among their chaos; I rebelled against everything my family embraced,” he says. When he moved to New York in 1993, he found himself telling people about his hippie childhood in California. “Distance loosened my lips,” he says. “I talked about my mother’s near abortion with me in Mexico, living on a converted school bus, growing up poor in the shadow of luxury. People kept asking: ‘Why aren’t you more fucked up? Why are you so different from the people who raised you?’ So, I wanted to get to the bottom of it, work out how I avoided the worst of their fates and crack our family history open. And, at the same time, try to answer the deeper, more universal question – how did I become me?”

‘I spent 10 years, from childhood into early adulthood, unable to speak the words mom or love’ ... Cathy and Jason Tougaw. Photograph: Jason Tougaw

Neves may have been the start of the family’s folklore, but it was Ralph who took it and ran. Ralph was his son, Tougaw’s grandfather, a charismatic, foul-mouthed jockey who found fame and fortune then squandered it. The press labelled him the Portuguese Pepperpot. His glamorous wife, Midge – Tougaw’s grandmother – was best friends with Betty Grable. She drove a pink T-Bird, a new one every year during the 50s and early 60s. In photographs, they are impossibly glamorous: boarding jets, posing at the races, her swathed in furs and pearls, him dapper in his racing silks, their three polished children in suits, ties and pretty dresses.

In 1936, during a race, Ralph was thrown from his horse and trampled. He was taken to hospital, declared dead and his body wheeled to the morgue. A friend, a doctor on duty that day, paid his private respects and, for reasons known only to him, administered a shot of adrenaline into Ralph’s heart. Ralph jumped up, delirious with shock, raced from the hospital and hailed a taxi to the racecourse.

This was the first of many narrow scrapes. “What’s my names spelled backward?” he would later say to his grandchildren. “Seven. Lucky Seven.” He lived his life at full speed. He had six wives; Midge was the second. “His recklessness wouldn’t have been half so alluring without his talent,” says Tougaw.

‘His recklessness wouldn’t have been half so alluring without his talent’ ... Jason Tougaw’s grandfather, Ralph. Photograph: Jason Tougaw

By the time Ralph and Midge’s children, Gary, Craig and Cathy – Tougaw’s mother – were in their late teens, they had rebelled against their formal, middle-American upbringing and embraced 60s California counter-culture with gusto – LSD, surfing and the rejection of wealth. Cathy got pregnant in 1968, aged 18, with her teenage boyfriend, the first of many abusive men who would enter and exit her and Tougaw’s life. Charlie, his father, was a junkie, and gone, by the time Tougaw could crawl.

Growing up, Tougaw wanted stability – and nothing to do with his heroin-addicted father, by now in prison. The very idea of Charlie terrified him. He begged his mother to send him to school. He dreamed of a “normal” life. He and his mother moved around southern California as homes, boyfriends and husbands – one of whom abused Tougaw – came and went. As a teenager, he witnessed his mother and her friends weighing out bags of cocaine on the dining table and was sworn to secrecy. But his grandmother, Midge (by now divorced from Ralph), was his rock. “Nanny was steadier and more stable than my mother,” says Tougaw. “I would look at her and think: ‘I want to be like that.’”

As a way of coping, he retreated into his mind. He suffered from panic attacks, nightmares, bed-wetting and hypochondria. He was bullied at school. He navigated his childhood with the help of friends, cousins, adolescent sexual experiments and the New Romantic scene he adored. But he was angry. “I spent 10 years, from childhood into early adulthood, unable to speak the words ‘mom’ or ‘love’,” says Tougaw.

At family get-togethers, accidents, bad luck, ill health and unfortunate choices in partners were all explained away: “It’s our Port-a-ghee blood,” they would say, referring to the colloquial immigrant word for Portuguese. The schizophrenia of his great-grandfather and his cousin Bryan was part of, but not all, the story. “We’ve constructed a robust, sensational family mythology – it excuses failings, even encourages bad behaviour,” says Tougaw.

Why was Tougaw one of the more fortunate ones? How did he rise above the pernicious family lore? Did he get lucky, genetically, by dodging the illness that plagued his great-grandfather and his cousin or is there another reason? “A few years ago, I read about psychological studies of resilience,” he says. “They focused on how a minority of maltreated children exceed expectations. They identified a variation in a gene, 5-HTT. I became convinced I possessed it. As a kid, I had a tendency to bob and float, dodging the worst predators.” For the rest of his family, he says, “our collective memory tells a story of resilience barely achieved. Great-grandpa Neves didn’t make it, but Ralph did. So did his children. We find ourselves in deep shit and we dig our way out. That’s the spine of our story.”

‘Nanny was steadier and more stable than my mother’ ... Tougaw’s grandmother, Midge, with Ralph. Photograph: Jason Tougaw

Today, Tougaw has a good relationship with his mother. “She keeps saying my memoir is ‘going to land her in the big house’ – an outdated term for prison.” His father died in 2011.

Writing his family’s story, says Tougaw, has had unexpected consequences. “I’m more reconciled with our differences. I’d even go so far as to say I am grateful for my genes. I’ve discovered our reckless strain in me more than I realised: we have a certain daring about us, a desire to be different, a creativity and an inherent distrust of rules and formal behaviour.” Coming out as gay, as Tougaw did in the 80s, would have been a lot harder in a family more wedded to conventional behaviour. “It was no big deal,” he says.

Were there any huge revelations? “I realised that your family history is part of you and you have to accept, rather than fight, that,” he says. “I thought I would suddenly figure everything out, but I didn’t. In fact, that’s the point: I’ve realised that you have to live with what you can’t figure out. I also realised I’m proud of our family lore – and I think I have been all along.”

• The One You Get by Jason Tougaw is published by Dzanc Books