It’s been a decade since Lehman Brothers collapsed, careening America deeper into the Great Recession that led nearly 9 million Americans to lose their jobs. Many politicians and economists tout our strong recovery since the financial and housing collapse and lost jobs, with the official unemployment rate tumbling to 3.9 percent. But the proportion of men who were working last month was more than 4 percentage points lower than it was a decade ago.

It’s not news that the percentage of men working has been declining for decades. The male labor force participation rate — the term of art used by economists — declined by about 5 percentage points in the 28 years before the Great Recession. But that rate fell sharply after the financial crisis, dropping by nearly the same amount in the decade following. Now, about 69 percent of men are part of the labor force compared to about 86 percent in the early 1950s. Including the unemployed, just 66 percent of men are working. (Meanwhile, the rate for women, which had steadily increased until 2001, is now 3 percentage points lower than at its peak, and 2.4 points lower since Lehman declared bankruptcy.)

The idea that our economy no longer has a place for a certain slice of the American male population has fully permeated our national consciousness. Working-class white men, often without decent (or any) jobs, have been endlessly discussed as the constituency that propelled Donald Trump to the White House. But the reality is much more complex than that. Non-working men of all races, millennials, over 50, middle- and upper-middle-class men, and formerly incarcerated men all make up this group.

I talked to hundreds of them, as well as the women in their lives and the professionals trying to help them, for my new book, Man Out: Men on the Sidelines of American Life, to understand who these men are, why this has happened, and what happens when men don’t work. I came away convinced that not working has had and will continue to have profoundly negative effects on not only these non-working men but our entire society.

My conversations with non-working men

All told, as many as 20 million men between their early 20s and late 60s, plus 2 million men behind bars, are neither in school nor retired nor working.

Take Lorne, a middle-aged man in the Midwest with two degrees, who lost his job in information technology during the Great Recession (I changed all the names in this article to preserve anonymity). He hasn’t “dropped out” of the workforce, he said emphatically: “I’ve been kicked out.” When the subject of available low-wage jobs came up, he said that he’s not “culturally suited for them.” Lorne is angry — at human resources departments that he says “discriminate” against him, as well as toward women, journalists, and society in general.

Lorne may fit the stereotype of many Trump-voting men. But I also spoke with guys like Yates, a former partner at a law firm in the Northeast who quit his job in his 40s. Although he became a stay-at-home dad, he did not look for work, instead depending on his soon-to-be ex-wife’s income. He got into trouble with the IRS, abused prescription drugs, and became addicted to pornography; eventually his wife divorced him. He is one of many in the well-educated, relatively well-to-do category.

There are plenty of young men who aren’t working either — of civilian, non-institutionalized American men between the ages of 21 and 36, at least 21 percent aren’t working compared to just over 12.5 percent between the ages of 35 and 54. Ben, an unemployed 20-something in Maine, lives with his parents, who “scream and yell about what will become of me,” he said. He feels ashamed and depressed, and spends his days on the internet or playing his trumpet and harmonica.

Mark, a young man in Arkansas, repeatedly dropped out of college and quit the odd jobs that he took to get by. He currently isn’t working and isn’t looking to work. Both of them spoke to me about fears that they would never get on a career track or find a life partner. “A long-term relationship is not worth my trouble,” Ben said. “Marriage is dead to me.”

I also spoke to many women whose husbands didn’t work. Many of them told me about not only increased emotional and financial stress but also problems including alcoholism, porn addiction, and increased anger from the men in their lives.

Then there were men who fit into an often-overlooked category of the unemployed: formerly incarcerated and currently incarcerated men. Jack, who was released from a New York prison at age 51 after spending 15 years behind bars and having lost many of his prime working years, is now looking for jobs without success. Due to decades of mass incarceration policies, he is one of about 17 million to 20 million formerly incarcerated men in the United States.

Despite “ban the box” laws requiring employers not to ask potential employees if they were ever in prison, it is easy for human resources departments to discover these men’s pasts. As Jack said, a company that had posted a job opening found about his record and told him, “We’re not really looking for anyone.”

Whatever the cause, non-working men are an important problem for society

Did these men drop out, or were they pushed out of the workforce? This is hotly debated. Progressives cite stagnant male wages, automation, and outsourcing “pushing” them out. Conservatives tend to see a moral dimension: The work ethic has declined, men have “dropped out,” and they turn to government benefits, wives, and families to support them. Both explanations are partly correct, although the degree to which one or the other is the major factor differs among individual men.

Regardless of the reasons, having one of the lowest male labor force participation rates among developed countries has many consequences. Given traditional beliefs about “masculinity,” men who fail to “provide” too often suffer a host of psychological ills. Non-working men are much more likely to overdose on drugs — men account for two-thirds of the three-quarters of a million deaths from all types of drug overdoses from 1999 to 2017, according to the Centers for Disease Control and Prevention. They’re more likely to die by suicide too. More than 33,000 men died by suicide in 2014, a nearly 50 percent increase since the late 1990s and 3.5 times the number of women who died by suicide in the US, the CDC reported.

The absence of 20 million male workers translates, by some calculations, into more than $1 trillion a year in annual GDP. Since we’re talking about some $600 billion in wages not paid to these men — and remember that the wage gap means that the lost earnings of male workers are higher than that of female workers — it means up to $90 billion lost in Social Security and Medicare taxes alone.

It is troubling, while overall marriage rates decline, that not being married correlates with a 5 percentage point lower labor force participation rate for men between the ages of 25 and 54. President Obama’s last surgeon general, Vivek Murthy, called loneliness America’s biggest public health hazard: One-third of men between ages 15 and 64 lived alone in 2012, 50 percent higher than in 1970, while the proportion of women living alone has remained fairly stable. This differential is largely caused by the sharp increase in numbers of men not living with minor children.

A decade out from the financial crisis of 2008, non-working men are one of the groups that continue to suffer hidden from view. On the policy side, we need to equalize access to a good education, increase training and apprenticeships, raise wages or provide wage subsidies, and expand unemployment insurance and other parts of our safety net.

Culturally, we need to teach values like responsibility and hard work, and reduce the tolerance in society and popular culture for “bad” behavior. We cannot descend into the maelstrom that the Roman historian Livy called a time “when we can endure neither our vices nor their cure.”

Andrew L. Yarrow, a journalist and historian who has also worked in public policy, is the author of the new book Man Out: Men on the Sidelines of American Life, which addresses many of the challenges and problems of millions of American men.

First Person is Vox’s home for compelling, provocative narrative essays. Do you have a story to share? Read our submission guidelines, and pitch us at firstperson@vox.com.