Men have been steadily disappearing from the workforce for more than half a century.

In the 1950s, nearly every man in his prime working years was in the labor force, a category that includes both those who are employed and those actively applying for jobs. The "participation rate" for men ages 25 to 54 stood at 97.7% in early 1956, but drifted downward to a post-war record low of 88.4% at the end of 2012. (It ticked up very slightly at the start of this year to 88.6%.)

So where have all the men workers gone?

Some went into prison. Others are on disability. And still others can't find jobs and have simply given up looking.

The trend is particularly pronounced among the less educated. As the job market shifted away from blue-collar positions that required only a high-school degree to more skilled labor, many men were left behind, labor analysts say. It's harder these days to find well-paying jobs in manufacturing, production and other fields traditionally dominated by men without college diplomas.

But college men are leaving, too. The participation rate of those older than 25 and holding at least bachelor's degree fell to 80.2% in May, down from 87.2% in May 1992.

"The proportion of guys doing nothing has risen," said Gary Burtless, a senior fellow at the Brookings Institution.

The cycles of national economic prosperity since World War II have done little to stem this downward slide. In fact, even when the unemployment rate hit a 30-year low in the early 2000s, the share of men in the workforce continued its steady decline.

The Great Recession accelerated the trend, pushing the participation rate for men in their prime working years below 90% for the first time. It has yet to recover, even as the general economy improves.

Here's a closer look at the reasons why men are dropping out of the labor force.

Incarceration: A growing number of men have spent time in prison, which makes it much harder to find a job once they complete their sentences.

Looking at those born just after World War II, some 1.2% of white men and 9% of black men had been to prison by 2004, according to Bruce Western, a Harvard sociology professor. But looking at those born 30 years later, some 3.3% of white men and 20.7% of black men had been to prison.

Disability: More men have been pouring into the federal disability system, especially in recent years, when the Great Recession and its aftermath pushed up the national unemployment rate. Men who might have found a job in a better labor market were instead turning to this safety net system, according to David Autor, a Massachusetts Institute of Technology economics professor and co-author of Wayward Sons, which looks at the growing gender gap in education and the workforce.

In 1982, around 1.9% of working-age men were receiving disability benefits. By 2012, that number had climbed to 3.1%, according to data crunched by the National Academy of Social Insurance.

Once on the disability rolls, few people get off. Only 2.2% left the program in the first quarter of 2013.

Lack of education: A few decades ago, men could graduate high school and make a decent living on a factory floor or at a construction job. As the labor market becomes more skilled, those guys are being left behind.

In the 1960s, more men than women were enrolling in and completing college. "That's completely reversed itself," Burtless said. "Women's persistence and enrollment rates have given them a real edge over men."

Women born in 1975 were roughly 17% more likely than their male counterparts to attend college and nearly 23% more likely to complete a four-year degree, according to data in Wayward Sons.

"A lot of non-college men have chosen not to work rather than participate in jobs that don't pay that well and are not very satisfying," Autor said.

Related: College degrees with the best bang for your buck

The decline of men in the labor force has broad implications for families, taxpayers and the economy. Fewer employed men means more people on the dole and fewer taxpayers to contribute to the nation's economic growth.

Also, fewer of these men are in stable family relationships, contributing to growth of single-parent households. That fuels the widening income inequality gap and stunts the upward economic mobility of the next generation.