One of the most discussed problems in the U.S. labor market in recent years is the decline in the labor force participation rate, or the share of the population that’s employed or actively looking for a job. A big reason for this overall decline since 2000 is the aging of the U.S. population, but dig into the decline a bit more and there’s more to it than growing legions of retirees. The participation rate of prime-age workers, those 25-to-54 years old, also is falling. For women, this is a new trend in the 21st century, but for men the beginning of the decline dates back to the 1950s. Why are more and more men in their prime working years increasing not working? A new report from the President’s Council of Economic Advisers sheds some light on a potential answer, but first some context.

The U.S. labor force participation rate for prime-age men peaked in 1954 at just under 98 percent. The rate declined for the next decade before the decline accelerated around 1965. Over the next 50 years, the participation rate for these men declined by 8.3 percentage points as of last month. In May 2016, only 88.4 percent of prime-age men had a job or were actively looking for one.

What’s behind this decline? Economists divide the reasons into supply-side explanations (prime-age men are less willing to work) and demand-side explanations (employers are less willing to hire prime-age men). But the report by the Council of Economic Advisers finds relatively little evidence for the supply-side explanations. They find that increased reliance on programs such as Social Security Disability Insurance can explain, at the most, 0.5 percentage points of the 7.5 percentage point decline since 1967. And the share of men who are neither working nor seeking work and who also are married to a working spouse is low—less than 25 percent) and declining—meaning these men aren’t opting-out because they can rely on the earnings of their spouses.

Instead, the Council of Economic Advisers finds much more evidence for demand-side explanations. The decline in labor force participation is strongest among men with a high school degree or less (14 percentage points), which is also the group that experienced the most significant decline in their wages relative to other workers in the economy. Declining employment and declining wages indicates that demand-side issues are mostly to blame. Indeed, the report also shows that in states where the absolute return on work (wages at the low end) and the relative return on work (less low-end wage inequality) are higher, prime-age men are more likely to be in the labor force.

A decline in the demand for less-educated men in the United States seems to be part of longer-term trends such as technological change, globalization, or changing labor market institutions. What’s more, these same trends also have decreased demand for middle-skill workers, which in turn may have caused declining employment for low-skilled men. As these middle-skilled workers lost their jobs and moved into lower-paying occupations, they pushed less-educated men out of the labor force all together.

This “cruel game of musical chairs” can be seen in the wake of economic recoveries over the past five decades. The employment-to-population ratio for prime-age men has declined after every recession (except for one) since 1965 and never reached its previous high before the next recession. And the employment losses for prime-age men during a recession don’t bounce back as much amid each recovery. (See Figure 1.)

Figure 1

As the Council of Economic Advisers notes, the decline in male labor force participation is a sign of a longer-term decline in fluidity in the U.S economy. Less job creation by firms and less job movement by workers mean that the labor market is less able to bounce back from negative shocks. If the long-term trends mean less demand for middle and low-skill workers, then those at the bottom of the ladder have a hard time getting back on it. To help counteract some of the long-term trends then, policymakers should focus on strengthening the recoveries from recessions.