The job market is a whole lot better than it was during the worst of the Great Recession, but around 7 million American men between the ages of 25 and 54 – mostly too old to be in school and too young to retire – are neither working nor looking for work. That amounts to 12% of all men in those prime working ages – and that doesn’t count another 2 million who are looking for work but haven’t found it.

The large number of prime-age men who are on the sidelines is a problem that that predates the Great Recession. The share of prime-age men who are working or looking for work has been falling for half a century. This issue has drawn the attention of Jason Furman, chair of President Obama’s Council of Economic Advisers. He came by The Hutchins Center at Brookings recently to talk with me about this.

You can watch a video of our conversation here:

Here are some highlights:

Who are these men?

They’re disproportionately low-income men who either dropped out of high school or didn’t go beyond a high school diploma.

What are they doing?

“They’re not spending any more time on child care, not spending any more time on chores. They are spending a lot more time watching TV than men who are in the labor force,” Furman says.

Why are there so many more of them even though the economy has bounced back?

Surely some of these guys don’t want to work or are getting by on their wives’ income. But Furman says less than a quarter of the men who aren’t in the workforce have a working wife and this fraction has been decreasing for 50 years. (It turns out not having a job or looking for one isn’t a great way to find a wife.) Some of these men are getting by on government benefits, such as Social Security disability. But the increase in men on disability isn’t nearly large enough to account for the growth in the ranks of men not at work. Related Content Report Men not at work The Avenue Where are the nonworking prime-age men? Social Mobility Memos What we know—and don’t know—about the declining labor force participation rate

Furman argues that one big reason these men aren’t working or looking for work because they’ve realized that employers aren’t interested in hiring workers with few skills or little education so they’ve given up.

Why should we worry?

Some men choose not to work and can afford not to. That’s great, Furman says. But for many, probably most, dropping out of the work force not only means a lack of income but also a loss of the dignity that comes with not working. What’s more, dropping out of the labor force is “associated with depression, with drug use, with suicide, with a range of bad outcomes,” he notes.

So the men would be better off if they were working, but so would the rest of us. As the baby boomers retire in large numbers, Furman says, “it’d be great to make up for that….to take people that are here in the United States that probably would be a lot happier working, and enable them to have a path to get job. That’d grow the workforce, increase our growth and help with all sorts of things like fiscal sustainability.”

What should we do about it?

Government policy can make a difference, Furman argues. Improving education and access to college could help by making workers more attractive to employers. So would spending more on helping people find jobs, as other countries do. Providing child-care subsidies and paid leave could draw more men – and women — into the workforce. Expanding the tax subsidies offered to low-wage workers so jobs are more attractive would be a plus, too.