Will Marshall is the president of the Progressive Policy Institute . The views expressed are his own.

(CNN) The collapse of marriage in our poorest communities -- and its tragic impact -- is a familiar story. But increasingly, marriage is becoming a marker of class privilege in America, something increasingly reserved for the affluent. If progressives want to tackle the scourge of inequality, then the retreat from marriage is an issue they can't ignore.

The reality is that the retreat from marriage is pervading the working middle class -- the two-thirds of Americans without a college degree. This is occurring even as in upscale America, marital bonds remain comparatively strong.

"This is the marriage gap, and it's something new in America," declares a manifesto on "marriage opportunity" unveiled in a recent Washington Monthly cover story . It was penned by four astute social and political analysts, David Blankenhorn, Jonathan Rauch, Barbara Dafoe Whitehead and Bill Galston. (Full disclosure: I'm a signer of their statement.)

"Over the past several decades, the norm of marriage has eroded across all economic and educational classes, but much less among the elite," they write. "But for millions of middle- and lower-class Americans, marriage is increasingly beyond reach, creating more fractured and difficult family lives, more economic insecurity for single parents, less social mobility for those on the lower rungs of the economic ladder, more childhood stress, and a fraying of our common culture."

True, overall U.S. marriage rates have fallen from 72% of U.S. adults in 1960 to just 51% in 2012, according to The Economist. But drill a little deeper into the data, and a marital class divide emerges. Less than half of men with high school degrees are married, compared with 76% of men with college degrees. The pattern is similar among women, except that those with graduate degrees have somewhat lower marriage rates than those with four-year college degrees. And because the college-educated tend to look for mates with similar education and earning power, their unions push them even higher up the income scale -- further widening the economic gulf between marital haves and have-nots.

Marriage in America, it seems, is becoming another luxury item for economic elites to enjoy. So why should we worry?

"Declining marriage rates may not be concerning on their own, but while Americans are forgoing marriage, they are not forgoing childbearing," say Isabel Sawhill and Joanna Venator of the Brookings Institution. More than 40% of children are born outside wedlock today (including nearly three-quarters of black children). And a raft of social research shows that children who grow up in married, two-parent families do better in school and in life than those who don't. It's true that more children are being raised by cohabiting couples, but in the United States (unlike some European countries) these relationships tend to be unstable and short-lived.

Some see the decline of marriage as stemming from a tectonic cultural shift -- the rise of feminism and the surge of women into work -- that has blurred the old division of familial labor between male breadwinners and female homemakers. But economic changes, mainly the disappearance of manufacturing jobs, automation and wage stagnation, have played a big part in the story, too.

Liberals are eager to talk about their favorite economic fixes for inequality: raising the minimum wage, bolstering unions and taxing the rich to pay for new social benefits for families of modest means. But they seem uncomfortable talking about marriage, which is seen as an unwelcome distraction from issues of economic power.

It's tempting to blame such reticence on President George W. Bush's ill-fated Healthy Marriage Initiative, which triggered a flurry of new federal marriage promotion programs over the previous decade. A centerpiece of Bush's "compassionate conservatism," the initiative managed to polarize the social policy community while having a negligible impact on falling marriage rates.

In truth, though, liberals have a long tradition -- at least since Daniel Patrick Moynihan's warnings in the 1960s of the consequences of family breakdown in black communities -- of shying away from "cultural" explanations for poverty and inequality.

They need to get over it. Inequality as we know it today arises from the intricate interplay of economic and cultural changes, and won't be reversed simply by redistributing income from affluent to downscale families. High marriage rates in upper middle-class America make the link between family structure and economic success abundantly clear. For example, economists Robert Lerman and W. Bradford Wilcox estimate that the median income of families with children would be 44% higher today if America had the same marriage rates we had in 1980.

Today, a record number of Americans -- one in five adults 25 and older -- have never been married. It's probably not because our society is evolving away from a fundamental social institution that has endured for, oh, most of human history. In fact, researchers say aspirations to marriage are as high as ever, but the practical barriers are getting harder to surmount.

Which brings us back to the "marriage opportunity" manifesto. Rather than actively promoting marriage -- something government has shown it doesn't really know how to do -- the opportunity agenda aims at removing impediments to marriage, especially for non-elite Americans. This includes a raft of familiar policy proposals for boosting the economic prospects of low-income workers and making taxes more family-friendly.

Intriguingly, the four authors point to the one cultural shift that's bucking the trend toward diminishing demand for weddings -- the public's dramatic turnaround on same-sex marriage. They envision a possible political bargain in which conservatives accept gays as allies in fortifying marriage, and liberals recognize that family breakdown is magnifying America's disparities of wealth and income.

Could expanding marriage opportunity really become common ground in America's intensely polarized culture wars? It seems like an extravagant hope. But Blankenhorn and company are emphatically right that both progressives and conservatives have a big stake in reversing the retreat from marriage.