WASHINGTON (MarketWatch) — After riots broke out in Baltimore in April, the son of the owner of the Orioles baseball team famously lamented the loss of middle-class jobs that “plunged millions of good, hard-working Americans into economic devastation.”

The lack of good jobs did not light the bonfire of violence. The funeral of a 25-year-old black man who died in the custody of police was the spark. Yet the unrest was also fueled by the simmering frustration of many Americans who still cannot find decent jobs six years after the end of the worst economic calamity in the U.S. since the Great Depression.

While some graduates from the nation’s most prestigious universities still have trouble finding good-paying work, the problem is especially acute for young people who lack degrees or never finished high school, especially African-Americans. And even many millennials who have gotten jobs are starting out at lower salaries compared to previous generations or find themselves stuck in positions that don’t even require a degree.

See MarketWatch’s page for news and charts on jobs

The plight of young people helps explain why the economy continues to underperform as it enters the seventh year of a recovery. The last time the U.S. has grown 3% annually — the historic average is 3.3% — was a decade ago.

Millennials are more frugal than their parents, live longer at home after they get out of school, get married less, have fewer kids and are less apt to buy big-ticket items such as cars or homes. Large student debts for many of them add to the burden.

“That’s a big reason why we are locked into this modest-growth world,” said Scott Wren, senior global equity strategist at Wells Fargo Investment Institute in St. Louis. “A lot of these people need a halfway decent paying job.”

Although a big surge in hiring nationwide since 2014 has made it easier for young people to obtain jobs and form families, the labor market continues to shut out millions of Americans. Unless more work and attractive job opportunities become available to them, the economy could underperform for years to come.

“There is no doubt millennials face a very challenging jobs market,” said Robert Dye, chief economist of Comerica Bank in Dallas. “We still have a way to go to put all those people back to work.”

Getting off on the right foot

By most measures the economy is much improved compared to a few years ago. Younger people are able to find jobs easier, especially those with college degrees, and more “boomerangers” are moving out of their parent’s homes.

Read: 76% of new hires in May had one thing in common

Young people still don’t have it easy, though. The unemployment rate for 25- to 34-year olds, for example, rose higher and fell more slowly after the Great Recession than it did for any other age group. Even now, the 5.8% jobless rate for that group is still above the 5.5% national average.

Highly educated millennials — those born from the early 1980s to the early 2000s — have fared the best. Yet unemployment is still higher now compared to 2007 for people below 35, no matter their race, sex or educational level.

“ “If you have a bachelor’s degree, you are going to find a job. The question is are you going to find a job commensurate with your skills.” ”

Nor are those who found jobs guaranteed to start out at a high salary. A new study by the consultant firm Accenture shows that almost half of all college graduates in 2013 and 2014 consider themselves underemployed or working in a job that does not require a degree. And some 41% say they make less than $25,000 a year.

That should come as no surprise. Until very recently, more than half of all new jobs that had been produced since the recovery began were in industries such as retail and hospitality that pay less than the national average, a MarketWatch analysis shows.

“If you have a bachelor’s degree, you are going to find a job,” said Joe Brusuelas, chief economist at McGladrey, a business-consultant firm. “The question is are you going to find a job commensurate with your skills.”

Education no guarantee

The chance of finding a well-paying job is much harder for young adults without a college degree or technical training, a nearly 16 million strong group that still exceeds the number of people aged 25 to 34 who possess college diplomas.

The unemployment rate of 25-to 34-year olds with just a high school diploma averaged almost 9% in 2014, for example. It was an even higher 13% for those who never finished school, according to unpublished Labor Department data.

Part of the trouble is what some economists call the hollowing out of the labor market. Exposure to more foreign competition and fundamental shifts in key U.S. industries has eliminated millions of jobs that used to be a pathway to the middle class.

Take manufacturing and construction. Millions of U.S. manufacturing jobs have disappeared or moved overseas to countries such as China since the turn of the century. And a housing bust wiped out more than 2 million construction jobs from 2006 to 2010.

Although both industries have started to recover, they employ far fewer people than they did before the Great Recession even though the U.S. population has grown larger.

What’s worse, these jobs have moved away from the places that really need them.

Baltimore and nearby suburbs, for instance, have lost more than half of their manufacturing jobs in the past 25 years, and now the sector employs just 55,000 people directly. That doesn’t count the elimination of jobs to support those workers at retail outlets, local restaurants and so forth.

Baltimore’s unemployment rate is much higher than the national average — 8.1% in March vs. 5.5% in the rest of the country. It’s a lot higher for black men without a college degree who used to be able to count on blue-collar jobs to earn a decent living.

Hollowed-out middle

The shifting U.S. labor market points to another quandary for young people known as job polarization. The economy seems to have a bigger appetite for both high-skilled and low-skilled labor and less than it used to for workers in between.

“We have a tight jobs market for highly educated and skilled workers,” said Kate Warne, an investment advisor at the brokerage Edward Jones. “We have a lot of slack for unskilled workers.”

Economists say that might explain why so many college graduates initially accept jobs that don’t match their skills. And if more college graduates fill these positions, that leaves even fewer work opportunities for the less educated.

“ “There are not enough seats for everybody – it’s a game of musical chairs.” ”

The inability of the U.S. economy to employ nearly everyone who want a jobs underscores another problem: The increase in the number of people who’ve dropped out of the workforce altogether.

Last year the so-called participation rate of 25- to 34-year olds in the labor market fell to a 32-year low. For men in that age group, it hit the lowest level ever.

“Job polarization could be driving this,” noted Maria Canon, an economist at the St. Louis Federal Reserve who has studied the issue. “Current jobs demand different skills than in the past.”

By contrast, more people near retirement age actually went back to work because the Great Recession threatened their nest eggs. The percentage of people ages 55 to 64 in the labor force even rose briefly to an all-time high.

These seemingly divergent trends are not unrelated. Older workers staying on the job can block the way for younger people.

“A lot of older folks need to work longer to rebuild their savings,” said Scott Anderson, chief economist at Bank of the West. “There are not enough seats for everybody — it’s a game of musical chairs.”

The road ahead

The consequences of an underperforming labor market can cause lasting damage. Studies show that people starting out under more dire straits earn substantially less over their lifetimes compared to those who entered the labor market in good economic times.

Financial stress also influences decisions on major events in life. A Pew study in 2014 found that one-third of people ages 25 to 34 cited financial insecurity as the reason they weren’t married. Older generations were less troubled by money.

Younger Americans find it easier to get a job these days, but going to work everyday is not the end of their struggles. Getty Images

Yet millennials are not necessarily doomed to live in an age of diminished expectations.

For one thing, the economy could eventually regain its usual vigor, as it’s repeatedly done in the past. The number of jobs created in 2014 was the highest in 15 years and the U.S. added a robust 280,000 jobs in May. That suggests the economy has regained momentum after a slowdown earlier in the year.

What could also shift the U.S. into a faster gear are new, ground-breaking technologies or a marked political shift in Washington that improves the climate for business.

Above all, a wave of retirements of baby boomers will increasingly open the door for younger workers. Rising demand for millennials might even spur the long-awaited rise in wages that puts them more on par with older generations.

Yet among many experts there’s a nagging unease that Americans might have to get used to slower long-term growth absent dramatic policy changes, another tech-induced boom, a population surge or some other unforeseen shift in how the country goes about its ways.

“Back in 2007, who would have thought we would be having a discussion about this right now,” said Christopher Probyn, chief economist of State Street Global Advisors.