The job market is improving, but don't get your hopes up that the labor force participation will rise significantly anytime soon.

As of June, only 63% of Americans over age 16 participate in the labor force, meaning they either have a job or are actively looking for one. That number is around its weakest level since the early 1980s.

Unless there's a major change in policy, the White House expects the labor force participation rate will flat-line and then continuing falling, even as the economy returns to full strength.

In a report issued Thursday, the president's Council of Economic Advisors laid out several hypothetical scenarios for how the labor force participation rate could evolve over the next decade. They're not technical forecasts, but they all have one thing in common: They're all declining lines.

Related: Great jobs report: Strong hiring, unemployment down

The U.S. population is aging: The finding isn't surprising. Independent economists generally agree that the labor force participation rate will continue falling in the future as the U.S. population ages. As baby boomers retire and live longer than prior generations, it's only natural that a greater portion of the U.S. population won't be active in the job market.

The big question, however, is: Just how much of the labor force's ongoing declines are due to baby boomers?

The CEA economists have their own estimates. The labor force participation rate declined from around 66% at the end of 2007, to 63% in mid 2014. They calculate that half of that decline is due to the aging population.

Another sixth is due to a common phenomenon that occurs in recessions: Some workers decide to hold off on looking for jobs until the economy improves. Perhaps they go back to school to beef up their résumé, or they may choose to take care of children or elderly parents, while waiting out the weak job market.

Finally, CEA estimates the other third of the decline was due to other factors, including ongoing historical trends.

Related: See who is getting hired these days...

Labor force participation for "prime-age" men (ages 25 to 54) for example, has been declining since the 1950s, and for prime-age women, it's been declining since the late 1990s.

Growing concern: Regardless of the reasons, declining labor force participation is a worrisome sign because it may mean the U.S. economy has less potential to grow going forward. The implications are far-reaching, when a smaller part of the population is producing goods and services, paying income taxes, and basically supporting a larger non-working population.

CEA economists note that they believe immigration reform is the single most powerful policy that could counteract a declining labor force.

Immigrants tend to be younger and participate more in the job market, and a report from the non-partisan Congressional Budget Office last year estimates that the Senate's immigration reform bill could increase the labor force by 6 million people by 2023.

Read the full CEA report on labor force participation here.