U.S. college enrollment is falling.

For years, America's college campuses swelled with more and more students. But enrollment peaked in 2010 at just over 21 million students. Attendance has dropped every year since.

By the fall of 2014 -- the most recent year government data is available -- there were 812,069 fewer students walking around college campuses.

Some say not to worry, the drop is happening because the economy is improving. More people are going back to work instead of signing up for additional degrees.

"Historically, as the economy improves and Americans get back to work, college enrollment declines," says U.S. Under Secretary of Education Ted Mitchell.

But here's what these young people are giving up: College graduates make almost double the salary of workers with only a high school diploma. If you care about the rise in inequality in America, declining college enrollment should alarm you.

"A college degree is the surest ticket to the middle class," President Obama said last year. Now there are 800,000 fewer Americans on that college path to the middle class.

Related: I'd be the first in my family to graduate college

Enrollment is falling fastest at these 2 schools

Dig a little deeper and the data gets even more troubling.

The two types of colleges with the biggest declines in enrollment are: community colleges and for-profit universities. Those schools draw heavily from low-income and minority households.

Community colleges are often lifelines for poor families. They are close to home, they don't require SAT scores, and they have cheap price tags.

President Obama proposed making community colleges free for two years because they are the gateway to higher education -- and better jobs -- for so many Americans. Since 2010, enrollment at community college has fallen by over 820,000 students.

"I happen to think it's a very white, middle class value that you're going to learn so much if you go away to college and live on campus," says Dr. Cecilia Cunningham, a long-time educator who now runs the Middle College National Consortium, a program in New York that encourages high school students to take community college classes.

The enrollment decline at for-profit institutions is different. Some for-profit schools have been criticized heavily for offering "worthless degrees" and leaving students with big debt.

Related: How much better off are college grads?

Finances are the top challenge

The overwhelming consensus is that America can -- and must -- do more to ensure that low-income and minority students can access college. The college enrollment drop since 2010 is the latest red flag that progress has stalled, if not retreated, on this front.

"Too many students and families feel that college is out of reach," says Mitchell, U.S. Under Secretary for Education. "Never in our history has the opportunity to complete college mattered so much to Americans' life outcomes."

High school senior Skylar Spencer is one of the students on the front lines of trying to figure out if college is in her future.

Spencer is class president at Morris Academy for Collaborative Studies, a public high school in one of the poorest neighborhoods of New York City. She's the type of leader who calls friends in the morning to ensure they get up for school, and she stays late to tutor others. Her personal motto is "The sky's the limit."

Related: N.Y. teen accepted by all 8 Ivy League schools

So easy for low-income students to give up

But it's almost the end of May and Spencer still has not finalized her college plans. By now, most high school seniors have already sent in their tuition payments.

But Spencer knows that paying for college is going to be a struggle. What's worse is that her mother recently lost her job. Now it's unclear if Spencer can even go in the fall.

"Everyone is telling me to relax, everything will work out. But I'm stressed like crazy," she says.

Spencer's dream is to attend the University of Nevada, Las Vegas, where she was accepted. But it could cost up to $20,000 a year, since she's from out of state. There's no way that can work. Spencer is waiting to hear back about more financial aid.

She was also accepted at one of New York's public universities: SUNY-Potsdam. She and her mother are figuring out if the finances will work to go there. In the meantime, Spencer is applying to more backup schools closer to home.

"I don't give up," Spencer told CNNMoney. Unlike thousands of poor students, Spencer is determined to be at college this fall.