Should only rich kids go to college? It seems like an absurd question. Yet evidence keeps mounting that, financially speaking, if you must borrow to pay for college you might be just as well off skipping higher education and going straight to work.

Some 37% of American households headed by an adult under the age of 40 have student loans outstanding—the highest share ever, according to a new report from the Pew Research Center. Their median student debt: $13,000. College loans in total run about $1 trillion and the Pew findings show that this burden weighs heavily on the finances of young Americans.

Households headed by a young college graduate with student loans outstanding have a typical net worth of just $8,700—a pittance compared to the typical $64,700 net worth of similar households only with no student loans outstanding. But that’s just the start. Those with student loans have total debts of $137,010—nearly double the typical $73,250 indebtedness of those without student debt outstanding.

These differences are far greater than the value of the student loans outstanding and speak to the snowball effect that debt has on many households. The findings suggest that, for many, their debt spiral can be traced to their first student loan. “It may be the case that the burden of student debt makes it more difficult for young adults to gain financial traction in other areas of their lives,” the researchers noted.

Perhaps most interesting, Pew found also that the typical net worth of households headed by a young adult without college debt and without a degree was $10,900—greater than the net worth of college grads that had taken on debt to get through school. Going to work sooner and avoiding the expense of college provided immediate payback. The clear loser in the analysis was the young head of household that took on student debt but never earned a degree. That person’s median net worth was just $1,200.

Richard Fry, a Pew researcher, cautioned not to read too much into the greater net worth of the debt-free non-graduate. “College-educated student debtors tend to have much higher household incomes than those who did not complete college (nearly $60,000 versus low $30,000s),” he said in an email. “The typical benefit from completing college is immediate in terms of household income. But if a young adult has to borrow they will be behind in building their nest egg.”

At some point the wealth gap closes and the larger income of the indebted college graduate offsets the early debt-free start of those who didn’t go to college. But it’s not clear how long that takes and it isn’t the case for everyone, especially in an economic period (like now) marred by young adult underemployment.

Findings like these have stirred a great debate in recent years: Is college worth it? Those who say a degree is not worth borrowing for have found a strong voice in PayPal founder Peter Thiel who established a “20 under 20” fellowship staking promising high school students to a $100,000 grant if they’d skip college and start a company. The Thiel Fellowship is entering its fourth year.

Suggesting that only the rich (or those who get full-ride scholarships and grants) go to college is about as politically incorrect as you can get. I would never take that position. It smacks of elitism and runs counter to the income inequality concerns that have made Tomas Piketty an overnight sensation.

Yet more than a third of young graduates themselves do not agree that their education has paid off, and evidence keeps mounting that student loans are the equivalent of wearing lead sneakers in an economic foot race. At the very least, anyone taking out these loans should understand the full nature of their costs.

Contact us at letters@time.com.