IN HIS PROPOSAL for reforming the curriculum at William and Mary College in Virginia, Thomas Jefferson wrote that it should nurture “those talents which nature has sown as liberally among the poor as the rich, but which perish without use, if not sought for and cultivated”. Inspired by Jefferson, Americans expect higher education to boost the chances of disadvantaged people, but it seems to be failing in that task—and in some of the other jobs its customers want it to do.

Higher education has two sets of customers: students and the government. Students want all sorts of things from it—to make friends, sharpen their minds and get away from home. But most of all they want it to improve their economic prospects.

Despite rising costs, college still does that. An investment in a four-year degree offers a return of around 15% a year for somebody working until the age of 65, a figure that has been steady since 2000. But the returns have held up not because graduates have done so well but because those with only high-school degrees have done so badly (see chart 5). And although average returns remain decent, the range is vast. According to Payscale, a pay consultancy, it varies from +22% to -21%. Rising inequality increases the range of possible outcomes, and hence the risk of taking on student debt.

Governments want three things from higher education: research, human capital and equity. On the research side, America’s government has little to complain of. Although several European countries have more Shanghai top 100 universities in relation to their population than the United States does, America still dominates the summit of research: 19 of the world’s top 20 universities in Leiden University’s ranking of most-cited scientific papers in 2014 were American.

On the human-capital side, things look less good. In 1995 America had the highest graduation rate in the OECD. Now it lags behind seven other countries. President Barack Obama has set a target for his country to return to the top of the graduation league by 2020, but it is unlikely to be met. Young American graduates are below the OECD average in numeracy (see chart 5) and literacy, and are doing relatively worse than older ones. Some of the explanation lies with the poor performance of America’s schools, but the most expensive tertiary-education system in the OECD might be expected to help students catch up.

Recent work by American academics suggests that it does not. Richard Arum of New York University and Josipa Roksa of the University of Virginia, authors of “Academically Adrift”, looked at the results of 2,300 students who took the Collegiate Learning Assessment (CLA), a test of critical thinking, complex reasoning and writing, and found that 45% of the sample showed no significant gains between their first and third years.

On equity, the results also look bleak. Graduation rates between rich and poor are diverging (see chart 5). Given the difference in spending on those at the top and at the bottom, that is perhaps not surprising. “Community colleges”, says Derek Bok, a former president of Harvard, “spend roughly $10,000 per student. Harvard probably spends over $100,000. And our students are much easier to teach.” The combination of state spending cuts, which have led some community colleges to restrict entry, and endowments lifted by booming stockmarkets is increasing the gap further.

In real terms, tuition fees have nearly doubled over 20 years. Big bills mean big debts (see chart 5). Nearly a third of students are in default, and the rate is rising. Student loans can rarely be discharged, even by bankruptcy, so default damages people’s credit history, makes it hard to get mortgages and thus both harms people’s welfare and acts as a drag on the economy. Given unprecedented default rates, there are worries that the federal government will be stuck with a lot of the debt.

Not what it seems

In most markets, the combination of technological progress and competition pushes price down and quality up. But the technological revolution that has upended other parts of the information industry (see article) has left most of the higher-education business unmoved. Why?

For one thing, while research impact is easy to gauge, educational impact is not. There are no reliable national measures of what different universities’ graduates have learned, nor data on what they earn, so there is no way of assessing which universities are doing the educational side of their job well. Universities are paid on the basis of research, not educational, output.

Students, meanwhile, are not buying education any more than the government is. They are buying degrees, whose main purpose is to signal to employers that an individual went to a—preferably highly selective—university. Harvard degrees are valuable because there are so few of them. Harvard therefore has no incentive to make them cheaper, nor to produce more of them: that would make them less precious.

This helps explain why America’s universities are failing to deliver equity. People are prepared to pay through the nose to buy advantage for their children, so top institutions charge ever higher prices and acquire ever more resources, while those at the bottom get less. That does not serve the Jeffersonian ideal of nurturing the talents of the poor as well as the rich for the greater good of society. So higher education has a divided soul: it is both a great collective enterprise to increase the nation’s welfare and a fight to the death between status-hungry parents.

Employers are not much interested in the education universities provide either. Lauren Rivera of Northwestern University’s Kellogg School of Management interviewed 120 recruiters from American law firms, management consultancies and investment banks. Their principal filter was the applicant’s university. Unless he had attended one of the top institutions, he was not even considered. “Evaluators relied so intensely on ‘school’ as a criterion of evaluation not because they believed that the content of elite curricula better prepared students for life in their firms…but because of the perceived rigour of the admissions process,” Ms Rivera wrote. After the status of the institution, recruiters looked not at students’ grades but at their extracurricular activities, preferring the team sports—lacrosse, field-hockey and rowing—favoured by well-off white men.

If employers are not interested in grades, students might as well take it easy. That is, indeed, what they seem to be doing. Time-use studies show that the time students spend in class or studying has dropped from 40 hours a week in the 1920s to the 1960s to 27 hours a week now. And since academics are promoted largely on the basis of their research, they might as well give up teaching. That is, indeed, what they seem to be doing. Tenured faculty—the ones with the well-paid, secure jobs—spend less and less time with undergraduates. Increasingly, teaching is done by “non-tenure-track” faculty on short contracts. Mr Arum and Ms Roksa conclude that “no actors in the system are primarily interested in undergraduate student academic growth.”

The peculiar way in which universities are managed contributes to their failure to respond to market pressures. “Shared governance”, which gives power to faculty, limits managers’ ability to manage. “It was thought an affront to academic freedom when I suggested all departments should have the same computer vendor,” says Larry Summers, a former Harvard president. Universities “have the characteristics of a workers’ co-op. They expand slowly, they are not especially focused on those they serve, and they are run for the comfort of the faculty.”

Cost control is especially hard. As Clark Kerr, who designed the Californian higher-education system in the 1960s, wrote: “The call for effectiveness in the use of resources will be perceived by many inside the university world as the best current definition of evil.” Bringing about change is also tough. Change is rarely welcome, but in most organisations competition makes it inevitable. Mr Kerr doubted that university faculty “can agree on more than the preservation of the status quo”. Academics’ resistance to change gains added strength from their belief that education is not an occupation but a calling; and that to defend it against barbarians is not self-interest but moral duty.

But the pressure for change is growing. Some of it comes from the federal government, which is trying to make higher education more equitable and to get more value for money. On the equity side, Mr Obama announced in his state-of-the-union address in January that attending community college would be free for most people. But since the least well-off already get grants to cover their living expenses as well as tuition costs, it is not clear how much difference that will make.

On value for money, the government has launched an attack on for-profit colleges. A report by a congressional committee published in 2012 found that for-profits had a 64% drop-out rate and spent 22% of revenues on marketing, advertising, recruiting and admissions, against 18% on teaching. The government is asking colleges to ensure that average debt repayment of graduates on their programmes is below a set percentage of graduates’ incomes. For-profits point out that they don’t control students’ borrowing, nor can they control incomes, which depend on the economic cycle. They maintain that the measure—currently stuck in the courts—would damage equity: since poorer students are more likely to get into financial trouble, “the powerful incentive”, says Andrew Rosen, chairman of Kaplan, “is to jettison the least-prepared students.”

Better information about the returns to education would make heavy-handed regulation unnecessary. There is a bit more around, these days, but it is patchy. The CLA has been used by around 700 colleges to test what students have learned; some institutions are taking it up because, at a time of grade inflation, it offers employers an externally verified assessment of students’ brainpower. Payscale publishes data on graduates’ average income levels, but they are based on self-reporting and limited samples. Several states have applied to the IRS to get data on earnings, but have been turned down. The government is developing a “scorecard” of universities, but it seems unlikely to include earnings data. “A combined effort by the White House, the Council of Economic Advisers and the Office of Management and Budget is needed,” says Mark Schneider, a former commissioner of the National Centre for Education Statistics. It is unlikely to be forthcoming. Republicans object on privacy grounds (even though no personal information would be published); Democrats, who rely on the educational establishment for support, resist publication of the data because the universities do.

There is pressure on the sector from the market as well as from the government. After years of big increases in tuition fees, universities are facing resistance from the customers, and financial prospects for the sector are looking gloomy. Moody’s has a negative outlook: universities are “expecting the weakest net tuition revenue in a decade in fiscal year 2015”. It expects tuition fees at public universities to rise by an average of only 1.9%, though at private universities the increase is likely to be a more comfortable 2.7%. In the past five years college enrolment among those finishing high school has fallen, as cash-strapped community colleges turn applicants away and for-profits restrict recruitment of marginal students.

“America seems to have hit a wall,” says Simon Marginson. The country that has given the world so many ideas about how to run higher education could do with some new ones itself.