The higher education system is a unique type of organisation with its own way of motivating productivity in its scholarly workforce. It doesn’t need to compel professors to produce scholarship because they choose to do it on their own. This is in contrast to the standard structure for motivating employees in bureaucratic organisations, which relies on manipulating two incentives: fear and greed. Fear works by holding the threat of firing over the heads of workers in order to ensure that they stay in line: Do it my way or you’re out of here. Greed works by holding the prospect of pay increases and promotions in front of workers in order to encourage them to exhibit the work behaviours that will bring these rewards: Do it my way and you’ll get what’s yours.

Yes, in the United States contingent faculty can be fired at any time, and permanent faculty can be fired at the point of tenure. But, once tenured, there’s little other than criminal conduct or gross negligence that can threaten your job. And yes, most colleges do have merit pay systems that reward more productive faculty with higher salaries. But the differences are small – between the standard 3 per cent raise and a 4 per cent merit increase. Even though gaining consistent above-average raises can compound annually into substantial differences over time, the immediate rewards are pretty underwhelming. Not the kind of incentive that would motivate a major expenditure of effort in a given year – such as the kind that operates on Wall Street, where earning a million-dollar bonus is a real possibility. Academic administrators – chairs, deans, presidents – just don’t have this kind of power over faculty. It’s why we refer to academic leadership as an exercise in herding cats. Deans can ask you to do something, but they really can’t make you do it.

This situation is the norm for systems of higher education in most liberal democracies around the world. In more authoritarian settings, the incentives for faculty are skewed by particular political priorities, and in part for these reasons the institutions in those settings tend to be consigned to the lower tiers of international rankings. Scholarly autonomy is a defining characteristic of universities higher on the list.

If the usual extrinsic incentives of fear and greed don’t apply to academics, then what does motivate them to be productive scholars? One factor, of course, is that this population is highly self-selected. People don’t become professors in order to gain power and money. They enter the role primarily because of a deep passion for a particular field of study. They find that scholarship is a mode of work that is intrinsically satisfying. It’s more a vocation than a job. And these elements tend to be pervasive in most of the world’s universities.

But I want to focus on an additional powerful motivation that drives academics, one that we don’t talk about very much. Once launched into an academic career, faculty members find their scholarly efforts spurred on by more than a love of the work. We in academia are motivated by a lust for glory.

We want to be recognised for our academic accomplishments by earning our own little pieces of fame. So we work assiduously to accumulate a set of merit badges over the course of our careers, which we then proudly display on our CVs. This situation is particularly pervasive in the US system of higher education, which is organised more by the market than by the state. Market systems are especially prone to the accumulation of distinctions that define your position in the hierarchy. But European and other scholars are also engaged in a race to pick up honours and add lines to their CVs. It’s the universal obsession of the scholarly profession.

At the very pinnacle of the structure of merit badges is, of course, the Nobel Prize. A nice thought, but what are the odds? Fortunately, other academic honours are a lot more attainable. And attain them we do.

Take one prominent case in point: the endowed chair. A named professorship is a very big deal in the academic status order, a (relatively) scarce honour that supposedly demonstrates to peers that you’re a scholar of high accomplishment. It does involve money, but the chair-holder often sees little of it. A donor provides an endowment for the chair, which pays your salary and benefits, thus taking these expenses out of the operating budget – a big plus for the department, which saves a lot of money in the deal. And some chairs bring with them extra money that goes to the faculty member to pay for research expenses and travel.

But more often than not, the chair brings the occupant nothing at all but an honorific title, which you can add to your signature: the Joe Doakes Professor of Whatever. Once these chairs are in existence as permanent endowments, they never go away; instead they circulate among senior faculty. You hold the chair until you retire, and then it goes to someone else. In my own school, Stanford University, when the title passes to a new faculty member, that person receives an actual chair – one of those uncomfortable black wooden university armchairs bearing the school logo. On the back is a brass plaque announcing that ‘[Your Name] is the Joe Doakes Professor’. When you retire, they take away the title and leave you the physical chair. That’s it. It sounds like a joke – all you get to keep is this unusable piece of furniture – but it’s not. And faculty will kill to get this kind of honour.

This being the case, the academic profession requires a wide array of other forms of recognition that are more easily attainable and that you can accumulate the way you can collect Fabergé eggs. And they’re about as useful. Let us count the kinds of merit badges that are within the reach of faculty:

publication in high-impact journals and prestigious university presses;

named fellowships;

membership on review committees for awards and fellowships;

membership on editorial boards of journals;

journal editorships;

officers in professional organisations, which conveniently rotate on an annual basis and thus increase accessibility (in small societies, nearly everyone gets a chance to be president);

administrative positions in your home institution;

committee chairs;

a large number of awards of all kinds – for teaching, advising, public service, professional service, and so on: the possibilities are endless;

awards that particularly proliferate in the zone of scholarly accomplishment – best article/book of the year in a particular subfield by a senior/junior scholar; early career/lifetime-career achievement; and so on.

Each of these honours tells the academic world that you are the member of a purportedly exclusive club. At annual meetings of professional organisations, you can attach brightly coloured ribbons to your name tag that tell everyone you’re an officer or fellow of that organisation, like the badges that adorn military dress uniforms. As in the military, you can never accumulate too many of these academic honours. In fact, success breeds more success, as your past tokens of recognition demonstrate your fitness for future tokens of recognition.

I am eager to pursue tokens of merit, and desperate to avoid becoming the department’s walking dead

Academics are unlike the employees of most organisations in that they fight over symbolic rather than material objects of aspiration, but they are like other workers in that they too are motivated by fear and greed. Instead of competing over power and money, they compete over respect. So far I’ve been focusing on professors’ greedy pursuit of various kinds of honours. But, if anything, fear of dishonour is an even more powerful motive for professorial behaviour. I aspire to gain the esteem of my peers but I’m terrified of earning their scorn.

Lurking in the halls of every academic department are a few furtive figures of scholarly disrepute. They’re the professors who are no longer publishing in academic journals, who have stopped attending academic conferences, and who teach classes that draw on the literature of yesteryear. Colleagues quietly warn students to avoid these academic ghosts, and administrators try to assign them courses where they will do the least harm. As an academic, I might be eager to pursue tokens of merit, but I am also desperate to avoid being lumped together with the department’s walking dead. Better to be an academic mediocrity, publishing occasionally in second-rate journals, than to be your colleagues’ archetype of academic failure.

The result of all this pursuit of honour and retreat from dishonour is a self-generating machine for scholarly production. No administrator needs to tell us to do it, and no one needs to dangle incentives in front of our noses as motivation. The pressure to publish and demonstrate academic accomplishment comes from within. College faculties become self-sustaining engines of academic production, in which we drive ourselves to demonstrate scholarly achievement without the administration needing to lift a finger or spend a dollar. What could possibly go wrong with such a system?

One problem is that faculty research productivity varies significantly according to what tier of the highly stratified structure of higher education professors find themselves in. Compared with systems of higher education in other countries, the US system is organised into a hierarchy of institutions that are strikingly different from each other. The top tier is occupied by the 115 universities that the Carnegie Classification labels as having the highest research activity, which represents only 2.5 per cent of the 4,700 institutions that grant college degrees. The next tier is doctoral universities with less of a research orientation, which account for 4.7 per cent of institutions. The third is an array of master’s level institutions often referred to as comprehensive universities, which account for 16 per cent. The fourth is baccalaureate institutions (liberal arts colleges), which account for 21 per cent. The fifth is two-year colleges, which account for 24 per cent. (The remaining 32 per cent are small specialised institutions that enrol only 5 per cent of all students.)

The number of publications by faculty members declines sharply as you move down the tiers of the system. One study shows how this works for professors in economics. The total number of refereed journal articles published per faculty member over the course of a career was 18.4 at research universities; 8.1 at comprehensive universities; 4.9 at liberal arts colleges; and 3.1 at all others. The decline in productivity is also sharply defined within the category of research universities. Another study looked at the top 94 institutions ranked by per-capita publications per year between 1991 and 1993. At the number-one university, average production was 12.7 per person per year; at number 20, it dropped off sharply to 4.6; at number 60, it was 2.4; and at number 94, it was 0.5.

Only 20 per cent of faculty serve at the most research-intensive universities (the top tier) where scholarly productivity is the highest. As we can see, the lowest end of this top sliver of US universities has faculty who are publishing less than one article every five years. The other 80 per cent are presumably publishing even more rarely than this, if indeed they are publishing at all. As a result, it seems that the incentive system for spurring faculty research productivity operates primarily at the very top levels of the institutional hierarchy. So why am I making such a big deal about US professors as self-motivated scholars?

The most illuminating way to understand the faculty incentive to publish is to look at the system from the point of view of the newly graduating PhD who is seeking to find a faculty position. These prospective scholars face some daunting mathematics. As we have seen, the 115 high-research universities produce the majority of research doctorates, but 80 per cent of the jobs are at lower-level institutions. The most likely jobs are not at research universities but at comprehensive universities and four-year institutions. So most doctoral graduates entering the professoriate experience dramatic downward mobility.

The incentive to publish is baked in from the very beginning

It’s actually even worse than that. One study of sociology graduates shows that departments ranked in the top five select the majority of their faculty from top-five departments, but most top-five graduates ended up in institutions below the rank of 20. And a lot of prospective faculty never find a position at all. A 1999 study showed that, among recent grads who sought to become professors, only two-thirds had such a position after 10 years, and only half of these had earned tenure. And many of those who do find teaching positions are working part-time, a category that in 2005 accounted for 48 per cent of all college faculty.

The prospect of a dramatic drop in academic status and the possibility of failing to find any academic job do a lot to concentrate the mind of the recent doctoral graduate. Fear of falling compounded by fear of total failure works wonders in motivating novice scholars to become flywheels of productivity. From their experience in grad school, they know that life at the highest level of the system is very good for faculty, but the good times fade fast as you move to lower levels. At every step down the academic ladder, the pay is less, the teaching loads are higher, graduate students are fewer, research support is less, and student skills are lower.

In a faculty system where academic status matters more than material benefits, the strongest signal of the status you have as a professor is the institution where you work. Your academic identity is strongly tied to your letterhead. And in light of the kind of institution where most new professors find themselves, they start hearing a loud, clear voice saying: ‘I deserve better.’

So the mandate is clear. As a grad student, you need to write your way to an academic job. And when you get a job at an institution far down the hierarchy, you need to write your way to a better job. You experience a powerful incentive to claw your way back up the academic ladder to an institution as close as possible to the one that recently graduated you. The incentive to publish is baked in from the very beginning.

One result of this Darwinian struggle to regain one’s rightful place at the top of the hierarchy is that a large number of faculty fall by the wayside without attaining their goal. Dashed dreams are the norm for large numbers of actors. This can leave a lot of bitter people occupying the middle and lower tiers of the system, and it can saddle students with professors who would really rather be somewhere else. That’s a high cost for the process that supports the productivity of scholars at the system’s pinnacle.

Another potential problem with my argument about the self-generating incentive for professors to publish is that the work produced by scholars is often distinguished more by its quantity rather than its quality. Put another way, a lot of the work that appears in print doesn’t seem worth the effort required to read it, much less to produce it. Under these circumstances, the value of the incentive structure seems lacking.

Consider some of the ways in which contemporary academic production promotes quantity over quality. One familiar technique is known as ‘salami slicing’. The idea here is simple. Take one study and divide it up into pieces that can each be published separately, so it leads to multiple entries in your CV. The result is an accumulation of trivial bits of a study instead of a solid contribution to the literature.

Another approach is to inflate co-authorship. Multiple authors make sense in some ways. Large projects often involve a large number of scholars and, in the sciences in particular, a long list of authors is de rigueur. Fine, as long as everyone in the list made a significant contribution to research. But often co-authorship comes for reasons of power rather than scholarly contribution. It has become normal for anyone who compiled a dataset to demand co-authorship for any papers that draw on the data, even if the data-owner added nothing to the analysis in the paper. Likewise, the principal investigator of a project might insist on being included in the author list for any publications that come from this project. More lines on the CV.

2.5 million new science papers are published each year. How many make a substantive contribution?

Yet another way to increase the number of publications is to increase the number of journals. By one count, as of 2014 there were 28,100 scholarly peer-reviewed journals. Consider the mathematics. There are about 1 million faculty members at US colleges and universities at the BA level and higher, so that means there are about 36 prospective authors for each of these journals. A lot of these enterprises act as club journals. The members of a particular sub-area of a sub-field set up a journal where members of the club engage in a practice that political scientists call log-rolling. I review your paper and you review mine, so everyone gets published. Edited volumes work much the same way. I publish your paper in my book, and you publish mine in yours.

A lot of journal articles are also written in a highly formulaic fashion, which makes it easy to produce lots of papers without breaking an intellectual sweat. The standard model for this kind of writing is known as IMRaD. This mnemonic represents the four canonical sections for every paper: introduction (what’s it about and what’s the literature behind it?); methods (how did I do it?); research (what are my findings?); and discussion (what does it mean?). All you have to do as a writer is to write the same paper over and over, introducing bits of new content into the tried and true formula.

The result of all this is that the number of scholarly publications is enormous and growing daily. One estimate shows that, since the first science papers were published in the 1600s, the total number of papers in science alone passed the 50 million mark in 2009; 2.5 million new science papers are published each year. How many of them do you think are worth reading? How many make a substantive contribution to the field?

OK, so I agree. A lot of scholarly publications – maybe most such publications – are less than stellar. Does this matter? In one sense, yes. It’s sad to see academic scholarship fall into a state where the accumulation of lines on a CV matters more than producing quality work. And think of all the time wasted reviewing papers that should never have been written, and think of how this clutters and trivialises the literature with contributions that don’t contribute.

But – hesitantly – I suggest that the incentive system for faculty publication still provides net benefits for both academy and society. I base this hope on my own analysis of the nature of the US academic system itself. Keep in mind that US higher education is a system without a plan. No one designed it and no one oversees its operation. It’s an emergent structure that arose in the 19th century under unique conditions in the US – when the market was strong, the state was weak, and the church was divided.

Under these circumstances, colleges emerged as private not-for-profit enterprises that had a state charter but little or no state funding. And, for the most part, they arose for reasons that had less to do with higher learning than with the extrinsic benefits a college could bring. As a result, the system grew from the bottom up. By the time state governments started putting up their own institutions, and the federal government started funding land-grant colleges, this market-based system was already firmly in place. Colleges were relatively autonomous enterprises that had found a way to survive without steady support from either church or state. They had to attract and retain students in order to bring in tuition dollars, and they had to make themselves useful both to these students and to elites in the local community, both of whom would then make donations to continue the colleges in operation. This autonomy was an accident, not a plan, but by the 20th century it became a major source of strength. It promoted a system that was entrepreneurial and adaptive, able to take advantage of possibilities in the environment. More responsive to consumers and community than to the state, institutions managed to mitigate the kind of top-down governance that might have stifled the system’s creativity.

Maybe it’s worth tolerating a high level of dross in the effort to produce scholarly gold

The point is this: compared with planned organisational structures, emergent structures are inefficient at producing socially useful results. They’re messy by nature, and they pursue their own interests rather than following directions from above according to a plan. But as we have seen with market-based economies compared with state-planned economies, the messy approach can be quite beneficial. Entrepreneurs in the economy pursue their own profit rather than trying to serve the public good, but the side-effect of their activities is often to provide such benefits inadvertently, by increasing productivity and improving the general standard of living. A similar argument can be made about the market-based system of US higher education. Maybe it’s worth tolerating the gross inefficiency of a university system that is charging off in all directions, with each institution trying to advance itself in competition with the others. The result is a system that is the envy of the world, a world where higher education is normally framed as a pure state function under the direct control of the state education ministry.

This analysis applies as well to the professoriate. The incentive structure for US faculty encourages individual professors to be entrepreneurial in pursuing their academic careers. They need to publish in order to win honours for themselves and to avoid dishonour. As a result, they end up publishing a lot of work that is more useful to their own advancement (lines on a CV) than to the larger society. Also, following from the analysis of the first problem I introduced, an additional cost of this system is the large number of faculty who fall by the wayside in the effort to write their way into a better job. The success of the system of scholarly production at the top is based on the failed dreams of most of the participants.

But maybe it’s worth tolerating a high level of dross in the effort to produce scholarly gold – even if this is at the expense of many of the scholars themselves. Planned research production, operating according to mandates and incentives descending from above, is no more effective at producing the best scholarship than are five-year plans in producing the best economic results. At its best, the university is a place that gives maximum freedom for faculty to pursue their interests and passions in the justified hope that they will frequently come up with something interesting and possibly useful, even if this value is not immediately apparent. They’re institutions that provide answers to problems that haven’t yet developed, storing up both the dross and the gold until such time as we can determine which is which.