EVEN those disturbed by rising income inequality accept the necessity of a system that rewards ability. And even the most die hard capitalist would not defend a system that apportions its rewards principally to the lucky. But what if talent and luck are increasingly hard to distinguish? Alan Krueger, the departing chairman of Barack Obama’s Council of Economic Advisers, hit upon just that possibility in a speech in June, when he observed, “The lucky and the talented – and it is often hard to tell the difference – have been doing better and better, while the vast majority has struggled to keep up.”

The focus of Mr Krueger’s speech was the spectacular, and growing returns to superstars, in particular in rock and roll. He attributes the first explanation for superstar economics to Alfred Marshall:

In the late 1800s, Marshall was trying to explain why some exceptional businessmen amassed great fortunes while the incomes of ordinary artisans and others fell. He concluded that changes in communications technology allowed “a man exceptionally favored by genius and good luck” to command “undertakings vaster, and extending over a wider area, than ever before. Ironically, his example of a profession where the best performers were unable to achieve such super star status was music. Marshall wrote, “so long as the number of persons who can be reached by a human voice is strictly limited, it is not very likely that any singer will make an advance on the £10,000 said to have been earned in a season by Mrs. Billington at the beginning of the last century, nearly as great [an increase] as that which the business leaders of the present generation have made on those of the last.” extending over a wider area, than ever before.”

The late Sherwin Rosen elaborated on this insight, developing the contribution of “imperfect substitution” and scale to superstar economics. Imperfect substitution means that you "would rather listen to one song by your favorite singer than a song and a half by someone else. Or, in another context, it means that if you need to have heart surgery, you would rather have the best surgeon in Cleveland perform it rather than the second and third best together.” In the corporate context, a board of directors would sooner pay $100 million for the best possible CEO than $10 million for the second best, on the presumption that the additional $90 million is a pittance relative to the additional value the best CEO can create for a multinational company.

But how does one get to be the best CEO, rather than second best? Here’s where luck comes in. Of course, luck is pervasive in life and it’s hard to beat the importance of being lucky enough to have the right parents. Nonetheless, there are features of the modern market that accentuate the role of luck through the self-fulfilling nature of popularity, the subject of this week’s Free Exchange column:

Abhijit Banerjee, now at the Massachusetts Institute of Technology, devised a model of “rational herding” in which market participants base their decision on a combination of their own information and the actions of others. Over successive rounds of transactions, participants responded less to their own information and more to the herd.

Herding is usually applied to asset markets. The buyer of a stock or a house seldom has the framework, means or inclination to calculate an asset’s intrinsic value. Instead, relative value prevails: is it worth more than it was yesterday? Is it worth more than others just like it? For this reason, buyers and sellers are always looking over their shoulder at what others in the same market are doing. A small rise in a stock leads to another, and so on; a small decline has the opposite effect.

Yet herd behavior also prevails in the markets for goods and services, too. Mr Krueger and our column cited an experiment in 2008 that doctored the rankings of pop songs offered for download. When the rankings were manipulated to make the least popular appear as the most popular, subsequent downloads shot up. Could the same behavior influence labour market outcomes? That’s what Kory Kroft of the University of Toronto, Fabian Lange of McGill University and Matthew Notowidigdo of the University of Chicago set out to determine with an experiment

in which they applied for 3,000 clerical, administrative, sales and customer-service jobs advertised online by submitting 12,000 fictitious cvs. The submissions were designed so that applicants with similar backgrounds, education and experience went for the same job. The only difference was how long the applicant had been jobless, a period that ranged from no time at all to as much as 36 months…They found that the odds of an applicant being called back by an employer declined steadily as the duration of unemployment rose, from 7.4% after one month without work down to 4-5% at the eight-month mark, where the call-back rate stabilised (see right-hand chart above).

The design and results of the experiment excluded three common explanations for why the long-term unemployed stay that way; that employers spot some qualitative flaw in the applicant’s resume; that the unemployed themselves search less energetically as unemployment lengthens; or that employers equate lengthening unemployment with atrophying skills:

These results strongly suggest that long-term unemployment is at least partly self-fulfilling. Like patrons who avoid restaurants purely because they are empty, employers were reluctant to hire someone other employers didn’t want.

If their inference is correct, the implication is troubling: someone who ends up unemployed through bad luck, and for some idiosyncratic reason doesn’t quickly land a job, finds his chances of reemployment diminish until he’s part of the long-term unemployed.

The opposite almost certainly holds true, as well. Someone lucky enough to graduate when the economy is booming and unemployment is low will spend relatively little time searching for his first job, creating a virtuous cycle that leads to more time employed at higher wages throughout his life. Or imagine someone who ten years ago could choose between working for Lehman Brothers or J.P. Morgan; it would have been impossible to know that, five years later, one would be bankrupt and the other not, but that decision would have profound consequences for his lifetime earnings.

A job applicant can explain long spells of unemployment in a job interview with a careful and convincing story about training, family obligations, or just plain bad luck, hoping that his other virtues would more than compensate. But the spread of employment screening software makes it hard even get to the interview, and lying is risky since online background checks have elevated the odds and consequences of getting caught.

Similarly, as social media and online market places proliferate, artists and entrepreneurs must worry that no matter how good their product is, it won’t matter if they can’t crack the most popular list. Separate studies have found evidence of herding on Prosper.com, a microloan site, and Kickstarter, which connects businesses to equity investors. On both, well-funded businesses are more successful at subsequent fund-raising.

Taken far enough, one could gloomily conclude that luck is the dominant determinant of labour market outcomes. This is surely an exaggeration. The rock song experiment found that the most popular song, even when falsely labeled least popular, clambered its way back up the popularity rankings once enough people had listened to it. And as has been well established, there is strong and growing correlation between education or skills on one hand and wages on the other. (Of course, luck and skill are not mutually exclusive). Moreover, the age old observation that successful people make their own luck applies; as “most popular” lists proliferate, so do efforts to game them, and counter-efforts to game the gamers, as Derek Thompson at the Atlantic describes here in the case of Yelp.

Nonetheless, this research opens a new and troubling dimension on inequality. Unlike deficiencies of skill, it’s hard to tell society's losers they should go back to school to become luckier.