We could start with birds, or we could start with Greeks. Each option has advantages.

Let’s flip a coin. Heads and it’s the Greeks, tails and it’s the birds.

Tails.

In the 1970s, a young American anthropologist named Michael Dove set out for Indonesia, intending to solve an ethnographic mystery. Then a graduate student at Stanford, Dove had been reading about the Kantu’, a group of subsistence farmers who live in the tropical forests of Borneo. The Kantu’ practise the kind of shifting agriculture known to anthropologists as swidden farming, and to everyone else as slash-and-burn. Swidden farmers usually grow crops in nutrient-poor soil. They use fire to clear their fields, which they abandon at the end of each growing season.

Like other swidden farmers, the Kantu’ would establish new farming sites ever year in which to grow rice and other crops. Unlike most other swidden farmers, the Kantu’ choose where to place these fields through a ritualised form of birdwatching. They believe that certain species of bird – the Scarlet-rumped Trogon, the Rufous Piculet, and five others – are the sons-in-law of God. The appearances of these birds guide the affairs of human beings. So, in order to select a site for cultivation, a Kantu’ farmer would walk through the forest until he spotted the right combination of omen birds. And there he would clear a field and plant his crops.

Dove figured that the birds must be serving as some kind of ecological indicator. Perhaps they gravitated toward good soil, or smaller trees, or some other useful characteristic of a swidden site. After all, the Kantu’ had been using bird augury for generations, and they hadn’t starved yet. The birds, Dove assumed, had to be telling the Kantu’ something about the land. But neither he, nor any other anthropologist, had any notion of what that something was.

He followed Kantu’ augurers. He watched omen birds. He measured the size of each household’s harvest. And he became more and more confused. Kantu’ augury is so intricate, so dependent on slight alterations and is-the-bird-to-my-left-or-my-right contingencies that Dove soon found there was no discernible correlation at all between Piculets and Trogons and the success of a Kantu’ crop. The augurers he was shadowing, Dove told me, ‘looked more and more like people who were rolling dice’.

Stumped, he switched dissertation topics. But the augury nagged him. He kept thinking about it for ‘a decade or two’. And then one day he realised that he had been looking at the question the wrong way all the time. Dove had been asking whether Kantu’ augury imparted useful ecological information, as opposed to being random. But what if augury was useful precisely because it was random?

For the Kantu’, the best option was one familiar to any investor when faced with an unpredictable market: they needed to diversify

Tropical swidden agriculture is a fundamentally unpredictable enterprise. The success of a Kantu’ swidden depends on rainfall, pest outbreaks and river levels, among other factors. A patch of forest that might yield a good harvest in a rainy year could be unproductive in a drier year, or in a year when a certain pest spreads. And things such as pest outbreaks or the weather are pretty much impossible to predict weeks or months in the future, both for humans and for birds.

In the face of such uncertainty, though, the human tendency is to seek some kind of order – to come up with a systematic method for choosing a field site, and, in particular, to make decisions based on the conditions of the previous year.

Neither option is useful. Last year’s conditions have pretty much no bearing on events in the years ahead (a rainy July 2013 does not have any bearing on the wetness of July 2014). And systematic methods can be prey to all sorts of biases. If, for example, a Kantu’ farmer predicted that the water levels would be favourable one year, and so put all his fields next to the river, a single flood could wipe out his entire crop. For the Kantu’, the best option was one familiar to any investor when faced with an unpredictable market: they needed to diversify. And bird augury was an especially effective way to bring about that kind of diversification.

It makes sense that it should have taken Dove some 15 years to realise that randomness could be an asset. As moderns, we take it for granted that the best decisions stem from a process of empirical analysis and informed choice, with a clear goal in mind. That kind of decision-making, at least in theory, undergirds the ways that we choose political leaders, play the stock market, and select candidates for schools and jobs. It also shapes the way in which we critique the rituals and superstitions of others. But, as the Kantu’ illustrate, there are plenty of situations when random chance really is your best option. And those situations might be far more prevalent in our modern lives than we generally admit.

Over the millennia, cultures have expended a great deal of time, energy and ingenuity in order to introduce some element of chance into decision-making. Naskapi hunters in the Canadian province of Labrador would roast the scapula of a caribou in order to determine the direction of their next hunt, reading the cracks that formed on the surface of the bone like a map. In China, people have long sought guidance in the passages of the I Ching, using the intricate manipulation of 49 yarrow stalks to determine which section of the book they ought to consult. The Azande of central Africa, when faced with a difficult choice, would force a powdery poison down a chicken’s throat, finding the answer to their question in whether or not the chicken survived – a hard-to-predict, if not quite random, outcome. (‘I found this as satisfactory a way of running my home and affairs as any other I know of,’ wrote the British anthropologist E E Evans-Pritchard, who adopted some local customs during his time with the Azande in the 1920s).

The list goes on. It could – it does – fill books. As any blackjack dealer or tarot reader might tell you, we have a love for the flip of the card. Why shouldn’t we? Chance has some special properties. It is a swift, consistent, and (unless your chickens all die) relatively cheap decider. Devoid of any guiding mind, it is subject to neither blame nor regret. Inhuman, it can act as a blank surface on which to descry the churning of fate or the work of divine hands. Chance distributes resources and judges disputes with perfect equanimity.

The sanitising effect of augury cleans out any bad reasons

Above all, chance makes its selection without any recourse to reasons. This quality is perhaps its greatest advantage, though of course it comes at a price. Peter Stone, a political theorist at Trinity College, Dublin, and the author of The Luck of the Draw: The Role of Lotteries in Decision Making (2011), has made a career of studying the conditions under which such reasonless-ness can be, well, reasonable.

‘What lotteries are very good for is for keeping bad reasons out of decisions,’ Stone told me. ‘Lotteries guarantee that when you are choosing at random, there will be no reasons at all for one option rather than another being selected.’ He calls this the sanitising effect of lotteries – they eliminate all reasons from a decision, scrubbing away any kind of unwanted influence. As Stone acknowledges, randomness eliminates good reasons from the running as well as bad ones. He doesn’t advocate using chance indiscriminately. ‘But, sometimes,’ he argues, ‘the danger of bad reasons is bigger than the loss of the possibility of good reasons.’

For an example, let’s return to the Kantu’. Besides certain basic characteristics, when it comes to selecting a swidden site in the forest, there are no good reasons by which to choose a site. You just don’t know what the weather and pests will look like. As a result, any reasons that a Kantu’ farmer uses will either be neutral, or actively harmful. The sanitising effect of augury cleans out those bad reasons. The Kantu’ also establish fields in swampland, where the characteristics of a good site are much more predictable – where, in other words, good reasons are abundant. In the swamps, as it happens, the Kantu’ don’t use augury to make their pick.

Thinking about choice and chance in this way has applications outside rural Borneo, too. In particular, it can call into question some of the basic mechanisms of our rationalist-meritocratic-democratic system – which is why, as you might imagine, a political theorist such as Stone is so interested in randomness in the first place.

Around the same time that Michael Dove was pondering his riddle in a Kantu’ longhouse, activists and political scientists were beginning to revive the idea of filling certain political positions by lottery, a process known as sortition.

The practice has a long history. Most public officials in democratic Athens were chosen by lottery, including the nine archons who were chosen by sortition from a significant segment of the population. The nobles of Renaissance Venice used to select their head of state, the doge, through a complicated, partially randomised process. Jean-Jacques Rousseau, in The Social Contract (1762), argued that lotteries would be the norm in an ideal democracy, giving every citizen an equal chance of participating in every part of the government (Rousseau added that such ideal democracies did not exist). Sortition survives today in the process of jury selection, and it crops up from time to time in unexpected places. Ontario and British Columbia, for example, have used randomly selected panels of Canadian citizens to propose election regulations.

Advocates of sortition suggest applying that principle more broadly, to congresses and parliaments, in order to create a legislature that closely reflects the actual composition of a state’s citizenship. They are not (just to be clear) advocating that legislators randomly choose policies. Few, moreover, would suggest that non-representative positions such as the US presidency be appointed by a lottery of all citizens. The idea is not to banish reason from politics altogether. But plenty of bad reasons can influence the election process – through bribery, intimidation, and fraud; through vote-purchasing; through discrimination and prejudices of all kinds. The question is whether these bad reasons outweigh the benefits of a system in which voters pick their favourite candidates.

By way of illustration: a handful of powerful families and influential cliques dominated Renaissance Venice. The use of sortition in selection of the doge, writes the historian Robert Finlay in Politics in Renaissance Venice (1980), was a means of ‘limiting the ability of any group to impose its will without an overwhelming majority or substantial good luck’. Americans who worry about unbridled campaign-spending by a wealthy few might relate to this idea.

Or consider this. In theory, liberal democracies want legislatures that accurately reflect their citizenship. And, presumably, the qualities of a good legislator (intelligence, integrity, experience) aren’t limited to wealthy, straight, white men. The relatively homogeneous composition of our legislatures suggests that less-than-ideal reasons are playing a substantial role in the electoral process. Typically, we just look at this process and wonder how to eliminate that bias. Advocates of sortition see conditions ripe for randomness.

once all good reasons are eliminated, the most efficient, most fair and most honest option might be chance

It’s not only politics where the threat of bad reasons, or a lack of any good reasons, makes the luck of the draw seem attractive. Take college admissions. When Columbia University accepts just 2,291 of its roughly 33,000 applicants, as it did this year, it’s hard to imagine that the process was based strictly on good reasons. ‘College admissions are already random; let’s just admit it and begin developing a more effective system,’ wrote the education policy analyst Chad Aldeman on the US daily news site Inside Higher Ed back in 2009. He went on to describe the notion of collegiate meritocracy as ‘a pretension’ and remarked: ‘A lottery might be the answer.’

The Swarthmore College professor Barry Schwartz, writing in The Atlantic in 2012, came to a similar conclusion. He proposed that, once schools have narrowed down their applicant pools to a well-qualified subset, they could just draw names. Some schools in the Netherlands already use a similar system. ‘A lottery like this won’t correct the injustice that is inherent in a pyramidal system in which not everyone can rise to the top,’ wrote Schwartz. ‘But it will reveal the injustice by highlighting the role of contingency and luck.’ Once certain standards are met, no really good reasons remain to discriminate between applicant No 2,291 (who gets into Columbia) and applicant No 2,292 (who does not). And once all good reasons are eliminated, the most efficient, most fair and most honest option might be chance.

But perhaps not the most popular one. When randomness is added to a supposedly meritocratic system, it can inspire quite a backlash. In 2004, the International Skating Union (ISU) introduced a new judging system for figure-skating competitions. Under this system – which has since been tweaked – 12 judges evaluated each skater, but only nine of those votes, selected at random, actually counted towards the final tally (the ancient Athenians judged drama competitions in a similar way). Figure skating is a notoriously corrupt sport, with judges sometimes forming blocs that support each other’s favoured skaters. In theory, a randomised process makes it harder to form such alliances. A tit-for-tat arrangement, after all, doesn’t work as well if it’s unclear whether your partners will be able to reciprocate.

But the new ISU rules did more than simply remove a temptation to collude. As statisticians pointed out, random selection will change the outcome of some events. Backing their claims with competition data, they showed how other sets of randomly selected votes would have yielded different results, actually changing the line-up of the medal podium in at least one major competition. Even once all the skaters had performed, ultimate victory depended on the luck of the draw.

There are two ways to look at this kind of situation. The first way – the path of outrage – condemns a system that seems fundamentally unfair. A second approach would be to recognise that the judging process is already subjective and always will be. Had a different panel of 12 judges been chosen for the competition, the result would have varied, too. The ISU system simply makes that subjectivity more apparent, even as it reduces the likelihood that certain obviously bad influences, such as corruption, will affect the final result.

Still, most commentators opted for righteous outrage. That isn’t surprising. The ISU system conflicts with two common modern assumptions: that it is always desirable (and usually possible) to eliminate uncertainty and chance from a situation; and that achievement is perfectly reflective of effort and talent. Sortition, college admission lotteries, and randomised judging run against the grain of both of these premises. They embrace uncertainty as a useful part of their processes, and they fail to guarantee that the better citizen or student or skater, no matter how much she drives herself to success, will be declared the winner.

Let me suggest that, in the fraught and unpredictable world in which we live, both of those ideals – total certainty and perfect reward – are delusional. That’s not to say that we shouldn’t try to increase knowledge and reward success. It’s just that, until we reach that utopia, we might want to come to terms with the reality of our situation, which is that our lives are dominated by uncertainty, biases, subjective judgments and the vagaries of chance.

In the novel The Man in the High Castle (1962), the American sci-fi maestro Philip K Dick imagines an alternative history in which Germany and Japan win the Second World War. Most of the novel’s action takes place in Japanese-occupied San Francisco, where characters, both Japanese and American, regularly use the I Ching to guide difficult decisions in their business lives and personal affairs.

Something, somewhere, is always playing dice

As an American with no family history of divination, I’ll admit to being enchanted by Dick’s vision of a sci-fi world where people yield some of their decision-making power to the movements of dried yarrow stems. There’s something liberating, maybe, in being able to acknowledge that the reasons we have are often inadequate, or downright poor. Without needing to impose any supernatural system, it’s not hard to picture a society in which chance plays a more explicit, more accepted role in the ways in which we distribute goods, determine admissions to colleges, give out jobs to equally matched applicants, pick our elected leaders, and make personal decisions in our own lives.

Such a society is not a rationalist’s nightmare. Instead, in an uncertain world where bad reasons do determine so much of what we decide, it’s a way to become more aware of what factors shape the choices we make. As Peter Stone told me, paraphrasing Immanuel Kant, ‘the first task of reason is to recognise its own limitations’. Nor is such a society more riddled with chanciness than our own. Something, somewhere, is always playing dice. The roles of coloniser and colonised, wealthy and poor, powerful and weak, victor and vanquished, are rarely as predestined as we imagine them to be.

Dick seems to have understood this. Certainly, he embraced chance in a way that few other novelists ever have. Years after he wrote The Man in the High Castle, Dick explained to an interviewer that, setting aside from planning and the novelist’s foresight, he had settled key details of the book’s plot by flipping coins and consulting the I Ching.