If you are a person in America, then there are equations trying to learn more about you. Some of these equations work for private companies and some of them work for the government, but they all generate correlations based on your behavior. Google “Ways to keep New Years resolution” and buy a sweatband on Amazon, and your Facebook ads all turn to gym memberships. Search Wikihow for “Join al-Nusra Front” and buy a hunting knife at Target, alarms go off at the NSA. Interacting with the world now involves an implicit agreement to be watched, and not just by surveillance cameras, global positioning satellites, and browser cookies, but increasingly by algorithms designed to predict and manage our future conduct.

University of Maryland law professor Frank Pasquale’s new book The Black Box Society is a tour of how computational intelligence has come to dominate three important parts of American life: reputation, search, and finance. Pasquale is invoking a couple different concepts with the title. Like a black box on an airplane, these algorithms take information from the noise around them; like a black box in computer science, they are hidden systems, only observable from the outside in terms of their inputs and outputs. But more like black holes, the algorithms are visible in their effects on their surroundings. Our economy—and the many vital life processes it manages—twists and turns based on the say-so of inscrutable mathematical processes.

Reputation in this context means more than what people say about us when we’re not there. Our every watchable choice, from shopping to clicking, paints part of a portrait. The computers know us as the pupil at the center of a giant Venn diagram, the intersection of an uncountable mass of circles, each one labeled “Middle child” or “Honey Bunches of Oats eater” or “Bisexual,” etc. (Rob Horning calls this identity we leak as we move through the digitally connected world the “data self.”) Retailers, advertisers, and data brokers nip around the edges, identifying meaningful correlations where they can. Often this is relatively harmless, or even helpful—as when Netflix suggests a new movie or Pandora a new song. Other times it reveals the clumsiness of its own methodology, as when Twitter promotes an ad for a black professionals dating service into my timeline. The algorithms don’t need to get it right every time, just enough to make the correlation worthwhile in the aggregate.

But the digital reputation-system’s subterranean operations can be downright vicious. Civil rights protections in hiring, housing, and credit are no match for algorithms that can guess your race or propensity for future pregnancy. Pasquale writes that “a surprising proportion of digital marketing is about finding marks for dubious loans, pharmaceutical products, and fly-by-night for-profit educators.” Gambling scammers target lists of recovering addicts, and, of course, there are the penis enlargement ads. Whether or not it's in our interests, we are constantly telling computers the best ways to wring money and time (which is also money) out of us. We inform on ourselves, and we collectively provide the mass of data necessary for them to guess about people like us. It’s the “know your enemy” school of marketing, and we’re the enemy.





When it comes to search, Google isn’t a result, it’s who you ask in the first place. But the concept is much bigger than any would-be hegemonic firm. The Internet without a search engine is a whole different creation, like a telephone network without a phone book. Search enables us to connect to what we don’t already know, which is almost everything in the world. Pasquale likens Google’s search dominance to the spread of the English language: There’s nothing that makes it necessarily the best, but it was on top at an important moment and has been able to retain its spot. But Google’s owners (like English speakers) have interests are not identical to those of the general population; Pasquale cites a handful of situations where their central algorithm seemed to reflect the company’s financials more than an “objective” measure of the best results. Google has endeavored to appear to consumers like a public utility, a benign if somewhat paternal presence, but their books aren’t open to us. Ranking algorithms are as arbitrary and temperamental as the corporate interests that refine them.



Compared to reputation and search, finance is the area where Americans are most aware of the dangers of algorithms run amok. It was algorithms that made it possible for banks to combine sub-prime mortgages into respectable looking investments, and another set of rating algorithms that projected they would be safe places to put money. At its root, finance is supposed to be the part of the economy devoted to the efficient allocation of resources. The opportunity for profit keeps capital flowing from investors who have it to people who can make good use of it. But as the finance sector has developed, we see how these incentives alone don’t ensure good outcomes.