By some esti­mates, human­i­ty now pro­duces 2.5 quin­til­lion bytes of data every day — more than a hun­dred times the amount of data in the entire Library of Con­gress. This data ranges from Face­book posts to mil­i­tary-grade satel­lite pho­tos. Increas­ing­ly, this data is ana­lyzed by com­plex math­e­mat­i­cal mod­els that deter­mine more and more aspects of our lives, from the adver­tise­ments we see to whether we have access to pri­vate insur­ance. Yet despite their grow­ing impor­tance, these mod­els often remain hidden.

O’Neil makes a convincing case that many mathematical models today are engineered to benefit the powerful at the expense of the powerless.

Advo­cates of such math­e­mat­i­cal mod­el­ing, in both the pub­lic and pri­vate sec­tors, por­tray it as a neu­tral and effi­cient alter­na­tive to fal­li­ble and biased human deci­sion-mak­ing. Math­e­mati­cian, data sci­en­tist and pop­u­lar blog­ger Cathy O’Neil, author of Weapons of Math Destruc­tion: How Big Data Increas­es Inequal­i­ty and Threat­ens Democ­ra­cy, doesn’t agree. She argues that many math­e­mat­i­cal mod­els are ide­o­log­i­cal tools that exac­er­bate oppres­sion and inequal­i­ty. Her exam­ples range from the crime mod­els used by police depart­ments to deter­mine which neigh­bor­hoods to patrol, to the recidi­vism mod­els used by judges to hand out prison sentences.

O’Neil is pas­sion­ate about expos­ing the harm­ful effects of Big Data – dri­ven math­e­mat­i­cal mod­els (what she calls WMDs), and she’s unique­ly qual­i­fied for the task. She earned a Ph.D. in math from Har­vard and land­ed a tenure-track at Barnard. But she became bored with the pace and insu­lar­i­ty of acad­e­mia, and left to work as a quan­ti­ta­tive ana­lyst at the hedge fund D.E. Shaw. There, she had a front-row seat for the 2008 finan­cial crisis.

This expe­ri­ence fun­da­men­tal­ly changed O’Neil’s rela­tion­ship to math­e­mat­ics. She real­ized that far from being a neu­tral object of study, math­e­mat­ics was not only ​“deeply entan­gled in the world’s prob­lems but also fuel­ing many of them.” Peo­ple in pow­er were ​“delib­er­ate­ly [wield­ing] for­mu­las to impress rather than clar­i­fy.” This dis­il­lu­sion­ment led O’Neil to get involved with Occu­py Wall Street and start edu­cat­ing the pub­lic about the dan­gers of WMDs through her blog, MathBabe.

She is care­ful to point out that there is noth­ing inher­ent­ly destruc­tive about math­e­mat­i­cal mod­el­ing. Sophis­ti­cat­ed data mod­el­ing enables much of mod­ern tech­nol­o­gy, from wire­less com­mu­ni­ca­tion to drug dis­cov­ery. How can one dis­tin­guish a destruc­tive math mod­el from an ordi­nary, or even help­ful, one? O’Neil iden­ti­fies three key fea­tures of WMDs: lack of trans­paren­cy, lack of fair­ness and, most impor­tant­ly, oper­a­tion on a mas­sive scale.

O’Neil grounds her argu­ment in case stud­ies of WMDs in a vari­ety of set­tings: finance, high­er edu­ca­tion, the crim­i­nal jus­tice sys­tem, online adver­tis­ing, employ­ment deci­sions and sched­ul­ing, and cred­it and insur­ance pro­vi­sion. The ​“val­ue-added” mod­el for teacher eval­u­a­tion, which looks at improve­ments in indi­vid­ual stu­dents’ test scores — a favorite of the so-called ​“edu­ca­tion­al reform” move­ment — is tout­ed as an objec­tive mea­sure of a teacher’s worth. Yet this is far from the truth. O’Neil cites an analy­sis by blog­ger and edu­ca­tor Gary Rubin­stein of New York’s 2010 val­ue-added scores for pub­lic school teach­ers. O’Neil explains that ​“Of teach­ers who taught the same sub­ject in con­sec­u­tive years, one in four reg­is­tered a 40-point dif­fer­ence. [This] sug­gests that the eval­u­a­tion data is prac­ti­cal­ly ran­dom.” O’Neil argues that this is because the val­ue-added mod­el, which relies on pre­dic­tions of stu­dent per­for­mance, suf­fers from a built-in log­i­cal flaw: No sta­tis­ti­cal mod­el can accu­rate­ly make pre­dic­tions about a class of 25 or 30 stu­dents — the sam­ple size is too small. Yet the high-stakes test­ing régime con­tin­ues to wreak hav­oc on the tra­jec­to­ries of stu­dents and teach­ers alike.

Oth­er WMDs, known as e‑scores, use data such as ZIP codes, web-surf­ing pat­terns and recent pur­chas­es to eval­u­ate a person’s cred­it-wor­thi­ness. Unlike the more famil­iar FICO cred­it scores that are freely avail­able and reg­u­lat­ed by the gov­ern­ment, these secre­tive e‑scores are ​“unac­count­able, unreg­u­lat­ed and often unfair.” Where­as FICO scores are based on your own finan­cial his­to­ry, e‑scores com­pare you to oth­er peo­ple with sim­i­lar pro­files. This may seem benign, but it can result in feed­back loops that rein­force exist­ing social inequities. If you live in a poor ZIP code, then your e‑score will drop, mean­ing less cred­it and high­er inter­est rates — essen­tial­ly, an algo­rith­mic redlin­ing of the poor and work­ing class.

Through these exam­ples, O’Neil makes a con­vinc­ing case that many math­e­mat­i­cal mod­els today are engi­neered to ben­e­fit the pow­er­ful at the expense of the pow­er­less. The prob­lem with most WMDs, O’Neil argues, is that it’s ​“almost always their objec­tive” to max­i­mize ​“the bot­tom line” rather than ​“fair­ness” or oth­er ​“eth­i­cal pri­or­i­ties.” By enact­ing these choic­es, WMDs serve an impor­tant ide­o­log­i­cal func­tion in mod­ern cap­i­tal­ism. Far from elim­i­nat­ing human bias, O’Neil sug­gests, WMDs ​“cam­ou­flage it with technology.”

To para­phrase the great evo­lu­tion­ary biol­o­gist Richard Lewon­tin, WMDs are ​“math as ide­ol­o­gy.” This argu­ment is implic­it through­out Weapons of Math Destruc­tion, but the book would have ben­e­fit­ed from an explic­it dis­cus­sion of the rela­tion­ship between WMDs and cap­i­tal­ism. This is a minor short­com­ing, how­ev­er. Cathy O’Neil has writ­ten an enter­tain­ing and time­ly book that gives read­ers the tools to cut through the ide­o­log­i­cal fog obscur­ing the dan­gers of the Big Data revolution.