These aren't new insights.

They're core to the classical liberal tradition and its skepticism about the ability of policymakers to innovate via central planning. If Uncontrolled were merely a restatement of the need for epistemic humility among wonks and legislators, interest in it might be confined to the right. The book is of broader interest, and may turn out to be important, because its author makes a compelling argument for an ideologically neutral method for improving policy, one that left and right might both plausibly embrace, even as it challenges both sides to rethink some of their reflexes.

That method is a specific kind of experimentation.

The first part of Uncontrolled is an enjoyable summary of how philosophers, scientists, medical researchers and corporations all discovered -- at different times and places -- that conducting lots of rigorous experiments is the most reliable way to figure out what works and what doesn't work. As Manzi takes pains to demonstrate, the gold standard in experimentation is the double-blind, randomized field trial with a control group. Fortunately, technology has made conducting these sorts of experiments more cost effective than ever.

But they're rare in public policy.

That's partly due to their limits. "The reason we have increasing trouble building compact and comprehensive predictive theories as we go from physics to biology to social science is the increasing complexity of the phenomena under investigation," Manzi writes, which also makes it "far harder to generalize the results of experiments."

For example:

We can run a clinical trial in Norfolk, Virginia, and conclude with tolerable reliability that "Vaccine X prevents disease Y." We can't conclude that if literacy program X works in Norfolk, then it will work everywhere. The real predictive rule is usually closer to something like "Literacy program X is effective for children in urban areas, and who have the following range of incomes and prior test scores, when the following alternatives are not available in the school district, and the teachers have the following qualifications, and overall economic conditions in the district are within the following range." And by the way, even this predictive rule stops working ten years from now, when different background conditions obtain in the society.



But that's no reason to give up. Government is just going to be a supplier of social welfare, education, policing, defense, medical research, and more. Thus the book's suggestion: conduct lots of rigorous experiments to test the effectiveness of every new policy that's implemented.

Make that rigor a reflexive part of government culture.

Says Manzi:

For the hypothetical literacy program described above, an experiment to test the program is not really a test of the program; it is a test of how well the program applies to a specific situation.



A brute-force approach to this problem would be to run not one experiment to evaluate whether this program works, but to run hundreds or thousands of experiments to evaluate the conditions under which it works. If it could be tested in a very large number of school districts, we might very well discover some useful approximation to the highly conditional rule that predicts its success. This is the opposite of elegant theory-building... But it might provide practically useful information.



Confident that its now plausible to do lots of large-scale, cost-effective policy experiments, Manzi wants the federal government to create an agency charged with running and refereeing them. And he wants states to enjoy greater freedom to experiment, if they agree to rigorously test the efficacy of everything they do, making the results transparent so best practices can spread.