I’ve decided lately that people regularly get confused, on a number of subjects, by the difference between science and engineering. (I want to say this post was inspired by a number of recent confusions. But it’s been gestating for at least a month, so I don’t think any of them are particularly recent any more).

This is an effortpost. Long effortpost is long. You have been warned.

Tl;dr: Science is sensitive and finds facts; engineering is robust and gives praxes. Many problems happen when we confuse science for engineering and completely modify our praxis based on the result of a couple of studies in an unsettled area.

(Thanks to Cowbirds in Love for the perfect comic strip)

The difference between science and engineering

The difference as I see it: science is finding out facts about the world. Engineering, in contrast, is the technique of using science to produce tools we can consistently use in the world. Engineering produces things that have useful effects. (And I’ll also point to a third category, of “folk traditions,” which are tools we use in the world that are not particularly founded in science.)

These things are importantly different. Science depends on a large number of people putting together a lot of little pieces, and building up an edifice of facts that together give us a good picture of how things work; it’s fine if any one experiment or study is flawed, because in the limit of infinite experiments we figure out what’s going on. (See slatestarscratchpad’s essay Beware the Man of One Study.

Similarly, it’s fine if any one experiment holds in only very restricted cases, or detects a subtle effect that can only be seen with delicate machinery. The point is to build up a large number of data points and use them to generate a model of the world.

Engineering, in contrast, has to be robust. If I want to detect the Higgs Boson once, to find out if it exists, I can do that in a giant machine that costs billions of dollars and requires hundreds of hours of analysis. If I want to build a Higgs Boson detector into a cell phone, that doesn’t work.

This means two things. First is that we need to understand things much better for engineering than for science. In science it’s fine to say “The true effect is between +3 and -7 with 95% probability”. If that’s what we know, then that’s what we know. And an experiment that shrinks the bell curve by half a unit is useful. For engineering, we generally need to have a much better idea of what the true effect is. (Imagine trying to build a device based on the information that acceleration due to gravity is probably between 9 and 13 m/s^2).

Second is that science in general cares about much smaller effects than engineering does. It was a very long time before engineering needed relativistic corrections due to gravity, say. A fact can be true but not (yet) useful or relevant, and then it’s in the domain of science but not engineering.

Why does this matter?

The distinction is, I think fairly clear when we talk about physics. In particular, we understand the science of physics quite well, at least on every-day scales. And our practice of the engineering of physics is also quite well-developed, enough so that people rarely use folk traditions in place of engineering any more. (“I don’t know why this bridge stays up, but this is how daddy built them.”)

But people get much more confused when we move over to, say, psychology, or sociology, or nutrition. Researchers are doing a lot of science on these subjects, and doing good work. So there’s a ton of papers out there saying that eggs are good, or eggs are bad, or eggs are good for you but only until next Monday or whatever.

And people have, often, one of two reactions to this situation. The first is to read one study and say “See, here’s the scientific study. It says eggs are bad for you. Why are you still eating eggs? Are you denying the science?” And the second reaction is to say that obviously the scientists can’t agree, and so we don’t know anything and maybe the whole scientific approach is flawed.

But the real situation is that we’re struggling to develop a science of nutrition. And that shit is hard. We’ve worked hard, and we know some things. But we don’t really have enough information to do engineering, to say “Okay, to optimize cardiovascular health you need to cut your simple carbs by 7%, eat an extra 10g of monounsaturated fats every day, and eat 200g of protein every Wednesday” or whatever. We just don’t know enough.

And this is where folk traditions come in. Folk traditions are attempts to answer questions that we need decent answers to, that have been developed over time, and that are presumably non-horrible because they haven’t failed obviously and spectacularly yet. A person who ate “Like my grandma” is probably on average at least as healthy as a person who tried to follow every trendy bit of scientistic nutrition advice from the past thirty years.

Trendy teaching as confusing science for engineering

So where do I see this coming up other than nutrition? Well, the subject that really got me thinking about it was “scientific” teaching practices. I posted a while back about a workshop on clickers I was attending, and got comments disparaging cargo cult teaching methods.

In general there’s a big split among university professors between people who want to teach in a more “traditional” way and people who want to teach in a more “scientific” way. With bad blood on both sides.

And my biggest problem with the “scientific” side is that some of their studies are so bad. I’d like good studies on teaching methods. I’d like a good engineering of teaching. But we don’t have one yet, and acting like “we have three studies, now we know the best thing to do” is just silly.

(Which shouldn’t be read as full-throated support for the “traditionalists”! The science is good enough to tell us some things about some things, and I do try to engage in judicious supplementation of folk teaching traditions with information from recent research. But the research is not in a good enough state to be dispositive, or produce an engineering discipline, or completely displace the folk tradition).

Other examples

A few weeks back lambdaphagy complained about the sad state of exercise science. I think he was really complaining about the lack of exercise engineering. We are doing basic research that tells us about how the body responds to exercise. We don’t know enough to give advice that improves much on “do the things people have been doing for a while that seem to work”.

A lot of “lifehacks” boil down to “We read a study, and based on this study her are three simple things you can do to X.” A study is science, not engineering. Sometimes helpful, but easy to overinterpret. Don’t take any one study too seriously, and if what you’re doing works, don’t totally overhaul it because you read a study.

Similarly for any comment about how you can be more effective socially by doing this one trick. Usually science, not engineering.

Lots and lots of economics and public policy debates sound like this. “This study shows that raising the minimum wage (increases/decreases/has no effect on) unemployment.” All three of those statements can be true! We’re starting to develop an engineering practice of economics policy, but it’s in its infancy at best.

Or see this essay’s account of scientifically studying the most effective way for police to respond to domestic violence charges, for a good example of confusing science and engineering. Bonus points for the following quote:

Reflection upon these results led Sherman to formulate his “defiance” theory of the criminal sanction, beginning with the inauspicious generalization that, according to the evidence, “legal punishment either reduces, increases, or has no effect on future crimes, depending on the type of offenders, offenses, social settings and levels of analysis.” This is a fancy way of saying “we don’t know what works.”

Marketing: engineering versus folk traditions

The field of marketing presents a good contrast between engineering and folk traditions. We have a mental image of a sleazy salesman, who has a whole host of interpersonal tactics that have been honed through centuries and millenia of sleazy sales tactics. And this works.

And there’s an entirely different field of marketing research and focus groups. And this shows what’s necessary to turn science into engineering. There’s a whole bunch of basic research about psychology that goes into designing marketing campaigns. But people also do focus groups, to gather a ton of data on how people respont to minute differences.

And, more importantly, they do A/B testing, which gives pretty good data on how actual people respond to actual differences. And by iterating a ton of A/B testing, you have a pretty good idea that people will buy 5% more if you use the green packaging, or whatever.

Science, Engineering, and the LW-sphere

This confusion is particularly endemic, I think, in the LW-sphere. Or at least I notice it more there, but it’s also a bunch of people I talk to a lot. In fact, sometimes they’re very good. From what I can tell, CFAR is basically an attempt to start from the (plentiful) science of human cognitive errors and biases, and develop an engineering of good-and-rational human thought.

(Insofar as they manage to do this, it would be wonderful. I’m extremely skeptical. But it’s entirely possible they’ll develop a somewhat-useful folk tradition).

But there are two particular places I think this confusion manifests itself strongly. The first is certain types of disregard for expert opinion. There’s a certain strain of thought in the community that reads basically like “People think X is the right way to do things, but I’ve read three studies and rationally determined that the experts are wrong, and we definitetly should do Y.” I suspect in many of these cases the skepticism of X is at least somewhat justified, but the confidence that we’ve scientifically determined that Y is good is not justified.

The more interesting one is something that su3su2u1 has complained about a lot. To a certain extent I think my split between engineering and science tracks his split between “I fucking love science” and people who have actually participated in the practice of science. Do we like science because of curiosity about the nature of the world? OR because we can accomplish these things and make awesome pictures? That’s science, not engineering.

Somewhere in the discussion of HPMOR (edit: it was raginrayguns here), someone suggested that fantasy authors use magic to skip the boring bits that link things they’re actually interested in. So it’s true, in the real world, that knowing more about the way the world works leads to having more power and ability to accomplish things. But it leads to this by means of developing an engineering praxis.

In HPMOR, understanding more things about how the world works directly leads to more power, just by having the knowledge. No engineering is necessary; you learn new facts and then you’re ipso facto more powerful. This simultaneously devalues engineering (by eliminating it as a thing), and misses the point of science for most people who are actually into science.