We’re all used to economic forecasts. We’re also used to them being wrong. But there was a time when forecasts were new and exciting, and people were genuinely surprised when they didn’t pan out. This was during the first decades of the previous century, an era that Harvard Business School historian Walter Friedman chronicles in his new book Fortune Tellers: The Story of America’s First Economic Forecasters.

The forecasters Friedman profiles are Roger Babson, founder of Babson College, whose simple “Babsonchart” of economic activity helped make him probably the most famous of the lot during his lifetime; Irving Fisher, the Yale professor whose 1929 pronouncements about a “permanently high plateau” for stock prices are mocked to this day but whose fingerprints are also all over modern academic economics; John Moody, who along with making economic forecasts created the now-controversial credit ratings business; and C.J. Bullock and Warren Persons of the Harvard Economic Service, who are almost completely forgotten now but briefly wielded global influence. Friedman also devotes a chapter to Columbia economist Wesley Mitchell and Commerce Secretary (that was his job until he was elected president in 1928) Herbert Hoover, who weren’t forecasters but whose joint interest in the business cycle led to the creation of many of the economic indicators we use today.

The book is fun reading, but what makes it especially interesting for any regular consumer of modern economic forecasts is how little today’s forecasters have improved on the methods of the 1910s and 1920s. Yes, there’s better data and more sophistication in today’s forecasts, but the basic principles were already being worked out by Babson, Fisher, and the like a century ago.

I interviewed Friedman about his book for an HBR Ideacast, which you can listen to below. What follows are edited excerpts from that and a post-Ideacast conversation.

Download this podcast

There’s a lot of failure in your book.

Friedman: There is a lot of failure in my book, but I think forecasting is a very maligned industry. Most people look at the history of forecasting and think of it as a march of folly, because they look at whether [forecasters] have become more accurate over time, and the truth is they really haven’t. No one’s been able to produce a reliable method of making predictions of where the economy’s going, and they won’t be able to. But in the effort to forecast they’ve done a lot of great things — they’ve introduced leading indicators, they’ve created econometric models, they’ve created important institutions like the National Bureau of Economic Research, and they’ve come up with ideas of how to conceptualize the economy.

The very idea that there’s this thing called the economy was something new.

Everybody talks about Silicon Valley being this bunch of entrepreneurs who created the computer industry in their garages. This was a bunch of entrepreneurs who gathered data either in their basements or in some subdivision of their office, and invented the idea of the economy. They made it tangible, so that people got a sense that the businesses around the United States were connected in some way, that they all went up and down together, and that what somebody did in Cleveland had an impact on what somebody did in New York.

Your book describes a rivalry between Babson and Fisher.

Babson really believes in historical trends. He’s just a trend analyst. Fisher doesn’t think trends are important at all; instead he believes in causation. He tries to figure out how the economy actually works. What effect does it have if we innovate? What effect does it have if we change the way we manage firms? What effect does it have if we plow earnings back [into firms]? Fisher tried to find ways to quantify these changes, and the effects they would have.

In the late 20s the Babson approach and the Fisher approach came into direct collision. Who won that battle?

If you were an investor, you liked Babson much better, because by luck he got it right in fall of 1929. Whereas Fisher, who gets it wrong in 1929 — and loses everything, including his house — his ideas start to spread in the post-World War II period. Fisher’s ideas really catch on, whereas Babson is much more marginal as a figure.

So this move to a more theoretical approach, has it resulted in better forecasts?

I think the move in forecasting that really has made a difference is the transition from the idea of a fixed business cycle — that is to say, a business cycle that you can’t alter in any way and that is a bit like a meteorological cycle — to one in which the government can actually try to flatten cycles. That really has changed things.

Is there a risk that you get overconfident because of that? One of the things that happened in the ’20s is that things kept going so well, and the forecasters who predicted a continued boom were right again and again. If you get very confident that government knows what it’s doing and can manage the economy well, does that increase risk at all?

I think the real danger is believing that any forecaster really knows what he or she is doing. One of the problems that people have with forecasting is that they tend to look at past accuracy as an indication whether or not to follow a forecaster, when in fact past accuracy is much more often based on luck, and what you really have to do is try to figure out what these forecasters are thinking about, what their reasons are, and then you can decide whether you want to follow one or another. If you just blindly look as to whether they got it right or wrong, then you’re probably going to fall for a guru one of these days.

That fits a theme that goes through this book, which is that you shouldn’t really entirely trust any of these people, yet at the same time they’re valuable to the economy.

I think you need to think of forecasts a bit like treasure maps, in that you always have to be skeptical of the person who would want to print and sell treasure maps en masse, when it’s much better for you to simply keep a treasure map, if you actually have a valid one, and get the treasure yourself. You have to keep in mind that these people are trying to sell their predictions, and that persuasion is as big a part of this industry as prediction is.

Who’s your favorite?

I think Babson is my favorite, because it’s just so hard to figure out whether he believed what he was promoting or not. He builds his whole idea about forecasting on Sir Isaac Newton, yet how can a rational person actually believe that booms and busts precisely equal out over time? And what does that say about the craziness of the profession? Who actually thinks that they can figure out where the economy’s headed with some certainty? You can’t really publish forecasts unless you have a conviction that you’ve figured something out. And yet most rational people go about that only with some real doubt.

The people who doubt ultimately will fail, because when they get a prediction wrong they’ll throw up their hands, they’ll say their model was wrong. And those who really believe that they’ve got the right method will just explain away all of their failures and just keep pushing their model.

That sounds a lot like Philip Tetlock’s hedgehogs and foxes, where the people who keep hammering at it and don’t care if their forecasts are wrong are more likely to be wrong. It’s the foxes, the people who are willing to take in new information and don’t have one overarching theory of the world, who are better at prediction. But what you’re saying is you’re never going to make it in the forecasting business if you have that approach.

I think there’s a big marketing side to this story, and the ones who were good marketers ultimately succeeded. The pioneers were all basically people with a very strong sales sense. Moody succeeds because he’s very aggressive at selling. He tells these stories about going on the train and putting his prospectus on every open seat. Babson sends his salesmen all around the country. Fisher believes in syndicating columns and getting his predictions in front of every newspaper reader every Monday, which is another smart thing to do. They’re all very good at that.

The ones that are really bad at that are the Harvard group. They’re completely useless. They believe that they’ll just basically market to their alumni. Well, it’s a very small pool, so they don’t make any money, and partway through the ’20s they just decide that they can’t make any money doing forecasts anyway, so that they’ll just try to keep enough [subscribers] so that they’ll make their budget every year and try to influence the field rather than make money.