The Understanding and Misunderstanding of Imperfect Information

Last Friday on his blog, Timothy Taylor, editor of the Journal of Economic Perspectives, wrote about whether imperfect information strengthens or weakens the case for free markets and for deregulation. Taylor frames his discussion by comparing and contrasting two recent papers. One paper, “Friedrich Hayek and the Market Algorithm,” by Samuel Bowles, Alan Kirman and Rajiv Sethi, appeared in the Journal of Economic Perspectives; the other, “The Revolution of Information Economics: the Past and the Future,” by Joseph Stiglitz is a NBER working paper. Although I agree with much of what Taylor has to say, I think he, like many others, misses some important distinctions and nuances in Hayek’s thought. Although Hayek’s instincts were indeed very much opposed to any form of government intervention, that did not prevent him from acknowledging that there is a very wide range of government action that is not inconsistent with his understanding of liberal principles. He was, in fact, very far from being the dogmatic libertarian anti-interventionist for which he is mistaken. So I am going to try to put things in a clearer perspective.

Taylor begins by referencing Hayek and the paper by Bowles, Kirman and Sethi.

Friedrich von Hayek (Nobel 1974) is among the most prominent of those who have made the case that imperfect information strengthens the case for free markets. . . . In one much-quoted example, Hayek offers a discussion of what happens in the market for some raw material, like tin, when “somewhere in the world a new opportunity for the use” arises, or “one of the sources of supply of tin has been eliminated.” Either of these changes (rise in demand, or a fall in supply) will lead to a higher market price. But as Hayek points out, no company that uses tin, nor any consumer who uses products made with tin as an ingredient, needs to know any details about what happened. No commission of government officials needs to meet to discuss how every firm and consumer should be required to react to this change in the price of tin. No government quota system for allocation of tin supplies needs to be established. No special government program for research and development into cheaper substitutes for tin, and no government-subsidized producers for potential-but-still-costly substitutes needs to be created. Instead, the shifts in demand or supply, and the corresponding changes in price, work themselves out with a larger number of small-scale shifts in the market. A government agency might collect information on who currently produces and uses tin. But that government lacks the granular information about all the different alternatives that might possibly be used for tin, and any sense of when a user of tin would be willing to pay twice as much, or when a user of tin would shift to a substitute if the price rose even a little. Indeed, this granular information about the tin market is not even theoretically available to a government planner or regulator! Many users of tin, or potential suppliers of additional tin, or potential suppliers of substitutes, don’t actually know just how they would react to the higher price until after it happens. Their reactions emerge through a process of trial and error. Hayek’s point becomes even more acute if one considers not just existing basic products, like tin, but the potential for innovative new products or services. One can make a guess about whether a certain type of new smartphone, headache remedy, spicy sauce, alternative energy source, or water-in-a-bottle will be popular and desired. But government planners–especially given that they are operating under political constraints–won’t have the knowledge to make these decisions. Hayek’s point is not only that government economic planners not only that government planners lack perfect information, but that it is not even theoretically possible for them to have perfect information–because much of the information about production, consumption, and prices does not exist. thus, Hayek wrote: [The market is] a system of the utilization of knowledge which nobody can possess as a whole, which. . . leads people to aim at the needs of people whom they do not know, make use of facilities about which they have no direct information; all this condensed in abstract signals. . . [T]hat our whole modern wealth and production could arise only thanks to this mechanism is, I believe, the basis not only of my economics but also much of my political views. . .

Taylor, channeling Bowles, Kirman and Sethi, is here quoting from a passage in Hayek’s classic paper, “The Use of Knowledge in Society” in which he explained how markets accomplish automatically the task of transmitting and processing dispersed knowledge held by disparate agents who otherwise would have no way to communicate with each other to coordinate and reconcile their distinct plans into a coherent set of mutually consistent and interdependent actions, thereby achieving coincidentally a coherence and consistency that all decision-makers take for granted, but which none deliberately sought. The key point that Hayek was making is not so much that this “market order” is optimal in any static sense, but that if a central planner tried to replicate it, he would have to collect, process, and constantly update an impossibly huge quantity of information.

After describing Hayek’s explanation of why imperfect information – a term that for Hayek involved both the dispersal of existing knowledge and the discovery of new knowledge – implies that markets are a better mechanism than central planning for coordinating a complex network of interrelated activities, Taylor turns to Stiglitz’s paper on imperfect information.

Joseph Stiglitz (Nobel, 2001) is among the best-known of those who have explained how imperfect information can hinder the functioning of a market, and thus offer a justification for government intervention or regulation. Stiglitz offers a readable overview of his perspective in “The Revolution of Information Economics: The Past and the Future” (September 2017, National Bureau of Economic Research Working Paper 23780). The paper isn’t freely available online, although readers may have access through a library subscription, but a set of slides from when he presented a talk on this topic at the World Bank in 2016 are available here. Stiglitz emphasizes two particular aspects of imperfect information: it leads to a lack of competition and especially to problems in the financial sector. He writes: The imperfections of competition and the absence of risk markets with which they are marked matter a great deal. . . . And in those sectors where information and its imperfections play a particularly important role, there is an even greater presumption of the need for public policy. The financial sector is, above all else, about gathering and processing information, on the basis of which capital resources can be efficiently allocated. Information is central. And that is at least part of the reason that financial sector regulation is so important. Markets where information is imperfect are also typically far from perfectly competitive. . . In markets with some, but imperfect competition, firms strive to increase their market power and to increase the extraction of rents from existing market power, giving rise to widespread distortions. In such circumstances, institutions and the rules of the game matter. Public policy is critical in setting the rules of the game.

There’s a lot going on here, and I think it’s a mistake to set up Hayek and Stiglitz as polar opposites. Although they surely are not in total agreement, Hayek did agree that the perfect-competition model is not descriptive of most actual markets. Hayek may have had a more benign view of the operation of “imperfect” competition than Stiglitz, but he certainly did not view perfect competition as a normative ideal in terms of which the performance of actual economies should be assessed. It is certainly true that imperfectly competitive firms attempt to increase their market power, either by colluding or by tacit understandings to refrain from “ruinous” competition, but perfectly competitive firms also seek to collude on their own or try to enlist the government to help restrain competition that drives profits down to – or even below – zero.

And it would be hard to think of a statement with which Hayek would have been less likely to disagree than this one: “public policy is critical in setting the rules of the game.” To suggest that Hayek conceived of a market economy as a system operating independently of the constraints of an evolving and increasingly sophisticated system of rules is to completely misunderstand Hayek’s conception of a market order and the legal underpinnings without which no such order could come into existence. The ideal of a free market is not for businesses and entrepreneurs to be able to do whatever they want, but for all agents to be subject to a system of general rules that lays out the acceptable means by which every individual may pursue his interests and try to achieve goals of his own choosing. Taylor continues:

Stiglitz also argues that in a modern economy, concerns over information are likely to become more acute. Looking forward, changes in structure of demand (that is, as a country gets richer, the mix of goods purchased changes) and in technology may lead to an increased role of information and increased consequences of information imperfections, decreased competition, and increasing inequality. Many key battles will be about information and knowledge (implicitly or explicitly)—and the governance of information. Already, there are big debates going on about privacy (the rights of individuals to keep their own information) and transparency (requirements that government and corporations, for instance, reveal critical information about what they are doing). In many sectors, most especially, the financial sector, there are ongoing debates about disclosure—obligations on the part of individuals or firms to reveal certain things about their products.

Taylor misses an opportunity here to dig deeper into Stiglitz’s analysis of what makes imperfect information so problematic. The most serious problems arise when substantial information asymmetries exist, allowing better-informed agents to make trades that exploit the ignorance or gullibility of their counterparties. Though not confined to the financial sector – the health sector being another area in which information asymmetries are especially acute and potentially disastrous to the relatively uninformed party – existing information asymmetries create opportunities and incentives for reprehensible behavior by financial institutions while encouraging them to engage in tireless efforts to find or create additional information asymmetries, devoting valuable resources to the search for and creation of those asymmetries.

In many previous posts, I have discussed how the financial sector, when seeking to profit from transitory informational advantages by anticipating short-term price movements, or by creating new financial products that counterparties do not understand as well as their creators do, wastes resources on a massive scale. The net social product of such activity is far less than the private gains reaped from those fleeting informational advantages. But Wall Street banks and other financial institutions pay huge salaries to the very bright people who help create these momentary informational advantages and these new financial products. The actual and potential harms created by the existence – and, even worse, the pursuit – of such information asymmetries calls for serious analysis and creative thinking to correct, or at least mitigate, the malincentives that lead to such socially wasteful activity. And I can’t think of any reason why Hayek would have opposed changing “the rules of the game” to correct those malincentives. So the idea that reforming the legal framework within which markets operate to eliminate inefficient malincentives somehow is indicative of hostility to or skepticism about free markets, an idea that seems to underlie much of what Taylor and Stiglitz are saying, is entirely misplaced.

Which is not to say that it is easy to change the rules to fix every malincentive besetting the market economy; some malincentives may be truly intractable. But when malincentives truly are intractable – a state of affairs that, unfortunately, is closer to being the rule than the exception — it is usually not obvious what the appropriate policy response is. The problem is compounded many times over, because the theory of second best teaches us that, as soon as there is a single departure from optimality, satisfying all the other optimality conditions will not achieve the next best outcome. A single departure from optimality in one market requires departures from optimality in all related markets, so trying to satisfy optimality conditions in n-2 out of n markets doesn’t get you to the second best outcome.

In the end Taylor tries to suggest an awkward reconciliation between the supposedly opposing visions of Hayek and Stiglitz.

Both Hayek and Stiglitz use a similar “straw man” argumentative tactic: that is, set up a weak position as the opposing view, and then set it on fire. Hayek’s preferred straw man is government economic planners who seek to dictate every economic decision. He was writing in part with economic systems like the Communist Soviet Union in mind. But arguing that a market is better than wildly intrusive and weirdly over-precise old-time Soviet-style economic planning doesn’t make a case against more restrained and better-aimed forms of economic regulation. Indeed, Hayek occasionally expressed support for a universal basic income and for certain kinds of bank regulation.

I get what Taylor is trying to say, but I’m afraid he has phrased it rather badly. As Taylor actually seems to recognize, Hayek wasn’t just arguing against a straw man, which suggests creating an opposing argument to refute that no one really believes in. But that was hardly the case in the 1930s and 1940s when Hayek was first making his systematic argumeents against central planning by thinking carefully about what knowledge we actually are assuming that individual agents possess in standard economic models, and what knowledge a central planner would need in order to replicate the optimal state of affairs that is associated with the equilibrium of the standard economic model. And in the post-neoliberal political environment in which we now find ourselves, it is not clear that what not so long ago seemed like a straw man has not come back to life.

However, Taylor’s assessment of Stiglitz seems to me to be pretty much on target.

Stiglitz’s straw man is a free market that operates essentially without government intervention or regulation. He likes to emphasize that in the real world of imperfect information, there is no conceptual reason to presume that markets are efficient. But arguing that imperfect information can offer a potential justification for government regulation doesn’t make a case that all or most government regulation is justified. especially given that the real-world government regulators labor with their own problems of political constraints and limited information. And indeed, while Stiglitz tends to favor an increase in US economic regulations in a number of specific areas, his vision of the economy always leaves a substantial role for private sector ownership, decision-making, and innovation.

Taylor sums up this confused state of affairs with two quotations. The first from Scott Fitzgerald. “The true test of a first-rate mind is the ability to hold two contradictory ideas at the same time.” Taylor adds:

In this case, the contradictory ideas are that markets can often be a substantial improvement on government regulators, and government regulators can often be a substantial improvement on unconstrained market outcomes.

Taylor then quotes Joan Robinson: “[E]conomic theory, in itself, preaches no doctrines and cannot establish any universally valid laws. It is a method of ordering ideas and formulating questions.” And, if we are lucky, coming up with some conjectures that might answer those questions.

But before closing, I would add another quote from the paper by Bowles, Kirman and Sethi, which seems to me to penetrate to the core of the problem of imperfect information:

[W]e wish to call into question Hayek’s belief that his advocacy of free market policies follows as a matter of logic from his economic vision. The very usefulness of prices (and other economic variables) as informative messages—which is the centerpiece of Hayek’s economics—creates incentives to extract information from signals in ways that can be destabilizing. Markets can promote prosperity but can also generate crises. We will argue, accordingly, that a Hayekian understanding of the economy as an information-processing system does not support the type of policy positions that he favored. Thus, we find considerable lasting value in Hayek’s economic analysis while nonetheless questioning the connection of this analysis to his political philosophy.

My only quibble with their insightful comment is that Hayek’s political philosophy did not necessarily exclude a role for government intervention and regulation, provided that interventions and regulations satisfied appropriate procedural standards of generality and non-arbitrariness. Hayek’s main concern was not to make government small, but to subject all laws and regulations enacted by government to procedural conditions ensuring that the substantive content of legislation and regulation does not aim at achieving specific concrete objectives, e.g., a particular distribution of income or the advancement of a particular special interest, but at making markets function more smoothly and more predictably, e.g., by prohibiting anticompetitive or collusive agreements between business firms. In principle, measures such as guaranteeing a minimum income, or providing medical care, to all citizens, prohibiting or taxing pollution by manufacturers or unduly risky behavior by financial institutions, is not incompatible with that philosophy. The advisability of any specific law or regulation would of course depend on an appropriate weighing of the expected costs and benefits of imposing such a law or regulation.