For all the recent media attention to power utilities, most coverage has been about symptoms rather than root causes. There's a battle over rooftop solar here, a coal or nuke bailout there, a fight against efficiency over there, but casual news consumers are offered no way of making sense of these battles or how they fit into a larger story. They're left with the vague impression that utilities hate clean energy out of sheer greed or malice. And that's not quite right.

Greed and malice are definitely involved. But they, too, are symptoms. No matter how individually virtuous utility executives may be, these running battles between utilities and clean energy will continue until the root problem is addressed and solved.

The root problem is simple: It's the way utilities are structured. They are monopoly providers of a whole bundle of electricity services in a given geographic area. But technology has evolved to the point that many of those services could be provided just as reliably, or better, by participants in competitive markets — if there were any such markets. Competitors keep trying to squeeze into the electricity space, and utilities keep using their monopoly power to try to squeeze them back out. That's what all the fights are about.

There's no longer any compelling reason for all those services to be bundled by a single "vertically integrated" monopoly. The only thing left that calls for monopoly control is the distribution grid itself, managing it and interfacing with customers. As for the rest — electricity generation, procurement, and management — they should be "unbundled," spun off into competitive markets to accelerate innovation.

Until regulators divest utilities of monopoly control over electricity services, there will be fights between utilities and emerging competitors. That's the root of the issue.

Changing that regulatory structure is a daunting prospect, for a whole range of reasons. The basic utility model has been around for nigh a century without changing much. Utilities and their regulators have developed cozy, familiar relationships. Those who benefit from the status quo have more access to legislators. And the general public is totally tuned out.

It's time to tune in.

When utilities got started, giant vertical monopolies made sense

The early-20th-century history of electricity and electric utilities is fascinating, but we're not going to dwell on it, except to say that it produced two key technologies.

The first was steam turbines, which are vastly more efficient at generating power and more scalable than the reciprocating steam engines they replaced. The second was alternating current (AC), which could carry power vastly greater distances than the direct current (DC) used in local distribution grids.

Together, these technologies enabled power plants to get bigger and bigger and farther and farther away from population centers, even as the power they generated and delivered got cheaper and cheaper.

This created two underlying structural conditions for the electricity market:

extremely high barriers to entry , because big power plants and long-distance transmission lines were very expensive

, because big power plants and long-distance transmission lines were very expensive enormous economies of scale, because the average cost of delivered power got cheaper with every new expansion of demand

Those two conditions yield what economists call a "natural monopoly." Given the high fixed costs of building plants and power grids, it didn't make sense to have multiple companies making those investments. And given the economies of scale, it didn't make sense to have multiple companies competing over the same set of customers. Instead, for a given geographic area, it made sense for one entity to do it all — build the power plants, generate the power, and deliver it to customers.

Because progressive reformers at the time didn't want a repeat of much-hated railroad monopolies, they came up with what is informally known as the "regulatory compact." In exchange for monopoly access to customers within a geographic territory, the utility would provide reliable, nondiscriminatory (available to everyone) power at lowest cost.

As quasi-public entities, utilities cannot profit. However, they can charge whatever rates are necessary to cover their fixed (infrastructure) and variable (fuel) costs and provide a reasonable return to investors. To ensure that their investments, rates, and returns are reasonable, they are overseen by state-based public utility commissions, which have access to their financial records and final say over their rates.

This model was designed for one purpose above all: to electrify the country. Utilities had every incentive to build, build, build, knowing that the more they built, the higher their returns — it was guaranteed by law. The more they built, the cheaper the power got; the cheaper the power got, the more people used; the more people used, the more utilities needed to build. It was an enormous, rapid expansion of economic activity and public welfare. Everyone benefited; everyone was happy.

Then, in the latter half of the 20th century, things started changing. The logic of the model began to crumble. The model, however, lives on.

Giant vertical monopolies have stopped making sense

Let's take a quick detour into economic theory. (I'm drawing here from a fascinating paper by Lynne Kiesling of Northwestern University. Kiesling co-authors the blog Knowledge Problem, which is recommended reading.)

When a firm is "vertically integrated," it owns most or all of its own supply chain, encompassing a variety of products and services. When does it make sense for a firm to be vertically integrated? Kiesling identifies two key factors:

economies of scale , which create high barriers to entry, usually in the form of high fixed costs

, which create high barriers to entry, usually in the form of high fixed costs high transaction costs, i.e., high costs (in time, money, or risk) of contracting with outside firms for services versus extending the boundary of your own firm to include them. Put colloquially, getting someone else to do a task (or make a part, etc.) safely, well, and on time is sometimes too much of a pain in the ass, so it makes sense to do it yourself.

Vertical integration can have benefits. Consider Tesla, Elon Musk's electric car company. The barriers to entry in the auto industry are extremely high (especially given Musk's goal to create a mainstream electric car market), and there are enormous economies of scale (see: gigafactory). Also, the supply chain necessary to create the high-quality parts Tesla needs doesn't yet exist. So the company, sensibly, is doing almost everything in house. Much the same is true of Musk's other company, SpaceX. And Steve Jobs was legendarily insistent on vertical integration at Apple to ensure quality.

For most of its history, the utility industry has exhibited the two key features that justify vertical integration: economies of scale and high transaction costs. We mentioned the former earlier.

On transaction costs, remember that until quite recently, everything on the grid was electromechanical (not digital). The way utilities found out how much electricity a customer used was by hiring someone to go to the customer's building, observe the little spinning wheel on the meter, and write down the numbers. The way utilities found out that power lines were down, or houses were without power, was by customers calling on the telephone to tell them.

That is, sadly, still the way utilities find out stuff in many places. In that circumstance, it's obviously impossible to share real-time information with a third party, and risky to contract with one. Without digital monitoring and automation, communicating the relevant information in the relevant timeframe is too laborious to be worth it.

The benefit of a vertically integrated monopoly electricity service is reliability. Since there's only one entity responsible and no competitive pressure to cut costs (in fact, the opposite: there's pressure from shareholders to spend more), utilities are free to solve the reliability problem by overbuilding capacity. Contracting for reliability services outside the utility is not only risky, it represents potentially lower returns.

The opportunity cost of vertically integrated monopoly — what you forgo when you choose that regulatory structure — is innovation. Utilities are large organizations with enormous sunk costs and years of bureaucratic inertia, overwhelmingly focused on reliability, with returns protected by law and every move watched by regulators. That is not a recipe for entrepreneurial spirit.

Given economies of scale and high transaction costs, which held for most of the industry's history, this trade-off made sense. But technological changes can reduce both, and when they do so, the logic of vertically integrated monopoly breaks down and it begins to make sense to "unbundle" some of the integrated services. And that's just what happened in the late 20th century.

In the early 1990s, ratepayers were still paying off the high capital costs of huge, old coal and nuclear plants even as smaller, nimbler natural gas plants started popping up and offering cheaper power, while higher voltage power lines stretched farther and gave customers more choices.

The demise of "bigger is always better" led, starting in the mid-1990s, to a wave of "restructuring," whereby power generation was unbundled from other electricity services and turned over to a competitive market. Independent system operators (ISOs) and regional transmission organizations (RTOs) were set up to manage these wholesale power markets, reduce the transaction costs of communicating the needs of distribution utilities to generation companies, and maintain the long-distance transmission grid.

Restructuring raced through about 20 states and then, in the early 2000s, Enron happened in California and the process came to a screeching halt, where it remains frozen today.

While wholesale competition made it to 20 states, retail-side competition made it to almost none, and only incompletely. The distribution grid and the services involved in it, generally considered still a natural monopoly, were left to regulated distribution utilities.

In electricity, the benefits of competition now exceed the benefits of monopoly

Utility regulations haven't changed much in the past decade. But technology has. On the generation side, the pressures that began facing big baseload coal and nuclear plants in the 1990s have only grown more intense, as natural gas gets cheaper and is joined by wind and solar. Meanwhile, regulatory mandates at the state and federal level, sophisticated demand management, and larger economic shifts have meant that ever-expanding demand — a key premise of the early regulatory model — has finally plateaued and even begun to fall. There is no longer any need for a stampede of new power plants. Economies of scale are no longer the overriding influence in electricity markets.

But perhaps most importantly, there has been a remarkable surge of innovation at the "distribution edge," which includes the interface between the grid and the customer and everything "behind the meter," where customers manage and use electricity.

Several developments have come together to yield these innovations. There's the falling cost of batteries and the rise of plug-in electric vehicles, which have expanded options for home energy storage. There's the extraordinary decline in the cost of distributed solar power on residential and commercial rooftops, which has expanded the market for home energy generation.

And above all, there's the dizzying expansion of information and communications technology (ICT), which has the potential to revolutionize every aspect of the electricity supply chain. The signal effect of ICT is to lower transaction costs — to make it much easier to communicate, coordinate, and automate grid interactions. It enables a larger and more heterogeneous group of market participants, all the way down to "smart appliances" that shift demand based on real-time price signals.

The 20th-century "hub and spoke" grid was built to be big, dumb, and one-way only, producing electricity at central-station power plants and spilling it through lines into consumer homes and businesses. The 21st-century grid (if we ever get there) is going to be:

modular: the grid will be composed at least in part of smaller microgrids that can be "islanded" off from the larger grid in case of service problems, providing their own power for a limited time

the grid will be composed at least in part of smaller microgrids that can be "islanded" off from the larger grid in case of service problems, providing their own power for a limited time smart: sensors and ICT technology will enable realtime information about distributed generation, consumption, grid congestion, and possible threats to reliability

sensors and ICT technology will enable realtime information about distributed generation, consumption, grid congestion, and possible threats to reliability multidirectional: rather than being passive recipients of electricity, thousands of consumers will also generate, store, and sell it, becoming participants in electricity markets — producer/consumers, or (gag) "prosumers"

That's a lot of buzzwords at once — I'll spell them out more fully in future posts — but the upshot is that electricity service is going to evolve from a pure commodity business (electrons, dumped into your home) with a limited number of participants to a bustling market that involves a wide range of differentiated products and services, offered by a wide range of participants of all sizes.

This will mean a couple things for you, the consu ... sorry, prosumer. You are going to participate in these markets, though most of that is likely to be automated, taken care of by your smart appliances and smart car and smart home energy management system.

And you are going to have more choices. You'll be able to choose exactly what trade-off of cost and reliability you want (even room by room). You'll be able to choose what level of carbon intensity you want or the geographic origin of your electricity. You'll be able to choose how much electricity you want held in reserve, in case of service disruptions, versus how much you want to make available on spot markets (this too can be automated). You'll find electricity services cross-packaged with other services like transportation. And in general, you'll be more resilient, better able to ride out storms, attacks, or other disruptions of grid service.

And who knows what else markets may come up with when the grid is open to all, like the internet. The ground is ripe for all kinds of innovations we can't now predict. But here's the important thing, the nut graph, so I'm bolding it:

In electricity, economies of scale no longer hold. Barriers to entry are no longer substantial (except regulatory barriers). Transaction costs are no longer high. There is no longer any justification for vertical integration or "bundling" of electricity services, even at the distribution level.

The traditional trade-off — high reliability for low innovation — no longer makes sense. Thanks to ICT, reliability can be maintained by smart coordination of third parties rather than overbuilding by a single party. And what's needed now more than ever, not only for the consumer's benefit but to achieve social goals like decarbonization, is innovation. And the best tool we know of to achieve rapid innovation is a well-structured, well-regulated, competitive market.

In other words, recent technological changes argue for finishing the work of electricity unbundling, not only at the generation level but at the distribution level as well. It's time to open electricity up and get it moving again.

What might the utility of the future look like, if regulators took the imperative to unbundle seriously? We'll grapple with that thorny question in my next post.