By now, many people are aware of the ongoing battles between customers with rooftop solar panels and power utilities around the country. At first blush, this can look like a standard story of scrappy outsiders versus evil corporations. But as I've been arguing, that's not quite right.

Utilities aren't evil. They are doing exactly what they are designed to do. The problem is the design. Right now, utilities operate in a regulatory environment that puts them intrinsically at odds with some of the coolest, most promising stuff happening in energy today: rooftop solar, energy storage in electric vehicles and household batteries, smart home energy management tech like the Nest thermostat, and various new ways of aggregating and managing demand.

These new technologies enable people to use less utility power. But utilities want people to use more utility power. So they fight the new technology. Until that fundamental conflict is resolved, utilities will be an impediment rather than a partner in the transition to a cleaner, smarter electricity grid.

States are beginning to understand this, and several are taking steps to reform how their utilities work. None, however, are going at it with the speed and gusto of New York.

Under the leadership of Gov. Andrew Cuomo, New York is undertaking an astonishingly comprehensive and ambitious effort to remake its energy systems and reduce its carbon emissions. One part of that broader effort is a program known as Reforming the Energy Vision (REV), which charges the state's public service commission (PSC) with developing a new vision for utilities and implementing it in the next few years. The goal is to realign the incentives facing utilities so that they can profit from, and benefit from accelerating, the spread of new clean, distributed energy technologies.

Reforming utilities is a perilous process, with intense politics and high stakes. Reliable service must be maintained throughout any transition — it's like rebuilding an airplane in flight.

But if New York succeeds, REV could spark a wave of utility reform, with salutary effects on US carbon emissions (and the success of President Obama's Clean Power Plan). If it blows up or falls apart, it could scare other states away from restructuring, just like Enron did in the early 2000s. It's arguably the most important clean energy policy initiative in the country today.

New York is trying to fix two major problems with utilities

Let's briefly review why utilities aren't working today. (A longer version of this argument can be found in this post, on why utilities are broken, and this one, on how to fix them.)

When utilities were first created, the entire area of electricity was seen as a "natural monopoly," with a single utility given exclusive purview over a region's power generation, long-distance transmission, and local distribution. This made sense because of economies of scale (bigger power plants and longer transmission lines made for cheaper power) and high transaction costs (it was a pain in the ass to contract with third-party providers for electricity services).

In the late 20th century, economies of scale began breaking down, mainly due to the introduction of small, efficient natural gas plants. In the 1990s and early 2000s, there was a wave of utility restructuring that split the power generation side off and handed it over to competitive markets. Deregulation made it to about 20 states (including New York) before the Enron fiasco froze it in place.

Today, economies of scale are breaking down further, and thanks to internet and communications technology, transaction costs have declined as well. The natural monopoly keeps shrinking. Just as markets took over wholesale power generation, it is now possible for third parties to provide many of the retail services, to both customers and the distribution grid, that were once the exclusive purview of utilities.

So one problem is determining the proper boundary of the natural monopoly, and how to open up the areas outside it to markets.

The other big problem is how utilities make money.

Right now, utility revenue comes from returns on capital investments in power lines, substations, and other infrastructure. Utilities make money by building stuff. Naturally, they like it when electricity demand rises — that justifies building more stuff. They don't like it when demand plateaus or falls. And they don't like anything that helps make more efficient use of existing infrastructure. Both those make it difficult to justify building more stuff.

That is obviously a perverse set of incentives in an age when demand is becoming a controllable ("dispatchable") resource and new sources and storage options are springing up at the distribution edge of the grid. The key to reform is aligning the interests of utilities with expanded use of DERs, customer choice, and social goals like reduced carbon emissions and energy intensity. Utilities somehow need to be able to make money off that stuff.

Why utility reform and grid modernization are particularly important in New York

The US grid is old and utilities are anachronistic everywhere, but the need for reform is particularly pressing in New York, for several reasons.

For one thing, the state's renewable portfolio standard (RPS) and its energy efficiency portfolio standard (EEPS) are both set to expire this year. The state needs new, better ways to encourage clean energy. That's one thing REV is intended to provide and why it is the subject of such intense interest from cleantech advocates and businesses in the state.

For another, while overall electricity demand in New York has been fairly steady for the past decade, the ratio of peak demand to average demand has been rising. In other words, baseline electricity demand has fallen somewhat (thanks to energy efficiency and the broader shift to a more service-based economy), but peak demand has risen (thanks to everyone having flat-screen TVs and air conditioners and turning them on at the same time). The same thing is happening across New England; the Energy Information Agency discusses it here.

This has two ill effects. First, it causes grid congestion during peak demand hours, which means utilities need to build more lines and substations, which means rates rise.

Second, it causes electricity generation to be wildly overbuilt. There have to be enough power plants to provide the maximum power needed during peak hours. That means lots of power plants are sitting idle lots of the time, in off-peak hours. Some only run about 40 hours a year. The average New York power fleet "capacity factor" (time spent running) is around 55 percent. It is expensive and inefficient.

Rather than build even more power plants and grid infrastructure, New York wants to attack the problem from the demand side. It wants to make its "load profile" smoother, less peaky. To do that, it needs to encourage energy efficiency, demand shifting, and distributed energy resources (DER)s — all of which, in the traditional regulatory model, reduce utility revenue.

Thus, REV.

New York has a new vision for its utilities

To run REV, Cuomo made a few key hires. As his "energy czar" — chairman of energy and finance for New York, in charge of the state's entire energy portfolio — he chose Richard Kauffman, a cleantech investor, senior adviser to Steven Chu at the Department of Energy, and a big believer in markets.

As chair of the PSC, he chose Audrey Zibelman, who's been consulting and testifying on electricity industry issues for 30 years. Notably, she was a driving force behind electricity restructuring in 1990s.

Kauffman and Zibelman came in with a clear understanding of the problems facing utilities. Their solution, in a nutshell, is to complete the work of restructuring. They want to create markets where third parties can compete to provide energy products and services on the retail side, things like energy storage, demand response, and distributed generation. They want the prices of those products and services to be set by the market, based on the real-time needs of the grid rather than by regulatory fiat.

And they want utilities — alongside their continuing core functions of resource planning, maintaining the grid, and providing reliable service — to help establish these markets and get them running smoothly.

It's a grand vision, but devilishly complex in practice, with all sorts of difficult questions to answer along the way. I won't get into nearly all of them, but it's worth looking at two key questions in a little more detail.

Problem No. 1: How far should a utility's monopoly extend?

Utility restructuring in the 1990s and early 2000s sprang from a recognition that power generation was no longer a natural monopoly. It could be served by competitive markets.

So states that restructured had to create — or, what is apparently the new term of art, "animate" — wholesale power generation markets.

The first thing policymakers did is take any ownership of power generation away from regulated distribution utilities, recognizing the obvious fact that they couldn't be fair, impartial purchasers of power if they were also in the business of generating it. Power generation could only be owned by unregulated subsidiaries.

The long-distance transmission system would be administered, and wholesale markets for power run, by independent, nonprofit organizations called independent system operators (ISOs).*

The key feature of this arrangement is that regulated utilities no longer have any stake in wholesale power markets or any ability to influence them. They don't own power generation and they don't oversee the market. They only care about purchasing reliable, low-cost electricity for their customers. (In theory.)

Responsibility for administering wholesale power markets was given to ISOs because their only incentive is to run the markets well. Order 888 from the Federal Energy Regulatory Commission (FERC), which established utility restructuring, said that the core purpose of ISOs was to "operate the transmission systems of public utilities in a manner that is independent of any business interest in sales or purchases of electric power by those utilities."

Today, for a whole variety of reasons, the retail side of electricity is no longer a natural monopoly either. Retail customers (businesses and homes) increasingly want choices about where their power comes from, its quality and reliability, and how it's stored or managed. Where there was once a one-way flow of power from big power plants into passive buildings, the grid is evolving into a multidirectional network, with every node a participant, generating power, storing it, and selling it. There's a whole array of technologies and services blooming along the edge of the grid, where the grid meets the customer power meter, and "behind the meter," inside homes and buildings. All those products and services are best provided by competitive markets.

The boundary of the natural monopoly has moved inward again. What remains of it is grid operations and reliability planning — that is, running the distribution grid itself. That's the only function that is still properly the purview of a publicly accountable, regulated utility.

So New York faces a question not unlike the one it faced some 20 years ago: how to animate new electricity markets, draw in third-party providers, and properly price new electricity products and services.

There are two broad options. If New York were to follow the same model used in wholesale restructuring, it could require regulated utilities to divest of all ownership or participation in retail markets and establish independent organizations — independent distribution system operators (IDSOs) — to oversee those markets. That is just the model advocated by many grid reformers, most notably former FERC Chair Jon Wellinghoff.

But there's a second possible way to restructure. Regulated New York utilities believe they should be given the market-administering role played by the ISOs in wholesale markets.

That would give utilities a hybrid role. They would be in charge of grid operations and reliability planning, per their natural monopoly. And they would also be in charge of animating and managing retail electricity markets, per the ISO function. In this hybrid model, utilities would be known as distributed service providers (DSPs).

The argument between the IDSO model and the DSP model is quite interesting, more so than the thicket of acronyms might indicate. But the wonky details are probably better saved for another post.

Suffice to say, the PSC found the utilities persuasive. In its Track One Order (which establishes the basic framework of reform; Track Two, which will pin down many details, is in process) it settled on the DSP model, under which the utility would plan and ensure reliability, run the grid, and administer the market.

The PSC did emphasize that DSP ownership of assets or participation in retail markets would be "the exception rather than the rule." A DSP can own DERs when it has tried to procure them but the market has not produced affordable choices. It can own some energy storage to improve grid functioning. And it can own DERs when they will benefit low-income residents who can't afford them, or for demonstration projects.

But for the most part, DSPs are not meant to participate in the markets they create. They will establish and enforce rules, provide reliable, open-access, real-time data, source (through procurement or rate design) services needed by the grid, coordinate and schedule resources, ensure cybersecurity, and possibly coordinate with wholesale markets via the New York ISO. (Much, much more can be found on all these responsibilities in REV's Market Design Platform Technology working group report.)

A great deal remains to be worked out. The details will become clearer when the Track Two Order is finalized (comments on the straw proposal are due today [UPDATE: the comment deadline has been pushed back; comments on track two are now due on the 26th]). What's clear is that lots and lots of this is uncharted territory.

Originally, state utilities were expected to come up with distributed system implementation plans (DSIPs), which would chart their transition to this new model, by the end of 2015. That always seemed crazy to me; the deadline has now been moved to the end of June 2016.

Problem No. 2: How will utilities make money?

Right now, utilities make money exclusively through returns on capital investments. That creates an obvious incentive to seek more capital investments. But today's grids and resources are capable of operating much more efficiently than they do now. Peaky loads can be met with demand management, efficiency, and DERs rather than new infrastructure.

To achieve that, however, utilities need to shift resources from capex (capital expenditures) to opex (operating expenditures). They need to shift from building to more intelligently managing, from machines to services.

Obviously that's hard to do if you only make revenue on capex!

So the PSC is trying to create new revenue streams for utilities, via performance-based regulation, or, in New York's specific argot, earnings impact mechanisms (EIMs).

The idea is that the utility will still be on the hook for its traditional responsibilities, like reliability. But there will also be performance-based incentives available for the achievement of other goals, like peak reduction. If the utility exceeds its performance goals, it can earn extra revenue as a reward.

So say a particular area of the grid is congested and suffering from high peaks. The utility faces a decision: Does it build a new substation, or does it procure demand-shifting services from a third party? Under the old model, building the substation was the only way it could make money. But if demand-shifting services help the utility meet its peak-reduction goal/EIM, under the new model it could receive a bonus for that. That gives the utility incentive to find new ways (other than building infrastructure) to reliably meet peak demand.

EIMs are conceived of as a transitional measure. Over time, as retail markets mature, regulators plan to phase them out. The idea is that the utility eventually won't need them, as it will have a variety of market-based earnings (MBEs) available, services it can make money providing to market participants.

This scheme raises some of the same objections from reformers as the DSP model, namely: If the services provided by the utility as MBEs can truly only be provided by regulated utilities, then they ought to be regulated services, with regulated prices, not market prices. And if they can be provided by third parties, then regulated utilities have no business providing them. There's a real danger of utilities shaping markets to favor the services that utilities provide.

So far, the EIMs and MBEs envisioned by the PSC are mostly related to animating markets, not competing in them. But they are not very clearly defined in the Track Two straw proposal. Reformers will be watching closely as the EIM/MBE model is filled in with details.

New York is walking a high wire, and everyone is watching

Across the country, lawmakers, regulators, utility executives, and grid geeks are closely tracking the progress of NY REV. It is not the only effort at utility reform underway, but it is the biggest and in many ways the cleanest.

To a remarkable extent, the process thus far has been free of the usual poisonous politics that surround utilities in other states. Cuomo's entire energy plan falls under his executive powers, so there's been no fractious fight with the legislature. Experienced wonks are running the relevant agencies. Utilities have seen which way the wind is blowing and are on board. A broad array of stakeholders has been involved from the very beginning (shout out here to the clean energy trade group Advanced Energy Economy, which has deftly coordinated many of these stakeholder meetings).

It would be naive to think the process is free of politics or the influence of incumbents. And it is sure to become more disputatious as it moves from generalities to specifics.

But this is about as close to clean-sheet-of-paper reform as utilities have seen in any US state. If it works, the effects could be seismic. As customers become more directly involved in energy, they will also become constituents for further reform. As New York utilities profit from grid-edge innovation, utilities in other states will see that the much-feared "death spiral" is not inevitable and that a prosperous future is possible. And as new markets, services, and jobs are created, other state lawmakers will see that reducing carbon emissions (and meeting EPA standards) can be an economic stimulus.

Or REV could all fail, in a thousand different ways, any of which could serve to dampen reform efforts elsewhere in the country. Utility reformers have been talking about opening retail electricity up to markets and innovation for a long, long time. Now, for good or ill, they are seeing their dreams put into action.

Further reading:

If you just can't get enough of REV and want to read more, here are some suggestions:

---

Correction: An earlier version of this article stated that ISOs own transmission lines. That's not right. They run the transmission systems in their territories, insuring reliability, but the utilities still own the actual lines.