As I noted in two previous posts, wind and solar power pose unique challenges to electricity grids, and those challenges put economic constraints on the growth of renewables.

For the most part, wind and solar — also known as variable renewable energy (VRE) — haven't run into those constraints yet, because their penetration remains relatively low. In 2014, wind generated just 4 percent of US electricity; solar produced less than 1 percent. But if wind and solar ever hope to supply 30 percent, 50 percent, or even 100 percent of electricity, they'll have to address the obstacles posed by current grids.

Those obstacles are varied — technologies, policies, and markets — but the cheapest, most ready-to-hand solutions are found in the rules that govern current grids, the operations, regulations, and markets that determine what resources are currently deployed to meet electricity demand, and when.

Before wind and solar, grids were fairly predictable

Let's review how power grids have worked for the past century or so, before VRE (wind and solar) came along:

Remember that electricity is a unique commodity in that it cannot be economically stored in large quantities. Excess generation not only goes to waste, it can destabilize a grid. Supply must always precisely match demand.

Until very recently, grid operators had a good handle on how to do that. There's a certain minimal level of electricity demand that persists around the clock. For that "baseload" demand, grid operators run big, lumbering coal and nuclear power plants, the kind that are very slow (and expensive) to shut down and restart. These plants run all the time, basically, except when closed for maintenance or unexpected faults.

When demand rises above that baseload level, grid operators fire up "mid-merit" plants, which are somewhat more agile, able to start up and shut down more quickly. These are usually natural gas combined-cycle plants.

And then when demand peaks, usually in the afternoon when the AC turns on and sometimes again in the evening when people get off work, grid operators fire up "peaker" plants, usually natural gas open-cycle turbine plants (and sometimes oil or diesel generators).

Demand can usually be forecast in advance, because people's daily habits are pretty predictable, and grid operators keep some just-in-case power plants — "operating reserves" — around for security and reliability.

It all worked quite nicely ... until wind and solar came along.

Wind and solar have made grid management more complicated

Now throw VRE into the mix. Grid operators cannot turn wind and solar plants up and down at will — these power sources are not "dispatchable." Instead, VRE generators push energy onto the grid whenever the wind is blowing or the sun is shining and don't when they aren't. Grid operators can't control wind and solar; they must accommodate them.

From a grid operator's perspective, a surge of VRE is akin to a dip in demand — it means less electricity is needed from all the other (dispatchable) power plants. An operator no longer uses dispatchable plants to meet total demand; she uses them to meet total demand minus whatever VRE is in the system.

Total demand minus VRE is known as "net load." Net load is the new benchmark for grid operators, the (moving) target they have to hit, minute by minute, hour by hour.

Here's the thing: Especially as rates of VRE penetration get higher, net load behaves quite differently than total demand. Overall electricity use rises and falls fairly gently, within a predictable range. Net load (overall use minus VRE) is more difficult to predict, because solar and wind can come and go erratically. It can rise or plunge quickly and swing over an enormous range, potentially falling almost to zero at times with high VRE penetration.

Following net load is an enormous challenge for grid operators. Above all, it places a premium on flexibility, on dispatchable resources that can scale up and down quickly to compensate for rising and falling VRE — at the lowest cost, with a high degree of reliability and security.

That's not what most of today's grids have. They are often burdened with an excess of inflexible baseload plants, operating under rules that encourage their continued use and within markets that fail to properly price flexibility.

Transitioning from today's grids to more nimble, responsive grids with high levels of VRE will require new policies from lawmakers and new technologies from innovators.

But the easiest, cheapest, and fastest way to boost flexibility in grids is to change the rules and markets that now govern them. This can be a complicated affair in practice, as the US has a variety of grids governed by a variety of regulatory and institutional structures. (Most notably, about half the country has established competitive wholesale electric markets, while the other half is supplied by old-fashioned vertically integrated monopoly utilities.) So details of these solutions will differ depending on circumstance, but I'm going to try to keep it general enough for those distinctions not to matter too much.

There are three changes, broadly speaking, that need to be accelerated in all electrical grids: resource flexibility, operational flexibility, and integration.

Grids need a more flexible portfolio of resources

"Resources" here refers to anything grid operators can use to ensure that supply meets demand — not only power plants, but programs that manage demand as well. The overall need is for resources that can more nimbly scale up and down in response to fluctuations in VRE. That will mean a couple of things:

1) Fewer baseload thermal plants, more mid-merit plants

On the dispatchable power plant side, that's going to mean fewer big, rigid baseload (coal and nuclear) plants and more mid-merit and peaker plants.

To understand why, consider the following scenarios drawn from an International Energy Agency (IEA) study of renewables integration. (I found it, and much else, through this excellent paper by Michael Hogan of the Regulatory Assistance Project.)

As a test case, IEA envisions a power system with 45 percent VRE. In the top scenario in the graphic, the rest of the resource base stays roughly the same as it is in most grids today: lots of baseload plants, a few mid-merits, and fewer peakers. In the bottom "transformation" scenario, investment incentives are shifted to encourage more flexible mid-merit generators and peakers.

In the first scenario, you've got a bunch of baseload plants that used to run all the time now running much less (their "capacity factor" has dropped from 90 percent to about 62 percent). And you've got a bunch of mid-merit plants that averaged 40 percent capacity factors now down to 11 percent. All those big plants won't run often enough to create a decent rate of return for investors, so they won't be able to remain open without subsidy. This is the nightmare scenario for utilities — being forced to subsidize a bunch of underutilized fossil fuel plants just to back up VRE.

In the second scenario, you've got way fewer baseload plants, but those that remain are running almost all the time. Their capacity factor stays at 90 percent, and the capacity factor of the (now more numerous) mid-merit plants stays at 40, so they are more attractive investments.

Here's the beauty part: Overall, the latter portfolio requires 40 percent less investment than the former. In other words, writes Hogan, "a more flexible mix of dispatchable resources, capable of shifting operations up and down in synch with the less controllable shifts in variable renewable production, will have far higher asset utilisation rates and require far less redundancy (and therefore far less investment) than a less flexible mix of thermal resources."

The transition to a high-VRE grid can be done more cheaply if there's a shift to more flexible generation resources. So how can grid regulators and operators encourage that shift?

For vertically integrated utilities, it's just a matter of smart central planning. For areas with wholesale energy markets, it means making the value of flexibility more visible in markets, to create incentives for investment in flexible resources. That might mean removing price caps in energy markets. It might mean tweaking capacity markets to better value "ancillary services" (e.g., speed and responsiveness). It might mean creating other, parallel forward markets for balancing or time-shifting services. Either way, the value of flexibility should be explicitly compensated.

Hogan adds (in a separate paper) that regulators should also push to allow as many participants as possible in energy and ancillary-services markets, and patrol vigilantly for undue concentration of market power; the more competition, the better, even in the restrictive confines of a vertically integrated utility area.

2) Better demand management

It used to be that regulators and grid operators had virtually no control over electricity demand. Their only job was to meet it with supply.

These days, however, there's a growing array of tools available to manage demand. To some extent, demand itself has become dispatchable; it can be shifted back and forth to respond to fluctuations in VRE.

For instance, a utility can pay a group of big industrial users to shift their operations away from times of high demand. Or a "demand-response aggregator" can sign agreements with dozens or hundreds of homeowners to automatically ramp down their water heaters by two or three degrees in times of grid congestion. Or, when electric cars become more common, utilities could agree to draw electricity from hundreds or thousands of car batteries when power is scarce, or store it in those batteries when VRE is plentiful.

When it is made dispatchable, demand response is cheaper and more nimble than virtually any supply-side option. In their efforts to increase grid flexibility, it's important that regulators and markets treat demand-side resources as equal to supply-side resources. Vertically integrated utilities should allow demand-response aggregators to compete with power plants for investment. In restructured areas, demand response should be allowed to participate in energy, capacity, or services markets alongside supply options.

3) More energy storage

Energy storage is a crucial way to "spread out" the surges in energy that come from VRE. And as times of wind or sunshine become more crowded with VRE, the ability to spread it out will become more valuable.

Storage can make money on energy-only markets through simple arbitrage, buying VRE when it's cheap and selling it back to the grid when it's more valuable. But regulators ought to structure capacity and services markets so that storage can participate in those, as well.

There's a lot to say about the potential (and limitations) of storage to enable more VRE, but I'll save it for a separate post. At the very least, regulators should structure rules and markets to encourage investment in storage.

Grids need more flexible operating rules

The procedural details of grid operation are not of keen interest to most people (ahem), so let's just briefly touch on a few key changes grid regulators and operators might consider.

First, scheduling and dispatch can be made more agile. Most grids schedule plant dispatch (which plants will produce energy, and how much) in hourly increments. It is possible to shift to faster, sub-hourly increments to increase flexibility, as some grids are now in the process of doing.

Second, weather forecasting can be radically improved with existing technology, making the level of VRE (and thus net load) much more predictable, not only a day ahead but even on an hourly or minute-by-minute basis. Better weather forecasts means grid operators need less redundant backup.

Third, locational pricing — which varies the price of energy based on where it is produced, reflecting geographical differences in demand and grid congestion — can be expanded and improved.

And finally, as mentioned above, markets can be tweaked to more accurately value services beyond energy.

Grids need to be integrated over larger areas

One way to smooth out the spiky, swingy behavior of net load is to integrate grids over a larger geographic area. As VRE is drawn from a larger and larger area, supply becomes steadier, fluctuations become less sharp, and prediction becomes more tractable.

In many cases, this will mean building new transmission lines to areas of intense sun or wind. That's a capital-intensive (and politically fraught) undertaking. Nobody likes new power lines going over their property and it's difficult to agree how to divide the costs.

However, there's plenty of grid integration possible without new infrastructure. Rather than grids themselves being expanded, areas of grid control ("balancing areas") can be expanded. Many of the benefits of geographical integration can be achieved by connecting balancing areas together under a single planning authority.

This has been done in the US, to some extent, through the creation of ISOs, which govern grid regions spanning multiple states. But even ISO regions could cooperate better among themselves.

In circumstances where balancing regions cannot be consolidated under a single authority — say, in areas spanning multiple vertically integrated monopoly utilities — there are still ways to achieve the same effects, by setting up structured exchanges of grid services between and among separate balancing authorities.

The benefits of consolidating balancing regions, says Hogan, include "better use of existing transmission infrastructure, less supply variability, less demand volatility, real-time access to more operating and contingency reserves, less need for backup generation capacity, more use of renewables, and more liquidity and less price volatility in the market due to more competition."

Grids are nowhere near physical limitations on VRE

Pondering the ultimate limitations of VRE is an interesting theoretical exercise, but the fact is, very few grids are anywhere near those limitations; for the most part, today's limitations are practical and solvable. There are simple, ready-to-hand changes in grid rules and markets that can unlock additional wind and solar power, if regulators and grid operators seize the opportunities.

Grid flexibility not only enables greater VRE, it also lowers costs by preventing overinvestment in fixed, inflexible resources, saving customers money in the long run. A smarter, nimbler, more responsive grid would be necessary even if wind and solar were not needed for social and environmental reasons. Given the imperatives of climate change, however, there's no longer any time to waste.