Earlier this year, Ellen Williams, the director of ARPA-E, the U.S. Department of Energy’s advanced research program for alternative energy, made headlines when she told the Guardian newspaper that "We have reached some holy grails in batteries.”

Despite very promising results from the 75-odd energy-storage research projects that ARPA-E funds, however, the grail of compact, low-cost energy storage remains elusive.

A number of startups are closer to producing devices that are economical, safe, compact, and energy-dense enough to store energy at a cost of less than $100 a kilowatt-hour. Energy storage at that price would have a galvanic effect, overcoming the problem of powering a 24/7 grid with renewable energy that’s available only when the wind blows or the sun shines, and making electric vehicles lighter and less expensive.

Illustration by Federico Jordan

But those batteries are not being commercialized at anywhere near the pace needed to hasten the shift from fossil fuels to renewables. Even Tesla CEO Elon Musk, hardly one to underplay the promise of new technology, has been forced to admit that, for now, the electric-car maker is engaged in a gradual slog of enhancements to its existing lithium-ion batteries, not a big leap forward.

In fact, many researchers believe energy storage will have to take an entirely new chemistry and new physical form, beyond the lithium-ion batteries that over the last decade have shoved aside competing technologies in consumer electronics, electric vehicles, and grid-scale storage systems. In May the DOE held a symposium entitled “Beyond Lithium-Ion.” The fact that it was the ninth annual edition of the event underscored the technological challenges of making that step.

Qichao Hu, the founder of SolidEnergy Systems, has developed a lithium-metal battery (which has a metallic anode, rather than the graphite material used for the anode in traditional lithium-ion batteries) that offers dramatically improved energy density over today’s devices (see “Better Lithium Batteries to Get a Test Flight”). The decade-long process of developing the new system highlighted one of the main hurdles in battery advancement: “In terms of moving from an idea to a product,” says Hu, “it’s hard for batteries, because when you improve one aspect, you compromise other aspects.”

Added to this is the fact that energy storage research has a multiplicity problem: there are so many technologies, from foam batteries to flow batteries to exotic chemistries, that no one clear winner is attracting most of the funding and research activity.

According to a recent analysis of more than $4 billion in investments in energy storage by Lux Research, startups developing “next-generation” batteries—i.e., beyond lithium-ion—averaged just $40 million in funding over eight years. Tesla’s investment in its Gigafactory, which will produce lithium-ion batteries, will total around $5 billion. That huge investment gap is hard to overcome.

“It will cost you $500 million to set up a small manufacturing line and do all the minutiae of research you need to do to make the product,” says Gerd Ceder, a professor of materials science at the University of California, Berkeley, who heads a research group investigating novel battery chemistries. Automakers, he points out, may test new battery systems for years before making a purchase decision. It’s hard to invest $500 million in manufacturing when your company has $5 million in funding a year.

Even if new battery makers manage to bring novel technologies to market, they face a dangerous period of ramping up production and finding buyers. Both Leyden Energy and A123 Systems failed after developing promising new systems, as their cash needs climbed and demand failed to meet expectations. Two other startups, Seeo and Sakti3, were acquired before they reached mass production and significant revenues, for prices below what their early-stage investors probably expected.

Meanwhile, the Big Three battery producers, Samsung, LG, and Panasonic, are less interested in new chemistries and radical departures in battery technology than they are in gradual improvements to their existing products. And innovative battery startups face one major problem they don’t like to mention: lithium-ion batteries, first developed in the late 1970s, keep getting better.