Shale companies’ strategy to supercharge oil and gas production by drilling thousands of new wells more closely together is turning out to be a bust. What’s more, the approach is hurting the performance of older existing wells, threatening the U.S. oil boom and forcing the maturing industry to rethink its future.

To maintain America’s status as an energy powerhouse, shale companies in recent years have touted bunching wells in close proximity, greatly increasing the number of wells drawing on a promising reservoir. The added...