American oil companies—primed to reap the benefits of rising prices after years of wringing more from wells for less—are seeing profits erode in the face of rising costs.

Those operational challenges make balancing lofty growth objectives and demands for fiscal restraint increasingly difficult. If the companies continue to stumble, the result could be a higher cost of capital to finance the ongoing U.S. energy boom or a slower pace of growth.

Two-thirds...