A new type of nuclear reactor—smaller than a rail car and one tenth the cost of a big plant—is emerging as a contender to reshape the nation's resurgent nuclear power industry.

Three big utilities, Tennessee Valley Authority, First Energy Corp. and Oglethorpe Power Corp., on Wednesday signed an agreement with McDermott International Inc.'s Babcock & Wilcox subsidiary, committing to get the new reactor approved for commercial use in the U.S.

Although...