An American B-52 bomber on a routine mission over the South China Sea unintentionally flew within two nautical miles of an artificial island built by China, senior defense officials said, exacerbating a hotly divisive issue for Washington and Beijing.

Pentagon officials told The Wall Street Journal they are investigating why one of two B-52s on the mission last week flew closer than planned to Cuarteron Reef in the Spratly Islands, an area where China and its neighbors have competing territorial claims. A senior U.S. defense...