A top challenge for Rex Tillerson as Donald Trump’s secretary of state would be navigating the U.S.’s sensitive relationship with Beijing. On at least one flashpoint, the South China Sea, he has some familiarity.

Under Mr. Tillerson’s leadership, Exxon Mobil Corp. waded into one of the region’s thorniest diplomatic disputes when it signed a deal with Vietnam seven years ago to drill for natural gas, partly in waters also claimed by China, drawing Beijing’s ire and garnering attention from U.S. officials.

Some...