WASHINGTON—The Trump administration is locked in a diplomatic quandary as it tries to pressure North Korea: How to sanction Pyongyang’s top ally and patron—China—without alienating an economic powerhouse that is arguably the best hope to slow its rogue nuclear-weapons program.

That predicament helps explain why the U.S has targeted only one small Chinese bank with sanctions so far, even though American officials and analysts believe financial institutions in China represent key nodes in Pyongyang’s illicit finance network.

...