Shane Huntley and his team have tracked Iranian hackers as they spread disinformation in the U.S., unmasked North Korea’s responsibility for a crippling global computer virus and probed Russians linked to the 2016 hack of the Democratic National Committee.

Mr. Huntley doesn’t work for the National Security Agency or another government spy shop. He heads Google’s in-house counterespionage group, the Threat Analysis Group, which has emerged as an important force in the battle against hackers and a leading example of tech giants...