WASHINGTON—The U.S. Treasury Department and Washington's allies are ramping up efforts to hit Islamic State's finances, particularly focusing on steps to choke off its oil sales, its donations from the Persian Gulf and its extortion rackets.

But current and former U.S. officials said the campaign presents a unique challenge to Washington's counterterrorism agencies, as one of the first to target an organization that maintains a covert international support network while also functioning as a state through the control of large...