Scammers recently found a new workaround for one of the advertising industry’s most significant efforts to protect marketers from being tricked into buying ads on fake websites.

The industry initiative, a nearly two-year-old project dubbed Ads.txt, lets a publisher display a simple text file on its site listing every company authorized to sell its ads. Buyers being offered ad inventory can check the publisher’s site to see whether the seller is listed.

But the world of online ad fraud is an almost never-ending game of Whac-A-Mole between the industry and the bad actors looking to siphon off marketers’ ad dollars.

Late last year, DoubleVerify Inc., a company that offers software for advertisers and ad vendors to authenticate ad inventory, identified a scheme it says was designed to take advantage of the growing adoption of Ads.txt.

DoubleVerify estimated the scam could have taken between $70 million and $80 million of advertisers’ spending a year had it gone unchecked.

First, the fraudsters scraped content from legitimate sites to create copies. Then they deployed “botnets” of consumer devices infected with malware to generate fake page views on the mock sites.

Usually, this is where Ads.txt could go some way to preventing fraud: Buyers offered the resulting ad impressions could check the legitimate sites’ Ads.txt files to see whether the impressions come from authorized vendors.

But in this scheme, the fraudsters opened accounts with vendors listed as approved “resellers” in publishers’ Ads.txt files. Resellers don’t have direct relationships with publishers to sell the specifically listed portion of inventory in the file, instead buying it from intermediaries and selling it onward. The fraudsters then sold their spoofed inventory through the publishers’ authorized resellers, knowing buyers who checked the publishers’ Ads.txt files would find the resellers there.

Heading off the fraud would have required the authorized resellers to scrutinize the fraudsters’ faked inventory more closely.

“The further down the long tail of resellers, aggregators and ad networks you go, the looser the processes are” when it comes to vetting, said Roy Rosenfeld, head of DoubleVerify’s fraud lab.

DoubleVerify said it immediately notified clients and partners affected by the scam and set up new processes to protect against its technique. The company declined to disclose the specific sites affected, but said of the bigger publishers impacted, “high-profile news sites” were disproportionately targeted, followed by entertainment sites.

Around 41% of the top 1,000 websites ranked by the analytics service Alexa Internet Inc. had implemented Ads.txt as of Jan. 30, according to FirstImpression.io, an ad-technology company. That is up from around 34% a year earlier.

Some vendors have previously attempted to talk their way into being listed on publishers’ Ads.txt files even though they didn’t have a relationship with them, ad industry trade publications have reported.

“Publishers have to understand that the only entries in the Ads.txt file can be entries that a publisher can verify they are getting money from,” said Neal Richter, chief technology officer of Rakuten Marketing and one of the authors of the Ads.txt standards. “The other issue is that there needs to be a process and a standard that every exchange adheres to.”

The fight against ad fraud remains a continuing battle, although some of the industry’s efforts to tackle the problem appear to be paying off. A study from ad fraud-detection firm White Ops Inc. and the Association of National Advertisers estimated advertisers wasted about $6.5 billion in 2017 on ads served to fraudulent traffic, a slight decline from the $7.2 billion estimated for 2016.

Last November, the Justice Department charged eight people with operating two alleged ad schemes involving scores of faked websites and infected computers across the world, costing advertisers tens of millions of dollars. Companies including Alphabet Inc.’s Google, Facebook Inc., Verizon Communications Inc.’s media subsidiary Oath Inc. and White Ops joined in a secretive effort to unravel the operations.

Industry experts also are hopeful about an Ads.txt upgrade called Ads.cert, which uses cryptographically stamped digital signatures to validate whether the source of an impression is what it claims to be and to give marketers a better view of what they are buying. Ads.cert is still in beta testing mode.

SpotX Inc., an ad-tech company that helps publishers sell video ads, confirmed it was contacted by DoubleVerify about the new Ads.txt fraud scheme in October and took immediate action.

“You need to have other processes in place to sniff out the funny business,” said Nick Frizzell, SpotX vice president of brand safety and inventory operations. That can include working with third-party verification firms and contacting suspicious vendors to ask for documentation to prove they have a relationship with the publisher for which they say they are selling ads.

“The recommendation we have is: Don’t just take Ads.txt data as the be-all and end-all in regards to what you should buy,” Mr. Frizzell said.

Write to Lara O’Reilly at lara.oreilly@wsj.com