Google is issuing refunds for ads that ran on websites with fake traffic, people familiar with the situation said, as the web giant develops a tool to give marketers more transparency about the ads they buy through its platform.

In the past few weeks, the Alphabet Inc. unit has informed hundreds of marketers and ad agency partners about the issue with invalid traffic, known in the industry as “ad fraud.” The ads were bought using the company’s DoubleClick Bid Manager over the course of a few months this year, primarily in the second quarter.

Google’s refunds amount to only a fraction of the cost of the ads served to invalid traffic, which has left some advertising executives unsatisfied, the people familiar with the situation said. Google has offered to reimburse its “platform fee,” which ad buyers said typically ranges from about 7% to 10% of their total purchase.

The company says this is appropriate because it doesn’t control the rest of the money spent. Typically, advertisers use DoubleClick Bid Manager to target audiences across vast numbers of websites in seconds by connecting to dozens of online ad exchanges, marketplaces that connect buyers and publishers through real-time auctions.

The ad spending flows through to the exchanges. The problems arise when ads run on publisher sites with fraudulent traffic, including those where clicks are generated by software programs known as “bots” instead of humans. This is an issue of growing concern to marketers. It is difficult to recoup the money paid to those sites when the issue is discovered too late.

Advertisers often receive small credits from Google and their other ad-tech vendors when they detect discrepancies, but in this case, for some buyers, the instance of fraud discovered was larger than usual.

It’s the latest evidence of how the complexity of the digital advertising ecosystem—an industry where marketers and ad sellers are separated by layers of middlemen and automation—can cause tensions between Madison Avenue and big players like Google. Just a few months ago, some marketers suspended their campaigns from Google’s YouTube after revelations their ads appeared next to hateful or otherwise unsavory videos. YouTube has taken steps to assuage marketers’ concerns, and many brands have now returned to the platform. Ad agencies, too, have battled with Google to let them access more of its extensive data to help them improve how ads are targeted and measure whether they are effective.

Scott Spencer, director of product management for Google, acknowledged that refunds have been paid, but he declined to provide a dollar figure for the amount being returned. Some ad buyers said the refund amounts range from “less money than you would spend on a sandwich” to hundreds of thousands of dollars.

Some agencies and advertisers were affected more than others, depending on their level of spending during the period and the types of ads they bought.

“Today, we can’t disclose the information about third parties,” Mr. Spencer said. “So when we aren’t able to catch invalid traffic before it impacts our advertisers and we’re unable to refund their media spend, it hurts us, even if we’re not responsible.”

Google is working on a fix it hopes will provide some clarity over which technology providers in the ad-buying chain are responsible for issuing refunds. It is also working on technology to ensure advertisers automatically receive a full credit back from Google and its partners if incidents occur.

The company said it is entering discussions with the 100-plus exchanges, ad networks and publishers that DoubleClick Bid Manager plugs into, asking them to display to ad buyers whether they are willing to refund the entire media spend if ad-fraud instances occur. Buyers could then opt to filter out the sources of inventory that don’t have such a policy.

Mr. Spencer said Google expects “high rates of adoption” among exchanges, and that the ones it had spoken to so far had been “very supportive” of the effort.

Another point of contention among those receiving refunds is that they haven’t been given details about where their ads ended up or specific details about the exploits the fraudsters used, so that advertisers and agencies can apply their own safeguards in the future.

“We need to be very careful about commenting on or discussing specifics about bots or our detection,” Mr. Spencer said. “Often fraudsters will change their approaches and strategies based on our public comments.”

Of the billions of dollars of online advertising each year, a percentage is inadvertently placed on sites with fake traffic, with fraudsters siphoning off advertisers’ money.

The industry’s efforts to rein in fraud appear to have an impact. Some $6.5 billion in ad spending will be wasted this year to fraud, down 10% from 2016, according to a report released in May by the Association of National Advertisers and ad-fraud detection firm WhiteOps.

The methods the fraudsters use are highly sophisticated. Some infect unsuspecting consumers’ computers with malware to form a “botnet” that clicks on ads on bogus sites.

Fraudsters are often adept at covering their tracks, which can make their activity difficult to spot until after the event has occurred.

For years, Google has had teams dedicated to filtering out fraud before an advertiser makes a bid on an ad. Those teams can also prevent exchanges from being paid if an ad has already been bid on but invalid traffic is quickly detected. The teams also work to discover historical instances of fraud, which is what happened in this particular case.

In the recent cases Google discovered, the affected traffic involved video ads, which carry higher ad rates than typical display ads and are therefore an attractive target for fraudsters.

Google has also joined a number of industry initiatives, such as the “Ads.txt” project launched in May by the Interactive Advertising Bureau, an industry trade body. The tool lets premium publishers insert a text file on their web servers to list all the ad tech vendors authorized to sell their inventory so ad buyers can confirm which platforms are selling legitimate ads.

“When people talk about [ad fraud], there’s a big specter to it and a big concern about invalid traffic in digital,” said Mr. Spencer. “It’s not that large in terms of a percentage of what people are buying, but it can be a little bit scary to buyers, and our goal is to remove that to improve the trust overall in the ecosystem.”

—Alexandra Bruell contributed to this article

Write to Lara O’Reilly at lara.o'reilly@wsj.com