Big ad buyers and marketers are upset with Facebook Inc. FB 2.66% after learning the tech giant vastly overestimated average viewing time for video ads on its platform for two years, according to people familiar with the situation.

Several weeks ago, Facebook disclosed in a post on its “Advertiser Help Center” that its metric for the average time users spent watching videos was artificially inflated because it was only factoring in video views of more than three seconds. The company said it was introducing a new metric to fix the problem.

Some ad agency executives who were also informed by Facebook about the change started digging deeper, prompting Facebook to give them a more detailed account, one of the people familiar with the situation said.

Ad buying agency Publicis PUBGY 1.10% Media was told by Facebook that the earlier counting method likely overestimated average time spent watching videos by between 60% and 80%, according to a late August letter Publicis Media sent to clients that was reviewed by The Wall Street Journal.

A spokeswoman for Publicis Media, a division of Publicis Groupe SA, referred calls to Facebook. Publicis was responsible for purchasing roughly $77 billion in ads on behalf of marketers around the world in 2015, according to estimates from research firm Recma.

GroupM, the ad buying unit of WPP PLC, also was notified of the discrepancy by Facebook, another person familiar with the matter said.

On Friday, Facebook apologized. “The metric should have reflected the total time spent watching a video divided by the total number of people who played the video. But it didn’t,” said David Fischer, vice president of business and marketing partnerships, in a Facebook post. “While this is only one of the many metrics marketers look at, we take any mistake seriously.”

Facebook had said in an earlier statement: “We recently discovered an error in the way we calculate one of our video metrics.” It added: “This error has been fixed, it did not impact billing, and we have notified our partners both through our product dashboards and via sales and publisher outreach. We also renamed the metric to make it clearer what we measure. This metric is one of many our partners use to assess their video campaigns.”

The news is an embarrassment for Facebook, which has been touting the rapid growth of video consumption across its platform in recent years.

Due to the miscalculated data, marketers may have misjudged the performance of video advertising they have purchased from Facebook over the past two years. It also may have impacted their decisions about how much to spend on Facebook video versus other video ad sellers such as Google’s YouTube, Twitter, and even TV networks.

Media companies and publishers are affected, too, since they’ve been given inaccurate data about the consumption of their video content across the social network. Many use that information to help determine the types of content they post.

For the past two years Facebook only counted video views of more than three seconds when calculating its “Average Duration of Video Viewed” metric. Video views of under three seconds were not factored in, thereby inflating the average. Facebook’s new metric, “Average Watch Time,” will reflect video views of any duration. That will replace the earlier metric.

In its note to clients, Publicis said the change was an attempt to obfuscate Facebook’s earlier miscalculations.

“In an effort to distance themselves from the incorrect metrics, Facebook is deprecating [the old metrics] and introducing ‘new’ metrics in September. Essentially, they’re coming up with new names for what they were meant to measure in the first place,” the memo said.

The miscounting could also fuel concerns among advertisers and media companies about the so-called “walled gardens” that companies including Facebook and Google are often described as operating. Both companies keep a tight grip on data, and only allow limited third-party tracking firms to plug into their systems.

Keith Weed, chief marketing officer of Unilever, said in an interview last year, tech companies that don’t let third parties measure their platforms is equivalent to “letting them mark their own homework.”

The Publicis note said, “This once again illuminates the absolute need to have 3rd party tagging and verification on Facebook’s platform. Two years of reporting inflated performance numbers is unacceptable.”

—Shalini Ramachandran contributed to this article.

Write to Suzanne Vranica at suzanne.vranica@wsj.com and Jack Marshall at Jack.Marshall@wsj.com