Facebook’s tool meets only two of experts’ five minimum standards. That’s a failing grade.

Facebook pledged in February to release an ad archive API, in order to make political advertising on the platform more transparent. The company finally released this API in late March — and we’ve been doing a review to determine if it is up to snuff.

While we appreciate Facebook following through on its commitment to make the ad archive API public, its execution on the API leaves something to be desired. The European Commission also hinted at this last week in its analysis when it said that “further technical improvements” are necessary.

The fact is, the API doesn’t provide necessary data. And it is designed in ways that hinders the important work of researchers, who inform the public and policymakers about the nature and consequences of misinformation.

Last month, Mozilla and more than sixty researchers published five guidelines we hoped Facebook’s API would meet. Facebook’s API fails to meet three of these five guidelines. It’s too early to determine if it meets the two other guidelines. Below is our analysis:

[1] ❌

Researchers’ guideline: A functional, open API should have comprehensive political advertising content.

Facebook’s API: It’s impossible to determine if Facebook’s API is comprehensive, because it requires you to use keywords to search the database. It does not provide you with all ad data and allow you to filter it down using specific criteria or filters, the way nearly all other online databases do. And since you cannot download data in bulk and ads in the API are not given a unique identifier, Facebook makes it impossible to get a complete picture of all of the ads running on their platform (which is exactly the opposite of what they claim to be doing).

[2] ❌

Researchers’ guideline: A functional, open API should provide the content of the advertisement and information about targeting criteria.

Facebook’s API: The API provides no information on targeting criteria, so researchers have no way to tell the audience that advertisers are paying to reach. The API also doesn’t provide any engagement data (e.g., clicks, likes, and shares), which means researchers cannot see how users interacted with an ad. Targeting and engagement data is important because it lets researchers see what types of users an advertiser is trying to influence, and whether or not their attempts were successful.

[3]

Researchers’ guideline: A functional, open API should have up-to-date and historical data access.

Facebook’s API: Ad data will be available in the archive for seven years, which is actually pretty good. Because the API is new and still hasn’t been properly populated, we cannot yet assess whether it is up-to-date, whether bugs will be fixed, or whether Facebook will support long-term studies.

[4]

Researchers’ guideline: A functional, open API should be accessible to and shareable with the general public.

Facebook’s API: This data is now available as part of Facebook’s standard GraphAPI and governed by Facebook Developers Terms of Service. It is too early to determine what exact constraints this will create for public availability and disclosure of data.

[5] ❌

Researchers’ guideline: A functional, open API should empower, not limit, research and analysis.

Facebook’s API: The current API design puts huge constraints on researchers, rather than allowing them to discover what is really happening on the platform. The limitations in each of these categories, coupled with search rate limits, means it could take researchers months to evaluate ads in a certain region or on a certain topic.

It’s not too late for Facebook to fix its API. We hope they take action soon. And, we hope bodies like the European Commission carefully scrutinize the tool’s shortcomings.

Mozilla will also be conducting an analysis of Google’s ad API when it is released in the coming weeks. Since Facebook’s ad archive API fails to let researchers do their jobs ahead of the upcoming European Parliamentary elections, we hope that Google will step up and deliver an API that enables this important research.