Last month, the Poynter Institute, home of PolitiFact, published through its International Fact-Checking Network a list of what it deemed “unreliable news websites,” promoting it as a “blacklist … useful for advertisers that want to stop funding misinformation.” The database was hand-curated by IFCN from five previous lists (including those produced by PolitiFact itself as well as FactCheck.org and Snopes), a process by which sites such as AlterNet and Think Progress were removed while others such as Breitbart and Daily Caller were retained. Yet two days after publishing the list, Poynter retracted it, citing “weaknesses in the methodology” and “inconsistencies.”

This saga raises many troubling concerns about fact-checking and its future. But first, some history.

Blacklists have emerged over the last few years as the go-to solution for the growing crisis of “fake news.” Their simplicity, ease of construction and ready implementation by major tech companies have led to their widespread circulation by academics, companies and NGOs. Microsoft even began distributing one blacklist-powered plug-in earlier this year with its mobile browser.

Poynter was the latest to climb aboard this bandwagon by combining the following: FactCheck.org’s Misinformation Directory, the Fake News Codex, OpenSources, Politifact’s Fake News Almanac and Snopes’ Field Guide to Fake News Sites and Hoax Purveyors.

It then began “removing several sites whose stories, though highly politicized, were mostly not fake: alternet.org, cato.org, heritage.org, nationalreview.com, thedailybeast.com, theintercept.com, thinkprogress.org, and weeklystandard.com.” Similarly, sites that “had mostly false verdicts from fact-checking sites” were retained, including “addictinginfo.org, breitbart.com, dailycaller.com, dailykos.com, and judicialwatch.org.” Poynter “determined this by checking the veracity of their stories at PolitiFact and Snopes.”

Yet within 24 hours Poynter had removed two more sites, The Washington Examiner and FirstPost, stating that “after reviewing our methodology, we found that neither met the criteria for inclusion.”

The methodology section of the report offers little guidance on precisely how determinations were made, other than emphasizing that “our list includes only sites whose stories are demonstrably false -- not merely biased or partisan.” The report’s lead author offered on Twitter that “if anyone knows a site not in our index, biased left or right, that regularly fails fact checks by IFCN @factchecknet verified fact-checkers, use our form to let us know.”

Poynter did not respond to a request for further detail on its methodology, so it is unclear what the actual inclusion criteria were, such as how many articles had to have been deemed false before an outlet was considered to have “regularly fail[ed] fact checks.”

The Daily Caller quotes IFCN as clarifying that “the total number of complaints is less than 2% of the whole database.” Given that there were 515 blacklist entries, this would suggest that only 10 websites on the entire list were disputed. Poynter did not respond to a request to provide a list of those sites.

It seems outlets on the list did not fare much better at getting a response from Poynter. One notes that it took more than 16 hours of phone calls and emails demanding to be removed before a Poynter representative finally responded that “we have decided to remove the database entirely. … You have my sincere apology and a promise to do better next time.”

Given that Poynter actively called upon advertisers to defund websites on its list, it is remarkable that it was not more responsive in fast-tracking requests for removal by sites that disputed their inclusion.

This entire episode could not be more remarkable. Perhaps most importantly, it reminds us how subjective fact-checking really is. Beneath the veneer of its precision, the fact-checking enterprise relies heavily on opinion and interpretation. That a list by and from fact checkers would be so quickly retracted reminds us that the lists that increasingly control our online world are far from indisputable.

For its part, Poynter seems eager to make the whole story disappear. Two weeks after retracting its list, there is little evidence it ever existed. Beyond redirecting the original URL to a Letter From the Editor and a single mention buried in its newsletter, all traces of the list seem to have been scrubbed from its site. Even the report’s author profile that previously cited the report has now been wiped clean.

While PolitiFact frequently includes screen captures and preserved copies of debunked material, Poynter notably chose not to make an archived copy of its list available with an appropriate disclaimer so that researchers and news outlets could further investigate its methodology and results. Instead, we are limited to the few snapshots the Internet Archive managed to capture before Poynter deleted it.

Neither Poynter nor IFCN has posted any kind of postmortem explaining what went wrong or what they’ve learned from the experience. More to the point, they seem to be exempting themselves from the same scrutiny they apply to everyone else.

Yet what makes Poynter’s blacklist most disturbing of all is that for the brief 48 hours it was online, it called upon the advertisers of those 515 websites to permanently blacklist them, cutting off their revenue. This raises the question of how many advertisers heeded that call before Poynter retracted its list and if Poynter has plans to address the damage inflicted. Again, the organization did not respond when asked about this.

Putting this all together, from their early roots debunking urban myths and highlighting political hyperbole, fact-checking organizations have steadily grown to the point that they now wield immense control over what we are allowed to see and say on social media platforms. Poynter’s blacklist represented a troubling new expansion of this practice, one in which fact checkers were calling upon funders to directly cut ties with sites.

This episode also represents a tacit acknowledgement that there is simply too much false information out there for today’s current game of fact-checking Whack a Mole and that new approaches are needed.

If a list summarizing fact-checking results and verified by fact checkers is ultimately retracted by those same fact checkers for not being rigorous, it underscores the question of why we should trust anything from the fact-checking community.