Journalistic fact-checking is typically described as the objective search for indisputable “truth.” Underpinning that definition is the belief that every argument can be definitively labeled “true” or “false,” without bias or personal views playing a role in that determination. In turn, companies like Facebook can use these ratings to de facto censor stories with very little backlash. In reality, fact-checking involves myriad tradeoffs that can have a huge impact on the free flow of information as fact checkers increasingly influence what we are permitted to see online. Two recent fact checks illustrate how evidentiary standards and literalism can have a significant impact on verdicts.

Last week Snopes reviewed a viral meme featuring a White House photograph of President Trump with a group of individuals, many of whom had red X’s over their faces. The image was captioned “This photo was taken right after the GOP voted to repeal the Afoordable [sic] Care Act last year. Every one [sic] with an X has since been voted out of Congress.”

Noting that many of the individuals with X’s over their faces actually won re-election, never served in Congress, or had left Capitol Hill for other jobs, the meme was rapidly dismissed by political journalists and commentators as “quite literally insane fake news.” Yet, Snopes issued a verdict that the meme was “True,” which it defines as “the primary elements of a claim are demonstrably true.” In turn, Snopes’ rating meant that the meme was not blacklisted by Facebook and instead free to continue going viral.

Why would Snopes assign such a verdict to a meme that the rest of the journalistic class had rejected as demonstrably and overwhelmingly false?

In the narrative describing its assessment, the fact-checking organization conceded its inaccuracy, but with an elastic caveat: “Although memes are frequently grossly inaccurate, this one got the general idea and numbers correct (even if the persons actually pictured in the accompanying photograph are difficult or impossible to identify).” In a subsequent update, it added that “some commenters who went through the photograph used in the meme in microscopic detail to try to discern the identities behind the tiny faces obscured with red X’s noted that they didn’t all correspond to the (unnamed) members of Congress whose seats were lost after they voted to repeal the ACA.”

In fact, when one looks more closely at the top of the fact check, Snopes achieved its “True” rating by recasting the meme from its original wording into an argument that “the Congressional seats of almost three dozen Republicans who voted to repeal Obamacare were lost to Democrats in 2018.”

While the original meme was explicit that the specific individuals with X’s across their faces were voted out of Congress for their ACA repeal votes, Snopes changed the meme’s meaning to an interpretation that it could argue was correct. Of course, once one moves away from the literal argument of a meme, one could easily assert that the meme is vastly misleading because a much larger cohort of Republicans won re-election after voting to repeal Obamacare.

Snopes’ co-founder David Mikkelson justified the site’s subjective restatement of the meme by offering that “our audience is intelligent enough to understand the difference between a literal representation and a symbolic one.”

When asked whether Facebook requires that its fact-checking partners treat claims literally, a spokesperson offered only that “we provide public guidance to fact-checkers as to what kind of content should garner which rating (true, false, mixed, etc)—but we ultimately defer to the judgement of third-party fact-checkers.”

In other words, Facebook places blind trust in the decisions of fact-checkers without holding them accountable to any kind of centralized policies regarding how to interpret claims or evidence. Given that Snopes is an International Fact-Checking Network signatory, I asked that organization’s director, Alexios Mantzarlis, whether the IFCN believes fact-checkers should treat claims literally.

While noting that the group does not comment on specific fact checks, he offered that “the question of literalism versus contextualism is a central challenge of fact-checking.” Mantzarlis approvingly cited a 2016 book addressing this topic written by University of Wisconsin journalism professor Lucas Graves. He also referred to a piece he wrote on the subject earlier this year when the issue arose in the form of a spat between The Weekly Standard and ThinkProgress. “My preference as a fact-checking editor was to stick to literalism while providing context,” he told me. “But the current U.S. political context makes that a little harder.”

Asked to amplify on the point about the Trump-era “political context,” Mantzarlis cited the following passage from his article:

The tension between literalism and contextualism is one fact-checkers face every day. … [S]wing too far towards literalism and you’re a bone-headed bean counter. Get too contextual and you’re taking wild leaps of interpretation — something fact-checking was explicitly set up to avoid. Donald Trump's peculiar communication style is extremely well-suited to triggering this inherent limitation of fact-checking.

We spent an entire news cycle two years ago discussing [the] argument that journalists were taking him “literally but not seriously,” while his supporters were doing the opposite. He didn’t actually mean that unemployment was 42 percent, Barack Obama was born in Kenya and climate change was a hoax perpetrated by the Chinese. Wake up, sheeple! Read between the lines! This interpretative license can be heavily colored by personal biases. In a message, Graves told me that “we're very selective in how we use nuance. It's easy to be forgiving of imprecision in arguments we agree with, to argue that they should be evaluated in context.” “Meanwhile, we're usually slavish literalists when it comes to ideas we reject.”

I asked Mantzarlis whether the IFCN itself had formal written guidance on the subject. “We do not,” he said. “We expect that these decisions fall within the normal remit of editorial decisions taken by fact-checkers.”

In short, while offering that his personal preference is to stick to literal treatment of claims, Mantzarlis acknowledges the natural tension resulting from the tendency of politicians of all affiliations make statements never intended to be treated literally. As PolitiFact notes, “In the world of speechmaking and political rhetoric, there is license for hyperbole.”

There is also room, apparently, for purely subjective judgments.

Last week PolitiFact fact-checked the links between Hillary Clinton, the Clinton Foundation and the controversial Uranium One deal, once again finding the claim of a connection to be “Mostly False.” Yet, far from ironclad statements of fact, the piece is filled with such language as "the details remain murky," "anything that we could find," "it's somewhat unclear what the post means when it says," "one would assume he would have been aware," "claim makes it seem like," "might not have known about it," “we couldn’t independently verify” and “if he is telling the truth.” Even Wikipedia frowns upon the use of such vague language in its articles.

These phrases are not merely peripheral commentary. They form the core argument structure supporting PolitiFact’s verdict.

Fact-checking is frequently directed at the actions and claims of government officials, yet these are among the most difficult to definitively verify. Much of the underlying evidence that could support or refute claims is not publicly available and fact-checkers must resort to citing news coverage or third-party quotes reported in the media.

When asked about the use of such language in fact checks, Facebook did not comment beyond its original statement, while the IFCN’s Mantzarlis declined to comment further.

Putting this all together, fact checks that move beyond literal verification towards subjective reinterpretation or rely on tortured turns of phrase that litter their conclusions with ambiguity, undermine the public’s confidence in their neutrality. When fact checks restate concrete claims into straw man arguments or base conclusions on a foundation of fog, they become incredibly corrosive to public trust in the fact-checking enterprise. Most of all, they remind us of just how much subjectivity goes into fact-checking and that they are often more storytelling than science.