Obama/Biden spew less malarkey than Romney/Ryan overall and during debates.



That's why, earlier this week, to the world (well...more like only 500 people in the world). I statistically analyzed fact checker report cards from Truth-O-Meter and The Fact Checker to compare the factuality of the 2012 presidential and vice presidential candidates overall, and during the debates that had happened so far. I promised I'd get back to you on the third debate, and with a summary of how the two parties did in the debates overall, compared to one another, and to their usual selves.



Unless Bidama and Rymney (or is it Obiden and Romyan?) blast each other in the next few weeks as much as they have in the last year, this is probably my final 2012 election malarkey analysis until Election Day. This is the one of the most comprehensive, sophisticated, and detailed analyses of the 2012 presidential candidates' factuality.



Share it with your friends. Discuss its results. Debate its merits. Tell me precisely why you think I'm full of shit. Supersize the histograms and past them to brick walls like you're Shepard Fairey. Because this stuff matters . It matters because the facts matter. It matters because we should understand how confident we can be in our judgments about people.



Enough histrionics. Let's get to the science. If you've never been here before, quickly skim



Election Day fast approaches. A few remain undecided. The majority have decided, and look for any last bits of manure to fling at the other side. In the era of fact checking websites, a candidate's factuality has become important to a lot of people, including me.That's why, earlier this week, to the world (well...more like only 500 people in the world). I statistically analyzed fact checker report cards from Truth-O-Meter and The Fact Checker to compare the factuality of the 2012 presidential and vice presidential candidates overall, and during the debates that had happened so far. I promised I'd get back to you on the third debate, and with a summary of how the two parties did in the debates overall, compared to one another, and to their usual selves.Unless Bidama and Rymney (or is it Obiden and Romyan?) blast each other in the next few weeks as much as they have in the last year, this is probably my final 2012 election malarkey analysis until Election Day. This is the one of the most comprehensive, sophisticated, and detailed analyses of the 2012 presidential candidates' factuality.Share it with your friends. Discuss its results. Debate its merits. Tell me precisely why you think I'm full of shit. Supersize the histograms and past them to brick walls like you're Shepard Fairey. Because this stuff matters . It matters because the facts matter. It matters because we should understand how confident we can be in our judgments about people.Enough histrionics. Let's get to the science. If you've never been here before, quickly skim how I calculate the malarkey score and how I do my statistical comparisons before continuing. If you read my last 2012 presidential campaign update, not much has changed. So you might want to scroll down to my analysis of the third debate, and the debates as a whole. Full report cards

I collected the full Truth-O-Meter and Fact Checker report cards for Obama, Biden, Romney, and Ryan this morning. Let's start with what we observe. That is, what can we say about the factuality of the two sides if we take the report cards at face value? Here are the revised overall malarkey scores for each individual candidate and each ticket.



Individuals candidate malarkey Obama 48 Biden 53 Romney 57 Ryan 52 Tickets collated

(average malarkey in statements) ticket malarkey Obama/Biden 46 Romney/Ryan 56 average

(average malarkey of individuals) ticket malarkey Obama/Biden 48 Romney/Ryan 57 Okay. Not much has changed since last time. We observe that Obama spews less malarkey than Romney, and Biden less than Ryan. We observe that the blue team's statements are less full of malarkey than the red team's, and the Democratic candidates themselves are less full of malarkey than the Republican candidates. But not by much. No candidate or ticket appears much better or worse than half full of malarkey.



The trouble is, for each candidate (and party), we only have a small sample of the statements they've made. That introduces sampling error. We must calculate the certainty with which we can make judgments about the candidates and parties given the data we have.



To the right are the probability distributions of malarkey scores for the four candidates, labeled with the 95% confidence intervals on either side of the expected malarkey score. The white line lies at a malarkey score of 50.



How certain can we be that the candidates are much better or worse than half full of malarkey? From the probability distributions shown at right, I calculated the probabilities.



Odds are 9 to 1 that Obama's less than half full of malarkey, but not by much. It's almost 100% likely the Romney is more than half full of malarkey. The difference is greater than for Obama, but still not much difference than a half buck of malarkey.



The odds are only around 2 to 1 that Biden is more than half full of malarkey. Again, not by much. The same is true for Ryan.



We can only be pretty certain about how the presidential candidates compare to a half bucket of malarkey. What about the party tickets as a whole?



To the left are the probability distributions of the collated and average malarkey scores. Based on these distributions, the odds are nearly 9 to 1 that the Obiden's collective statements are on average less than half full of malarkey (but not by much), while it is almost certain that Rymney's are more than half full of malarkey (by a wider margin, but still not by much).



It's a statistical toss-up whether Obama and Biden are on average less than half full of malarkey, whereas the odds are over 19 to 1 that Romney and Ryan are on average more than half full of malarkey (but still not by much).



But how do the candidates and tickets compare, and how certain can we be about those comparisons?



To the right are probability distributions of the ratio between a Republican malarkey score and a Democrat malarkey score. Red bars occur when the Republican malarkey score is greater than the Democrat score. The comparisons run from very murky (for the v.p. candidates) to pretty clear (for the collated ticket report cards and the presidential candidates).



I am essentially 100% certain that Romney spews more malarkey than Obama...but not much more. Not even twice as much. Not even one and a half times as much. I'm also nearly 100% certain that Obiden collectively spew less malarkey than Rymney. But again, they're not that different. Basically, I can't tell a difference between Biden and Ryan. If there is a difference between them, it is tiny, but in favor of Biden. I can, however, give 9 to 1 odds that Obama and Biden are on average more factual than Romney and Ryan. But not that much more factual!



We draw two lessons. First, and I repeat from last time, the differences in factuality between the two parties aren't as large as either side would have you believe. That said, there is a clear difference. The differences we can be certain about favor Democrats. And all of the differences we've measured, regardless of our certainty in them, suggest that the Democratic candidates are more factual than the Republicans.



On to the debates.



Debates



This morning, I collected the Truth-O-Meter rulings of claims made during the final debate. To the right are the observed malarkey scores for that debate.



opponent malarkey Romney 57 Obama 46

Once again, Romney spews more malarkey than Obama. At least, that's what the data says. But how strong is the evidence? Let's bust out the simulator. To the left are the probability distributions of Obama's malarkey score in all three debates. The odd are only about 2 to 1 that Obama was more than half full of malarkey than the first debate. But they're better than 4 to 1 that he was less than half full of malarkey in the second. The odds are only 3 to 2 that he was less than half full of malarkey during the last debate. As for Romney, the odds are better than 3 to 1 that he was more than half full of malarkey during the 1st debate. His report card from the 2nd debate was surprisingly truthful, the odds 4 to 1 that he was less than half full of malarkey. As for the third debate, we're back to 2 to 1 odds that he spewed more than half a bucket of malarkey. How did the presidential candidates perform in the debates overall? To answer this question, I calculated two summary malarkey scores for the debates.



First, I collated the candidates' report cards from each debate, then calculates a malarkey score from it. This measures the average falsehood of the statements a candidate made across all the debates. The odds are about 2 to 1 that Obama's statements during the debates were less than half full of malarkey. The odds are about the same that Romney's statements were more than half full of malarkey.



Second, I averaged the candidate's malarkey score across the three debates. The odds are again about 2 to 1 that Obama was on average less than half full of malarkey during a debate. Statistically, we can't tell whether Romney was on average more or less than half full of malarkey during a debate.



People seemed very let down by Obama's performance in the first debate. Mostly, it had to do with his demeanor. But could people have been subconsciously disappointed with his truthfulness during that debate as well, perhaps cued by his subtle facial expressions and body language that he as being more false than he usually is?



Maybe. In any case, the odds are 4 to 1 he spewed more a few more falsehoods during the first debate than he usually does. The odds are again 4 to 1 that his performance improved during the second debate, when he appeared to be more factual than usual. Statistically, we can't tell a difference between the usual Obama and the Obama in the final debate.



In contrast, we can't tell a difference between the usual Romney and the Romney in the first debate. The odds are 49 to 1 that Romney was more factual during the 2nd debate than he usually is. Yet Romney lost that debate.



As for the third debate, the odds are only 2 to 1 that Romney was more factual in the final debate than he usually is. Whatever the case, he seems to have lost that one, too.



So much for the persuasive power of facts!



We've seen how the presidential candidates' compare to themselves, but how did their debate performances compare to one another? To the right are the probability distributions of the ratio of Romney's malarkey score to Obama's in each of the debates. Red means Romney spewed more malarkey, blue means Obama did. The odds are only 2 to 1 that Romney spewed more malarkey than Obama during the first debate. It's a toss-up who was more factual in the second debate. And the odds are better than 3 to 1 that Obama was more factual than Romney in the final debate. So the evidence is pretty damn weak, but favors Obama for two of the debates. But even if Obama was more factual than Romney, he probably wasn't that much more factual. What about overall performance during the presidential debates?



To the right are the probability distributions of the ratio of Romney's collated debate report card to Obama's (top), and the ratio of Romney's average debate malarkey score to Obama's (bottom). The odds are nearly 3 to 1 that Obama's statements during the debates were more factual than Romney's. The odds are only about 2 to 1 that he was more factual on average than Romney was in a given debate. Again, the evidence is fairly weak, but it favors Obama. Even if it favors Obama, the differences in factuality aren't that great.



To prepare for my analysis of the collective debate performance of each party's ticket, I review my analysis of the the vice presidential debate. To the right are the probability distributions of Biden's and Ryan's malarkey scores during the vice presidential debate.



The odds are about 3 to 2 that Biden was less than half full of malarkey during the debate. Contrastingly, the odds are nearly 9 to 1 that Ryan was more than half full of malarkey during the debate. So Biden was probably right. It was all just a bunch of stuff! Well, not all of it. Actually, not much more than half of it was malarkey.



Still, given the small amount of evidence we have from the debate (which introduces a lot of sampling error), it's quite interesting that the odds remain so high that Ryan spewed so much malarkey. Perhaps it was Biden's mastering of the facts after all that dampened the Republican's Romentum! Well, at least it was his ability to point out Ryan's factual missteps. But remember, Biden was about half full of malarkey during the debate, too.



That said, the plot at right suggests that Biden was probably about as factual as he usually is during the debate, whereas the odds are better 6 to 1 that Ryan was less factual than usual.



I'd like to think that Ryan's subtle cues of his own falsehood were a letdown to some undecided voters who had expected more from him, but the polls were pretty split about who won the debate.



Regardless of whether people think Ryan or Biden "won" the debate, the probability distribution of the ratio of Ryan's to Biden's debate malarkey score suggest better than 9 to 1 odds that Ryan spewed more malarkey than Biden. In this case, the mean difference between the two scores is actually fairly large. Or at least larger than we've come to expect from these candidates, who are all basically half charlatans. So here's a shout-out to Biden, whose spirited use of the term "malarkey" inspired the name of my factuality score. Good work, Mr. Vice President. Or at least, I'm over 90% sure it's good work.



Is there a way to analyze the collective malarkey scores across all four debates, and across the presidential and vice presidential candidates from each party? I'm Brash Equilbrium, baby! Of course there is.



I came up with two measures of malarkey overall all four debates. First, I simply collated the statements from the presidential and vice presidential candidates into two summary report cards for each party.



Second, I took the average of a presidential candidate's average malarkey across all three presidential debates, and the vice president's debate malarkey score. Let's unpack that a bit. Step one was to average the presidential candidate's malarkey across all three debates. Step two was to take the average of that average and the vice president's malarky score from the vice presidential debate.



Why did I take an average of averages? Because I wanted to measure the average malarky score of an individual on a party's ticket, not the party's average score across the four debates. If I'd done the latter, I would have weighted presidential candidates more heavily, which I already do in the collated measure since presidential candidates were more heavily fact checked, and had more debates, than vice presidential candidates.



Okay, let's look at some graphs and calculate some odds.



At left are the probability distributions for the collated and candidate average debate malarkey for each party.



The odds are almost 3 to 1 that Obiden's collective statements during the debates were less than half full of malarkey. The odds are about 2 to 1 that the average Democratic candidate's average debate performance was more than half factual.



Contrastingly, the odds are better than 6 to 1 that the collective statements of Rymney were more than half full of malarkey. The case is similar for the average Republican candidate's average debate performance.



Not the similarities between the confidence intervals and means of the debate summaries and those of each party's malarkey scores calculated from their full report cards. These similarities make me confident that malarkey scores taken from full report cards are a pretty good predictor malarkey scores accrued during events like televised debates. Remember also that the candidates' overall malarkey scores were calculated from two fact checkers, whereas the debate data comes from just one.



Maybe there is something to this Malark- O -Meter thing after all. Which brings me to our final plot.



The odds are better than 7 to 1 that the Republican candidates' collective statements during the debates were more full of malarkey than the Democrats'. The difference isn't that big, but it's not trivial.



The odds are better than 6 to 1 that the average Republican candidate's average debate performance was more full of malarkey than the average average Democratic candidate's. Again, the difference isn't big, but it's not trivial.



If factuality were all you cared about in a candidate, there is pretty strong evidence that you should prefer the Democrats over the Republicans. That said, there is also pretty strong evidence that the differences that likely exist between the two tickets aren't massive. Still, they aren't trivial.



Of course, you don't only care about factuality. You care about policy. You care about issues. But therein lies the rub. When politicians design and advocate for policies, they ideally do so with some grounding in the facts. Evidence matters, or at least it should, just as much to policymaking as it does in a courtroom or a chemistry lab.



What about values? You should care about your candidates' values too, right? But how are your candidates' values informed by the facts?



You see where I'm going here. I understand that factuality isn't the only characteristic we should consider when deciding who gets our vote.



But it sure seems to be at the root of all the others!



VOTE!



