As with our direct measurements of Reputation percentages, both algorithms handle 55% noise with no difficulty: 100% of their answers are correct. Augur’s clustering algorithm maintains this performance even at 85% noise. By contrast, Truthcoin begins to have noticeable trouble at 65% noise, although it still gets 3 out of 4 outcomes right. At 85% noise, Truthcoin’s performance has deteriorated significantly: it only provides correct answers for 35% of events. These results suggest that, while PCA is a generally very useful statistical technique, it may simply not be well-suited to the problem of event consensus. It fails to meet Augur’s standard as a referee system for a prediction market.

An alternative explanation might be that, while Truthcoin consensus happens to do poorly in the presence of lots of white noise, it might perform better when presented with more structured reporting data. One way to investigate this possibility is to modify our simulation set-up a bit, so that the behavior of “liars” is a bit more like actual human liars (rather than simply being lazy). It seems reasonable that dishonest reporters could clump together into “conspiracies”, which might represent a reporter and all the people she knows in person, or all her friends from an Internet group. And, since they’re conspiring together, all reporters in a conspiracy give the same set of (typically incorrect) answers.

In this modified simulation set-up, the collusion parameter, gamma, is now the probability that a dishonest reporter will join a pre-defined group of other dishonest reporters. We ran simulations using this modified set-up. In our simulations, we set the number of available “conspiracies” (somewhat arbitrarily) to 5. (A more involved model of conspiracies might make use of, for example, growing preferential attachment networks.) Here are our results: