The world was caught completely off guard in 2016. It shouldn’t have been.

“Next time someone shows me a poll, I’m using it to wrap dead fish.”

This quote comes from CNN political commentator Ana Navarro, but in the days immediately following the election of U.S. President Donald Trump, it was a sentiment shared across the world. Liberals and conservatives were united in shock at the outcome, after virtually every expert agreed that Hillary Clinton would win the 2016 U.S. Presidential election with ease.

It was a damning failure after more than a year of focused campaigning with the sole goal of public introspection, of giving voters and politicians an understanding of wide-ranging public opinion. The entire media and political establishment seemed overcome with surprise and shame, paralyzed by a single, all-consuming question: Given all the trust these institutions have maintained over the history of political discourse, how could something like this possibly happen?

At Sentigraph, we understand that the question should really be: How could something like this have possibly taken so long to happen?

Modern techniques for analyzing public sentiment are, quite simply, broken. A combination of non-random sampling and corrupt question phrasing has made pure political polling all but useless — even with the help of sophisticated aggregators like FiveThirtyEight. These are sites that explicitly bill themselves as correction services for otherwise myopic or faulty polling practices, and even they were wrong by enormous margins. After the election, they were reduced to posting convoluted explanations for their failures or, interestingly, doing yet more polling about polls.

At a certain point, the alternate reality of accepted expert opinion can become so powerful that even experts, who can see the reality before them, assume that it’s their observations that are wrong, rather than the suspect, polling-born theories that contradict them.

Sentigraph was created to address these problems. The public needs to make use of the fact that humanity has never had more access to information about itself, but society today seems more confused than ever about the actual thoughts and motivations of the people who comprise it.

By using our proprietary social media mining practices, Sentigraph focuses on the real social data being produced by real people in the wild. Not the curated and politically savvy responses of a polling cohort, but the genuine social consciousness of a nation. That way, our analyses can work based on real intelligence and not inaccurate abstractions like a phone poll.

Our approach focuses on Twitter, the most vibrant and forward-looking platform for political and social discourse in the world today. That’s not to say that it’s the best platform for calm, reasoned debate, but calm, reasoned debate often leads to grossly inaccurate understandings of people’s beliefs. Twitter may be unpleasant at times, but the ultra-candid conversations it evokes can be sifted to find a uniquely meaningful snapshot of public opinion.

After mining real discourse out of social media, and condensing it into a more easily analyzed form, we send this body of work through IBM’s Watson for crunching by an advanced machine learning algorithm. Watson’s unrivaled abilities in sentiment analysis allows researchers to wring every drop of insight from their data, and organize that insight in the most intuitive and easily applied formats possible. By using a proprietary 3 data mining approach, then handing that data to the tried-and-true Watson platform, Sentigraph can combine unique insight with Amazon’s trusted customer service.

It also makes the sentiment analysis workflow quick and easy enough that it can address real-time concerns, taking not just a snapshot but a series of snapshots that show how subcultures evolve over time. Given the power of the tools available, we should never again be surprised by the size and strength of movements that are all around us, and which have been all around us for an astonishingly long time.

Americans will soon pick their candidates in the 2018 midterm elections, and it’s important that they have a better understanding of just what is motivating people to vote in a particular way. It will show politicians where, precisely, their constituents want help and who, precisely, needs to be targeted for additional campaigning. More to the point, it can combat the feeling of shock and helplessness that has permeated society.

The only realistic way to repeat the improbable shock of 2016 is to repeat the faulty analysis done in 2016: more polls aggregated with a higher level of confidence. With Sentigraph widely available as a tool for societal introspection, there’s simply no need for anything similar to ever happen again.