Imagine the visualization shows you how close your expectations are to the poll results after you sketch your line. The solid line that animates in in the below graph shows how Warren’s popularity in the polls did increase in the third week of June, but then dipped slightly around the first debate.

Consider how reflecting on your expectations may have changed what you take away from the poll visualization. Without this prompting, would you have noticed the slight dip in Warren’s popularity? Consider also what you might have noticed if your expectations had been different. What if you had expected that Warren’s popularity had gone down after the debate — would you feel more confident in your ability to project political opinion?

It’s clear that eliciting beliefs changes a user’s interaction. We wondered, is this a good thing?

Why Elicit Beliefs?

A couple years ago, we conducted an experiment to test different combinations of graphical elicitation of beliefs (for example, with and without feedback to make it explicit) against the typical visualization scenario in which the user is simply shown the observed data by default, as well as against text elicitation of beliefs. (Read a summary here). We found that when users had the ability to draw their expectations and see them against the observed data, they recalled the data 20–25% more accurately a short time later. Stating one’s expectations in a text format (e.g., by typing predicted values on a line), however, did not have the same effect. This leads us to suspect that viewing the gap between one’s beliefs and observed data visually is a powerful way to help a user realize how much they know (or don’t know).

A visualization that elicits users’ beliefs can prompt further critical reflection on data if it also shows what others expected a trend to look like.

In another experiment we conducted, we found that viewing others’ expectations can have a similar effect to viewing one’s own predictions, improving a user’s ability to remember data when others’ expectations are reasonably consistent. Viewing others’ beliefs can also prompt users to think more critically about the data they are viewing. Especially when a user’s expectations are contradicted by the data, seeing what other people thought can impact how much a user updates their beliefs toward the trend in the observed data.

While social influence may in some cases lead users to take trustworthy data less seriously, we think that acknowledging that people have beliefs before they see data can be informative for users. Without visualization techniques that prompt critical thinking — whether by graphically eliciting and representing prior beliefs, visualizing uncertainty directly, or otherwise acknowledging limitations — some visualization users may default to trusting datasets blindly. Recognizing that prior beliefs can contain valuable information is a first step toward thinking of data as a tool for informing what we think, not replacing it with each new data sample we see.

An Authoring tool for Belief Elicitation: TheyDrawIt!

Creating a visualization that smoothly graphically elicits and visualizes prior beliefs from users can be time consuming. So, we created TheyDrawIt!, an authoring tool for producing interactive “belief-driven” visualizations of time series data. TheyDrawIt! visualizations can be customized in various ways but use a consistent design pattern of elicit beliefs, show observed data against beliefs, and (optionally) show prior users’ beliefs.

To get a sense of how it works, let’s walk through the earlier 2020 Democratic Nomination polls example and re-create it in TheyDrawIt!

Importing Data

TheyDrawIt accepts data in Google Spreadsheets where all data is in a single sheet. Your Google Spreadsheet should have at least one column that has a recognizable Date/Time format in chronological order. For our example, we’re using poll data we scraped from RealClear Politics on the 2020 Democratic Presidential Nomination.

Design — Customize your visualization

After you’ve loaded your data, TheyDrawIt will ask you to specify which column contains the Date/Times you want to use as the x-axis of your visualization. You can then specify each of the other columns (or lines) you want to include in the visualization. This set should include the column you want your users to predict. While your users will only be able to predict one column (line), other columns can be shown by default to provide context that may help users as they formulate their predictions.