When on social media, we like to present the most flattering version of ourselves. But our Google searches reveal our ugliest angles, our darkest thoughts and our greatest fears.

Seth Stephens-Davidowitz, a former Google data scientist and the author of Everybody Lies: What the Internet Can Tell Us About Who We Really Are, has made a career out of analysing what we type into the search engine. He’s interested in our lonely admissions and secret confessions (“I am sad” or “Is it normal to want to kill my family?”); the questions we are too embarrassed to ask one another (such as “Why is my poop green?” – the third most popular question beginning with “why is” on Google); and all the other things that thousands of people type into the blank search box each year: “People are annoying”, “I regret having children”, “I love my girlfriend’s boobs”.

In Everybody Lies, he writes: “Google searches are the most important data set ever collected on the human psyche.” His findings are so intriguing that this lack of humility is almost forgivable. Google searches indeed offer more data than even the most ambitious surveys or scientific studies. Although we tend to lie to ourselves, our friends, our social media followers and opinion pollsters, our private Google searches are a repository of discomforting truths.

Stephens-Davidowitz is unlikely to have been surprised at the eruption of white supremacist violence in Charlottesville, Virginia, in August. Political scientists seeking to explain the US’s persistent racial inequality often focus on subconscious bias, because most white Americans vehemently deny that they are racist.

Yet Google data suggests that explicit racism is far more widespread than answers to phone surveys or social media posts suggest. In some US states, there were more searches for “nigger president” than “first black president” on the night of Barack Obama’s 2008 election. Eight years later, the best predictor of which regions would vote for Trump was the volume of racially charged internet searches.

Disheartening as it is to learn that American society is more racist – and more sexist and more Islamophobic – than many would hope, some positives can be drawn from this grim data cache. It offers hints, for instance, of how best to defeat prejudice. Stephens-Davidowitz’s findings suggest that lecturing bigots doesn’t work – it can simply inflame their anger instead. In the aftermath of the 2015 terrorist attack in San Bernardino, California, Obama delivered a stirring speech urging Americans to remember that “freedom is more powerful than fear”. While he spoke, Google searches for the term “kill all Muslims” tripled.

Provoking curiosity may, however, be a better way to fight hatred. In a subsequent speech, Obama described American Muslims as “our friends and our neighbours, our co-workers, our sports heroes and… our men and women in uniform”. For the first time, searches for “Muslim athletes” and “Muslim soldiers” were more common than those for “Muslim terrorists”, “Muslim extremists” or “Muslim refugees”.

The finding aligns with other research into the role that curiosity plays in countering bias. Numerous studies have shown that education, scientific knowledge and self-reflection does not stop people from rejecting new information that contradicts their world-view. Yet research conducted by a team at Yale in 2016 suggested that people who were curious were also much more likely to be open-minded about information that does not reinforce their existing beliefs.

Those concerned by racism and xenophobia might want to consider ways in which teachers and policymakers can foster curious citizens, who are willing to be proved wrong about their prejudices, who are intrigued rather than threatened by difference, and whose interest in the world extends outwards to new cultures and perhaps also downwards, deep into our Google data mine.