Twitter's new head of news, Vivian Schiller, announced her first project today: a partnership with CNN and the New York startup Dataminr. The initiative will help journalists cover breaking news by making sense of the flood of public information on Twitter. Dataminr uses machine learning algorithms to analyze the Twitter firehose of data and highlight the needle in the haystack so CNN reporters can find the most important, relevant, and reliable facts and images from around the world.

Using machine learning algorithms to analyze the Twitter firehose of data

Dataminr is a five-year-old startup that has raised $48 million from investors like Box Group, Venrock and IVP. Until now it has worked with clients in financial services, helping banks and hedge funds make real-time investment decisions by processing huge amounts of social media content and extracting the high-value information. For example Dataminr told its financial clients that the AP tweet about an explosion at the White House was false five minutes before the AP itself corrected the facts.

Dataminr has now built a new tool tailored for journalists, chopping news up into different feeds and pinging reporters at their desks or via mobile app with alerts about breaking stories. It's hoping to be able to alert news outlets to breaking news before they would get it otherwise, as the service claims it was able to do when detecting the death of Osama Bin Laden before any official outlets reported it.





"There is actually a very different set of algorithms that help us determine if information spreading on Twitter will move markets, as opposed to becoming a big news story," said Dataminr CEO Ted Bailey. "So it's not a matter of more or less data, but of how the service is focused and tuned."

Sites like CNN have put Twitter front and center in their news gathering going back to 2009, when the Twitpic of passengers on the wing of a crashed plane floating in the Hudson became the iconic image that helped define the story. Today's announcement is a deepening of that practice, bringing a purpose-built product and official partnership. For example, CNN first learned about the shooting this weekend at a mall in Maryland through Dataminr, which has been deployed in-house for several months now, and picked up on a tweet from a first responder on the scene. The alert helped CNN be one of the first on the story.

The company says it now produces about two stories a day based off tips from its Dataminr alerts. "It's like bionic vision for our reporters," said Kenneth Estenson, general manager of CNN digital. "It helps us to see things faster than our competition and to act with confidence."