IBM is dropping its own Hadoop platform and adopting Hortonworks Inc.'s instead as part of a two-way deal that could give users of both companies increased access to enterprise-class capabilities for managing and analyzing big data.

IBM will end development of BigInsights, its distribution of Hadoop, and work to migrate existing users to the Hortonworks Data Platform (HDP). In return, Hortonworks will resell IBM's Data Science Experience suite of tools for collaborative analytics, as well as Big SQL, a SQL-on-Hadoop query engine developed by IBM. The two companies will also do joint development to expand the features of Apache Atlas, an open source data governance framework spearheaded by Hortonworks.

The deal should be "a shot in the arm" for the product offerings of both vendors, Ovum analyst Tony Baer said at DataWorks Summit 2017 in San Jose, Calif., where the deal was announced.

IBM "gets to say they have a Hadoop strategy again," after facing widespread doubts about the future of BigInsights over the past two years, Baer said. And Hortonworks can now offer its users a data science platform to help them coordinate work on advanced analytics applications, matching technologies previously released by Hadoop rival Cloudera Inc. and several analytics vendors, in addition to IBM.

During the conference's opening keynote session, Rob Thomas, general manager of IBM Analytics, said leaning on HDP as the underlying Hadoop platform will let IBM focus more on developing its data science and machine learning technologies. That functionality will then be made available to both IBM and Hortonworks users through the Data Science Experience, which IBM shortens to DSX.

Machine learning comes to the fore The heightened focus on products like DSX and Cloudera's rival Data Science Workbench aligns with the growing use of machine learning and other forms of artificial intelligence (AI) cited by various attendees at the conference, which was organized by Hortonworks and its former parent company, Yahoo. For example, Duke Energy Corp. is looking to use machine learning and AI tools to identify potential equipment problems by analyzing sensor data from its transmission network, said John Pressley, director of information architecture at the electric utility and natural gas distributor in Charlotte, N.C. Machine learning fueled by data science techniques could also enable more proactive and personalized customer service, Pressley said during a user panel discussion. "Where we want to take it next is more streaming and real-time information so there's nothing that stops us from answering a customer's call and understanding what their sentiment is on that phone call," he said. "And then we can be changing [promotional] offers as we get to know what that customer's most important thing is." Meanwhile, the planned development work on Atlas signals a continued elevation of data governance and security as key issues in big data environments, according to Gartner analyst Merv Adrian. "The huge majority of Hadoop adopters that are currently stalled in attempting to get to broad production use will need to deal with those issues when and if they break the logjam," Adrian said. IBM's involvement with Atlas could provide "real credibility and experience" to the still-emerging governance technology, he added. But the move raises questions about an existing Hadoop platform partnership between Hortonworks and Microsoft. In addition, it continues the consolidation of the Hadoop market, which is now down to four vendors: Hortonworks and fellow big data specialists Cloudera and MapR Technologies, plus cloud-platform market leader Amazon Web Services.