In retrospect, there was much more at stake than a mere $1 million when IBM’s Watson computer faced off against two Jeopardy! champions back in 2011. The bot’s victory gave Big Blue a shot at conjuring up a new line of business at the perfect possible moment. A series of advances in image and speech recognition was about to trigger a frenzy of investment and excitement about the moneymaking potential of artificial intelligence.

Six years later, it’s fair to ask whether that plan could have been better executed. IBM today is even more urgently in need of new business, with quarterly results released earlier this week showing that revenue has declined for 21 consecutive quarters. CEO Ginny Rommety has made a habit of talking about Watson as a kind of savior, and the company declared this week that this part of the business is growing. But IBM won’t release details on Watson’s financial performance, and skeptics abound. Last week, investment bank Jefferies released a report warning shareholders not to expect IBM’s investments in AI to repay themselves; Watson, it said, risks being eclipsed by competing AI platforms from Google, Amazon, and Microsoft.

Talking about Watson is a good way to trigger eye rolls from people in the machine learning and AI community. There’s widespread agreement that its triumph on the specific backward-question problem of Jeopardy! was notable. Making sense of language remains one of the biggest challenges in artificial intelligence. But IBM quickly turned Watson into an umbrella brand promising a bewildering variety of bold new applications, from understanding the emotional tone of Tweets to scouring genomes for mutations. It bought startups and rebranded their wares as Watson and touted cute but hardly lucrative projects like Watson-designed recipes and dresses. In one TV commercial Watson chatted with Bob Dylan, confessing “I have never known love.”

Overhyped

Critics say IBM executives overshot badly by allowing marketing messages to suggest that Watson’s Jeopardy! breakthrough meant it could break through on just about anything else. “The original system was a terrific achievement, there’s no question about that,” says Oren Etzioni, CEO of the Allen Institute for AI. “But they’ve really over-claimed what they can deliver in a big way; the only intelligent thing about Watson is their PR department.”

It’s often said that there’s no such thing as bad publicity, but buzz that's out of proportion with your product risks setting up customers for disappointment. Sabri Sansoy, an independent machine-learning consultant who has worked on projects for clients including ad-agency giant Ogilvy, says he routinely has to deflate dreams built on Watson marketing messages. “Everyone knows Watson,” says Sansoy, who sometimes uses IBM’s services. “But a lot of people get confused and think it’s one massive artificial general intelligence service that can do everything.”

In fact, like all the AI systems in use today, Watson needs to be carefully trained with example data to take on a new kind of problem. The work needed to curate and label the necessary data has been a drag on some projects using IBM’s system. Ashok Goel, a computer science professor at Georgia Institute of Technology, got written up in The Wall Street Journal and Backchannel after building a Watson bot to answer questions from students to his online course on artificial intelligence. But its performance was limited by the amount of manual labeling of data needed. “It had fairly high precision, but it did not answer a very large number of questions,” Goel says. “We have gradually moved away from IBM Watson for this reason.” (He continues to work with Watson on other projects, for example building a research assistant bot for scientists at the Smithsonian.)