After the difficulties experienced by Twitter bot Tay, the new service is providing amusement while learning how to automatically caption images

CaptionBot is Microsoft's latest AI experiment - and at least it isn't racist

After the somewhat awkward experience last month of having an AI Twitter bot go full-on racist in a few hours once it interacted with humans, Microsoft have released a new AI experiment on to the internet - CaptionBot.

The idea is that you upload a photo to the service, and it tries to automatically generate a caption that describes what the algorithm sees. You are then able to rate how accurately it has detected what was on display. It learns from the rating, and in theory, the captions get better.

The bot, from Microsoft’s Cognitive Services team, is the result of some hefty research into how to model objects in photographs so that a computer can understand them. They claim that their system can recognise “a broad range of visual concepts” and also performs entity extraction so that it can recognise celebrities. Which it clearly can do.

Facebook Twitter Pinterest Taylor Swift and Kanye West both identified in Microsoft’s CaptionBot app. Photograph: Microsoft

It incorporates three separate services to process the images. The Computer Vision API identifies the components of the photo, it mixes that with data from the Bing Image Search API, and runs any faces it spots through their Emotion API. This analyses human facial expressions to detect anger, contempt, disgust, fear, happiness, sadness or surprise.

Other services produced by the group include “How Old Do I Look” and the “What Dog?” robot.

Of course, we immediately tested CaptionBot by uploading pictures from Doctor Who, with admittedly very little success.



Facebook Twitter Pinterest A picture of some sort. Well spotted CaptionBot. Photograph: Microsoft

Although this result was possibly an astute comment on the state the show was in by the late 1980s.

Facebook Twitter Pinterest ‘Colourful’ for sure. Photograph: Microsoft

The internet has also wasted no time testing the limits of the technology - uploading pictures of Hitler to see whether they can make CaptionBot racist, or making it the butt of some pretty good jokes.

Ralph Harrington (@ralphharrington) Microsoft's image captioning tool sees through the so-called "moon landings" https://t.co/WWr7O1XeE3 H/T @robmanuel pic.twitter.com/BDd9X4KdNp

Paul Curry (@cr3) I was hoping to get a definite answer from https://t.co/b5DYRwRxWz but it's raised a lot more questions pic.twitter.com/WVz4gC9kLi

Nicholas Tollervey (@ntoll) A mixture of amazing, complementary and wrong from https://t.co/zMWockgew4 :-) Really great work @Microsoft pic.twitter.com/AfNTfJzhuf

On the ranking of slightly dodgy AIs, not recognising K-9 is a significant step up from “genocidal racist”, so congratulations Microsoft.

Has CaptionBot been able to recognise any of your images? Have you tormented it with your own fondly remembered childhood TV shows? Share what you found in the comments below.

This article contains affiliate links, which means we may earn a small commission if a reader clicks through and makes a purchase. All our journalism is independent and is in no way influenced by any advertiser or commercial initiative. By clicking on an affiliate link, you accept that third-party cookies will be set. More information.