How are you supposed to react when a robot calls you a “gook”?

At first glance, ImageNet Roulette seems like just another viral selfie app – those irresistible 21st-century magic mirrors that offer a simulacrum of insight in exchange for a photograph of your face. Want to know what you will look like in 30 years? There’s an app for that. If you were a dog what breed would you be? That one went viral in 2016. What great work of art features your doppelganger? Google’s Arts & Culture app dominated social media feeds in 2018 when it gave us a chance to bemoan being more Picasso than Botticelli, or vice versa.

The enduring popularity of these apps, dubious origins and privacy policies be damned, speaks to our basic insecurity. They cater to the part of us that, aware of how much time we spend looking at screens, starts to wonder what the screens see back – a shortcut to selfie-awareness.

But ImageNet Roulette, a project developed by the artificial intelligence researcher Kate Crawford and the artist Trevor Paglen, flips this basic formula on its head. Built in concert with their new exhibition, Training Humans, at the Fondazione Prada in Milan, the site’s goal is not to use technology to help us see ourselves, but to use ourselves to see technology for what it actually is.

Rise of the racist robots – how AI is learning all our worst impulses Read more

The site’s algorithm was trained on photos of humans contained in ImageNet, a dataset described by Crawford as “one of the most significant training sets in the history of AI”. Created in 2007 by researchers at Stanford and Princeton, ImageNet includes more than 14m photographs, mostly of objects but also of humans, that have been classified and labeled by legions of workers on Amazon’s crowdsourcing labor site, Mechanical Turk.

If you upload your photo, ImageNet Roulette will use AI to identify any faces, then label them with one of the 2,833 subcategories of people that exist within ImageNet’s taxonomy. For many people, the exercise is fun. For me, it was disconcerting.

As a technology reporter, I’m regularly tasked with writing those scolding articles about why you should be careful which apps you trust, so I usually eschew viral face apps. But after a day of watching my fellow journalists upload their ImageNet Roulette selfies to Twitter with varying degrees of humor and chagrin about their labels (“weatherman”, “widower”, “pilot”, “adult male”), I decided to give it a whirl. That most of my fellow tech reporters are white didn’t strike me as relevant until later.

I don’t know exactly what I was expecting the machine to tell me about myself, but I wasn’t expecting what I got: a new version of my official Guardian headshot, labeled in neon green print: “gook, slant-eye”. Below the photo, my label was helpfully defined as “a disparaging term for an Asian person (especially for North Vietnamese soldiers in the Vietnam War)”.

Facebook Twitter Pinterest How artificial intelligence classified Julia Carrie Wong’s headshot. Photograph: ImageNet Roulette

On the one hand, this is exactly the outcome that Crawford and Paglen were aiming for. ImageNet Roulette is not based on a magical intelligence that shows us who we are; it’s based on a severely flawed dataset labeled by fallible and underpaid humans that shows us its limitations.

“We want to shed light on what happens when technical systems are trained on problematic training data,” they wrote. “AI classifications of people are rarely made visible to the people being classified. ImageNet Roulette provides a glimpse into that process – and to show the ways things can go wrong.”

But my experience with ImageNet Roulette also occurred during a strange week for people of Asian descent in America, when the public was engaged in a fraught debate over whether or not it is funny to call Chinese people “chinks” and Asian Americans were grappling with the novel experience of having a national political “representative” who does not necessarily represent our views.

I found myself both oddly upset and oddly relieved to be labeled a gook. As a biracial “Jew chink” (yes, Shane Gillis, we actually exist outside your pathetic punchlines) with brownish skin and a bony nose, people usually assume that I’m any ethnicity but Chinese. Having a piece of technology affirm my identity with a racist and dehumanizing slur is strange.

Still, isn’t that what we’re all looking for when we look into these magic mirrors? We want to know how the world sees us. I got my answer.