Stay on Top of Emerging Technology Trends Get updates impacting your industry from our GigaOm Research Community

In most cases, people who use technology to augment their bodies or their senses do so because they want to either make up for something they’re missing — a limb, for example, or the sense of sight — or because they want to make themselves smarter, by using Google Glass or similar devices to look things up or take pictures. But cyborg Neil Harbisson did something very different: he essentially invented a new sense by combining one he doesn’t have with one that he does: using an antenna-like implant with a camera, he is able to hear colors.

I’ve seen a lot of different kinds of technology over the course of my journalism career, and I’ve had the chance to meet a number of fascinating scientists, entrepreneurs and artists, but I will say this: I have never heard or seen anything as mind-blowing as Neil Harbisson. Listening to him talk at the Mesh conference on Tuesday about what it’s like to hear colors, and how that has changed the way he experiences the world, was one of the most fascinating things I think I’ve ever seen or heard (Note: I am a co-founder of Mesh).

Harbisson was born with achromatopsia, a rare disorder that made him completely color-blind — instead of the rainbow of hues that most people can see, he sees only black and white and shades of grey. As a child, Harbisson became interested in music, and while at university in 2003 he ran into a cybernetics researcher, and the two came up with the idea of using a digital camera to turn the colors that Harbisson couldn’t see into sounds, based on the concept that a specific frequency of light is essentially equivalent to a sound wave.

Neil hears ultraviolet and infrared

Originally, the backpack-sized apparatus took only a small number of colors and turned them into tones Harbisson could hear through a bulky headset, but he wanted to go further. The device gradually got smaller and went wireless, and then he took the ultimate step: he had a sound-conducting chip implanted in his head, and last year a flexible shaft with the camera on it was permanently attached to his skull. In effect, he became a cyborg. With the latest software upgrade, he became able to hear ultraviolet and infrared and his chip now has a Bluetooth connection.

At Mesh, Harbisson talked about some of the daily irritations that come with looking so different: for example, he said that people often assume that his antenna is a kind of Google Glass device, and that he is recording them — he says he actually made the camera more obvious than it used to be as way of convincing people that he is not doing anything surreptitious. And he had regular problems flying anywhere because security officials wanted him to remove the device (Steve Mann, an early pioneer in wearable technology, had similar problems at airports).

Eventually, Harbisson convinced the authorities in Britain to let him have his passport photo taken with the camera attached, which he says made him the first cyborg to be legally identified as such, and now he has fewer problems with airport staff. But he told me he does often get accosted by people who think he is doing something suspicious — although just as often, people are curious and want to know more about what the antenna is and why he wears it.

What sound would you like to be today?

What fascinated me most about Harbisson, however, was the way he talked so matter-of-factly about being able to hear colors, and how this has changed the way he interacts with the world. One way is that he uses his new sense (which is really an induced form of what scientists call synesthesia, which some people are born with) to create art — so he performs music that is based on the sounds he hears when he looks at certain paintings, for example, or based on photos that are beamed to his chip via the Bluetooth connection.

[ted id=1512]

One of the things that color-blind people often find challenging is picking out what clothes to wear, but Harbisson has a fool-proof system based on his new sense: “I look through my closet and decide what chord I want to be today,” he says. So yellow pants, a blue shirt and a pink jacket like the ones he wore in his TED video would be roughly equivalent to a C-major chord, given the combination of the various notes.

When he first installed the chip, Harbisson said he found the constant sounds jarring and confusing, and he suffered from headaches for several weeks. But eventually his brain adjusted, to the point where he said that even when he turns the camera off (by covering it with his hand or when he is in a darkened room) he still hears musical notes. At some point, he even started to hear music representing color in his dreams, and says he couldn’t imagine living without it.

The point of designing the antenna, Harbisson says, wasn’t to make up for some perceived lack or disability, but to use technology to augment his experience of the world — and his ability to perform art — by essentially creating a brand new sense. Which raises a fascinating question: What other senses could we create or enhance in similar ways? How else could we use technology to not just replace a missing limb or sense, but to make them better than they were in the first place?

Those are some of the questions that Harbisson and others are trying to explore through the Cyborg Foundation, which he runs from Barcelona along with his partner and fellow cyborg artist Moon Ribas.

Post and thumbnail images courtesy of /newsrooms