Before joining FRL to build and lead a team dedicated to BCI research, Chevillet was at the Johns Hopkins University Applied Physics Laboratory (APL). “The thing that drew me to APL was their work combining cutting-edge robotics and surgically implanted electrodes to develop a prosthetic arm for wounded combat veterans that could be controlled just like their natural arm,” he says.

Inspired, Chevillet helped APL build up a broader program of related research projects in applied neuroscience to see where the technology could go in the future — and how it could potentially help even more people in a non-invasive way. “I led an internal research program to see if we could use some of the ideas from motor prosthetics and some of my other work in cognitive neuroscience to make a communications prosthetic for people who can’t speak,” he explains. “We were also trying to figure out if we could use a non-invasive approach rather than implanted electrodes — so the technology could be used by many more people.”

After being recruited by Facebook to build a team focused on BCI, Chevillet was encouraged to propose and pursue an ambitious, long-term research project that pushed the boundaries of what was possible and redefined the state of the art. And that’s when it hit him: BCI for communication using a wearable device wasn’t just relevant for people who couldn’t speak — it might also be a very powerful way for people to interact with their digital devices.

While voice is gaining traction as an input mechanism for smart home devices and phones, it’s not always practical. What if you’re in a crowded room, walking down a noisy city street, or in a hushed art gallery? As Chevillet realized, “Most people have used the voice assistant on their phone, but they hardly ever use it in front of other people.”

But what if you could have the hands-free convenience and speed of voice with the discreteness of typing?

Two teams, a shared goal

With that vision in mind, Chevillet set out to build an interdisciplinary team that could figure out whether his dream of a non-invasive, wearable BCI device might become a reality.

“We clearly needed somebody who really understood speech production and the neuroscience behind it, and we lucked out and found Emily — or she found us,” he says. As the team grew, the interdisciplinary nature of the problem attracted a diverse array of top-notch researchers, with backgrounds ranging from biomedical imaging to machine learning and quantum computing. But there was one problem the FRL team was not equipped to solve — before they could figure out exactly what type of wearable device to build, they needed to know whether a silent speech interface was even possible and, if so, which neural signals were actually needed to make it work.

As it stands today, that question can only be answered using implanted electrodes, which is why Chevillet reached out to his long-time colleague Edward Chang — a world-renowned neurosurgeon at University of California, San Francisco (UCSF), where he also runs a leading brain mapping and speech neuroscience research team dedicated to developing new treatments for patients with neurological disorders.

“I went to see Eddie in his office and just sorta laid out the vision,” Chevillet recalls. “I explained that I wanted to know if speech BCI could be possible using a wearable device, and how supporting his team’s research involving implanted electrodes might move our own non-invasive research forward.”

It turned out that Chang had long been planning to develop a communication device for patients who could no longer speak after severe brain injury — like brainstem stroke, spinal cord injury, neurodegenerative disease, and other conditions. But he knew that such an ambitious goal needed resources that weren’t readily available to pull it off in the near term. The opportunity seemed like an exciting one.

After discussing the importance of UCSF’s research aimed at improving the lives of people suffering from paralysis and other forms of speech impairment, as well as Facebook’s interest in the long-term potential of BCI to change the way we interact with technology more broadly, the two decided to team up with the shared goal of demonstrating whether it might really be possible to decode speech from brain activity in real time.

Promising — yet preliminary — results

Today in Nature Communications, Chang and David Moses, a postdoctoral scholar in Chang’s lab at UCSF, published the results of a study demonstrating that brain activity recorded while people speak could be used to almost instantly decode what they were saying into text on a computer screen. While previous decoding work has been done offline, the key contribution in this paper is that the UCSF team was able to decode a small set of full, spoken words and phrases from brain activity in real time — a first in the field of BCI research. The researchers emphasize that their algorithm is so far only capable of recognizing a small set of words and phrases, but ongoing work aims to translate much larger vocabularies with dramatically lower error rates.

The past decade has seen tremendous strides in neuroscience — we know a lot more about how the brain understands and produces speech. At the same time, new AI research has improved our ability to translate speech to text. Taken together, these technologies could one day help people communicate by imagining what they want to say — a possibility that could dramatically improve the lives of people living with paralysis.

The work done for the Nature Communications paper involved volunteer research participants with normal speech who were already undergoing brain surgery to treat epilepsy. It’s part of a larger research program at UCSF that has been supported in part by FRL in a project we call Project Steno. The final phase of Project Steno will involve a year-long study to see whether it’s possible to use brain activity to restore a single research participant’s ability to communicate in the face of disability. In addition to providing funding, a small team of Facebook researchers are working directly with Chang and his lab to provide input and engineering support. UCSF oversees the research program and works directly with research volunteers. Facebook researchers have limited access to de-identified data, which remain onsite at UCSF and under its control at all times.

Ultimately, the researchers hope to reach a real-time decoding speed of 100 words per minute with a 1,000-word vocabulary and word error rate of less than 17%. And by demonstrating a proof of concept using implanted electrodes as part of their effort to help patients with speech loss, we hope UCSF’s work will inform our development of the decoding algorithms and technical specifications needed for a fully non-invasive, wearable device.

We’re a long way away from being able to get the same results that we’ve seen at UCSF in a non-invasive way. To get there, FRL is continuing to explore non-invasive BCI methods with other partners, including the Mallinckrodt Institute of Radiology at Washington University School of Medicine and APL at Johns Hopkins.

And we’re starting with a system using near-infrared light.

Seeing infrared

Like other cells in your body, neurons consume oxygen when they’re active. So if we can detect shifts in oxygen levels within the brain, we can indirectly measure brain activity. Think of a pulse oximeter — the clip-like sensor with a glowing red light you’ve probably had attached to your index finger at the doctor’s office. Just as it’s able to measure the oxygen saturation level of your blood through your finger, we can also use near-infrared light to measure blood oxygenation in the brain from outside of the body in a safe, non-invasive way. This is similar to the signals measured today in functional magnetic resonance imaging (fMRI) — but using a portable, wearable device made from consumer-grade parts.