Can you understand me? Steve Read/The Image Bank/Getty

Vocoders just got a serious upgrade. A new speech synthesiser can translate mouth movements directly into intelligible speech, completely bypassing a person’s voicebox.

Although the synthesiser might not be immediately useful, it’s a first step towards building a brain-computer interface that could allow paralysed people to talk by monitoring their thought patterns.

To create the speech synthesiser, scientists at INSERM and CNRS in Grenoble, France used nine sensors to capture the movements of the lips, tongue, jaw and soft palate. A neural network learned to translate the sensor data into vowels and consonants, which are emitted from a vocoder. The output sounds, unsurprisingly, like a robotic monotone, but the words are distinguishable.


To make it work for people who are unable to move their vocal tract, we’ll have to learn to decode signals from the brain. Recent research has shown that the speech area of the motor cortex contains representations of the various parts of the mouth that contribute to speech, suggesting it might be possible to translate activity in that region into signals like the sensor data used in the synthesiser.

Journal reference: PLOS Computational Biology, DOI: 10.1371/journal.pcbi.1005119

Read more: Telepathy machine reconstructs speech from brainwaves