A new brain-computer interface (BCI) is the first to translate both speech and movement with a single device that reads data from implanted electrodes.
In a recent paper published in Nature Neuroscience, researchers revealed how their new BCI simultaneously communicates verbal and non-verbal information in seconds.
This is a major step forward in BCI research, as human communication relies on body language as much as spoken words to convey meaning. For the first time, one implant marries these two separate elements of communication, offering a breakthrough for individuals with severe paralysis.
A New BCI
The team’s new BCI translates electrical signals from the brain into text using artificial intelligence. The decoded information can then be used to control a digital avatar, guiding both its speech and movements.
The BCI field has made significant strides in recent years, allowing humans to control cursors and robotic arms or even communicate through synthesized speech. However, those projects have generally focused on a single mode of communication, whereas this new system is multifunctional, allowing it to capture more of the rich nuance of human communication.
“This gives us hope for neural prosthesis that will enter day-to-day usage,” said Christian Herff, a computational neuroscientist at Maastricht University in the Netherlands.
Since both are related to communication, it should be no surprise that the neural signals responsible for speech can overlap. That can make the business of separating them for two different functions at the same time extremely challenging. Additionally, when the brain commands the body to speak and move at the same time, the neural activity coordinating those activities can vary from what it would be if only one were occurring at a time.
Collecting Human Neural Data
Two volunteers worked with the researchers to test the new BCI, which involved surgically implanted arrays comprising 253 electrodes on the surface of the cortex. By covering portions of the sensorimotor cortex with so many electrodes, the team captured a large number of signals associated with attempted speech and movement.
Following a brainstem stroke, one volunteer experienced impaired speech and movement capabilities. The team instructed him to wave, shake his hands, clap, and nod. Additionally, they gave him five phrases to attempt to speak silently.
The other volunteer had amyotrophic lateral sclerosis (ALS), a motor neuron disease that causes progressive motor impairment and can lead to loss of speech. Although he could only move his facial muscles, he was capable of typing messages. Researchers gave him ten phrases to imagine vocalizing and ten gestures to imagine performing.
BCI Testing Results
Using the electrode implants, the BCI recorded neural activity generated by each attempt, including when participants attempted movement and speech separately or simultaneously. The team then trained AI models to translate those signals, finding that including data from simultaneous activities improved decoding performance.
With this neural activity mapped, the team then had the volunteers attempt to control a digital avatar using the BCI. Again, testing involved isolated and combined verbal and gestural communication. However, this time, instead of relying exclusively on preselected responses, the participants were asked questions in real time.
Accuracy varied between the volunteers, with the first achieving perfect BCI decoding of the tested movements and speech, while the second recorded 75% speech accuracy and 85% gesture accuracy.
While achieving such promising results for combined speech and gesture signals represents a major step forward in BCI research, more work is needed to continue refining the technology. Higher levels of speech accuracy have been achieved with speech-only devices, and this test was limited to individual sentences rather than continuous speech. Additionally, the device captured gestures rather than steady hand positions.
These limitations were partly intentional, as the team initially simplified its decoding tasks while making the jump to simultaneous actions. Further research could expand the system to recognize movements involving additional parts of the body and enable finer control of the many joints in the hands.
The paper, “Simultaneous Speech and Gesture Decoding for Multimodal Communication in Paralysis,” appeared in Nature Neuroscience on September 14, 2026.
Ryan Whalen covers science and technology for The Debrief. He holds an MA in History and a Master of Library and Information Science with a certificate in Data Science. He can be contacted at ryan@thedebrief.org, and follow him on Twitter @mdntwvlf.
Login
