Artificial intelligence turns brain activity into speech

8/29/2019

Artificial intelligence turns brain activity into speech

Arti cial intelligence turns brain activity into speech usm systems Aug 29 · 4 min read

Artificial intelligence turns brain activity into speech for most people who are frozen and unable to speak, the signals for what they want to say hide in their brains. No one could interpret those signs directly. But three research groups have recently made progress in transforming data from electrodes placed on the brain into computer-generated speech.

Using computational models called neural networks, they have, in some cases, reconstructed words and sentences that are understandable to human listeners.

None of the efforts described in recent months papers on the PrintPrint Server BioReceive, people have just been unable to recreate the expected speech. Instead, the researchers monitored parts of the brain that people read aloud, quietly, or listen to recordings. But it is understandable to show the reconstructed speech is “definitely exciting,” said Stephanie Martin, a neural engineer at the University of Geneva, Switzerland, who was not involved in the new projects.

People who have lost their ability to speak after a stroke or disease can use their eyes or make other small movements to control the cursor or select letters on the screen. If the brain-computer interface can directly recreate their speech, they can retrieve it again: control over tone and image, for example, or the ability to interfere with fast-moving speech.

The barriers are high. “We are trying to create a pattern of neurons that turn on and off at different time points and inflate speech sound,” says Nima Mesgarani, a computer scientist at Columbia University. “Mapping from one to the next is not very straightforward.” How these codes translate into speech sounds varies from person to person, so computer models must be “trained” on each person. And models do best with very accurate data, which requires skull opening. https://medium.com/@usmsystems23/artificial-intelligence-turns-brain-activity-into-speech-53bd82b52abb

1/3

8/29/2019

Artificial intelligence turns brain activity into speech

Researchers can make such invasive recording only in rare cases. When one removes a brain tumor, electrical readouts from the exposed brain help surgeons identify and avoid key speech and motor areas. Another is when a person with epilepsy is implanted with electrodes for several days to determine the source of the seizure before surgery. “We have a maximum of 20 minutes, maybe 30,” for data collection, Martin says. “We are limited.”

The groups behind the new papers produce highly valuable data by putting information into neural networks that process complex patterns by passing information through layers of computational “nodes”. Networks learn by adjusting the connections between nodes. In the experiments, the networks were subjected to recordings of speech produced and heard by a person and simultaneous brain activity data.

The Mesgarani team relied on data from five people with epilepsy. Their network analyzed recordings from the auditory cortex because they heard recordings of stories and people naming numbers from zero to nine. The computer then reconstructs the spoken numbers from the neural data alone; When “talking” computer numbers, a group of listeners named them with 75% accuracy.

Another group, led by computer scientist Tanja Schultz at University Bremen, Germany, is based on data from six people who underwent brain tumor surgery. The microphone captures their voices while reading single-letter words aloud. Meanwhile, electrodes are recorded from the brain’s speech planning areas and motor areas, which send commands to the vocal tract to pronounce words. Computer scientists Miguel Angrick and Christian Herf, now at Maastricht University, trained a network that mapped electrode readouts to audio recordings and then reconstructed words from invisible brain data. According to the computerized scoring system, 40% of computer-generated words are intelligible.

Computer audio of each word based on actual audio from the study participants, followed by speech planning and activity in motor areas of the brain.

Finally, neurosurgeon Edward Chang and his team at the University of California, San Francisco reconstructed whole sentences from speech and motor activity extracted from the brain, and three epilepsy patients read aloud. In the online test, 166 people heard one of the sentences and had to choose from 10 written choices. Some sentences are https://medium.com/@usmsystems23/artificial-intelligence-turns-brain-activity-into-speech-53bd82b52abb

2/3

8/29/2019

Artificial intelligence turns brain activity into speech

correctly identified by more than 80% of the time. Researchers have pushed this paradigm further: They used it to recreate sentences from recorded data when people quietly cried words. This is an important result, says Herff, “one step closer to the speech prosthesis we all have in mind.”

However, “What we’re waiting for is how [these methods] are going to do when the patients can’t speak,” says Stephanie Riès, a neuroscientist at San Diego State University in California who studies language production. The brain signals when a person silently “speaks” or “hears” their voice in their head aren’t identical to signals of speech or hearing. Without external sound to match brain activity, it may be hard for a computer even to sort out where inner speech starts and ends.

Decoding imagined speech will require “a huge jump,” says Gerwin Schalk, a neuro engineers at the National Center for Adaptive Neurotechnologies at the New York State Department of Health in Albany. “It’s unclear how to do that at all.”

One approach, Herff says, might be to give feedback to the user of the brain-computer interface: If they can hear the computer’s speech interpretation in real-time, they may be able to adjust their thoughts to get the result they want. With enough training of both users and neural networks, brain and computer might meet in the middle.

Arti cial Intelligence

Machine Learning

Ai Services

Ai Solution

Ml Services

About

https://medium.com/@usmsystems23/artificial-intelligence-turns-brain-activity-into-speech-53bd82b52abb

Help

Legal

3/3