Brain–Computer Interfaces
Speech Decoding
Reconstructing spoken language from cortical activity
It is possible to decode human speech from electrical activity in the brain. This webpage
explores the use of Machine Learning models
to predict human speech — mapping neural recordings from a grid of cortical electrodes,
through a recurrent encoder–decoder network, into the spoken words of a patient. The
ultimate goal of our work is twofold: first and foremost, we seek an elevated understanding of the
human brain and how it functions. To do so, we look to machine learning as a simpler version of this
problem. By comparing these two architectures, we hope to learn more about "learning." Secondly, we
hope to use a predictive model to enable a patient to continue to "speak" if their capacity for speech
has degenerated. Explore several visualizations of the neural networks used to decode speech from electrode
activity!
Model Architecture
Input
Raw ECoG Data
~250 electrodes
Stage 1
Temporal Convolution
12-sample filters
Side Output
Phoneme Predictions
multi-task loss
Stage 2
Encoder 3
400-unit RNN
Stage 2
Encoder 2
400-unit RNN
Stage 2
Encoder 1
400-unit bidirectional RNN
Attention →
Output
Word Prediction
softmax over vocab
Stage 3
Decoder 1
800-unit RNN
Stage 3
Decoder Embedding
150-unit
Input
Predicted word (t−1)
The previous prediction
Overview
Model Architecture
Raw ECoG signals from ~250 cortical electrodes are first passed through a temporal convolution layer that computes a smoothed envelope of the high-frequency neural activity. Three stacked recurrent layers then encode this sequence: a bidirectional 400-unit RNN captures context in both directions, followed by two additional 400-unit RNNs that build progressively abstract representations. An auxiliary phoneme-prediction head on the encoder regularizes training through a multi-task loss.
An attention mechanism connects the encoder stack to an 800-unit recurrent decoder, allowing it to focus on the most relevant encoder time-steps when generating each word. A 150-unit embedding layer encodes the previously predicted word and feeds it as input and a final softmax layer produces a probability distribution over the vocabulary at each step.
Overview
Brain Machine Interfaces (BMI)
The field of Brain Machine Interfacing will see a lot of fascination this century, as research seeks to explain the marvel that is the human brain. As this field advances, our understanding grows and we begin to explore the possibility of controlling technology directly with our minds.
Speech Decoding is a subfield that focuses specifically on interfacing with the synthesis and processing of speech. While we understand little of their actual functionality, we hope to expand our knowledge with research in this area.