Decoding of the speech envelope using the VLAAI deep neural network.

.

This repository contains a pre-trained subject-independent model that can decode the speech envelope from EEG signals. The model was presented in the paper:

Decoding of the speech envelope using the VLAAI deep neural network

by Bernd Accou, Jonas Vanthornhout, Hugo Van hamme, and Tom Francart.

Using the VLAAI network

Code for building the VLAAI network is available in the model.py file. This code is written in TensorFlow, using version v2.3.0 and Python 3.6. You can use it as follows:

# Import the model
from model import vlaai
# Instantiate the model
decoder = vlaai()

## If you want to compile the model (for training), you can do so as follow
# s:
# Import the loss and metric function
from model import pearson_loss, pearson_metric
# Compile the model
decoder.compile(optimizer='adam', loss=pearson_loss, metrics=[pearson_metric])

Pre-trained model versions (using the preprocessing and dataset ( single-speaker stories dataset, 80 subjects that listened to 1 hour and 46 minutes on average for a total of 144 hours of EEG data) in the paper) are available in the pretrained_models (see also this document for more information).

Three formats are provided:

1. TensorFlow SavedModel format (pretrained_model/vlaai)
2. HDF5 format (pretrained_model/vlaai.h5)
3. ONNX format (pretrained_model/vlaai.onnx)

You can choose to load the full model, including optimizer state, as follows:

import tensorflow as tf
# Import the loss and metric function
from model import pearson_loss, pearson_metric

# Load the model from TensorFlow SavedModel format
decoder = tf.keras.models.load_model(
    'pretrained_models/vlaai', 
    custom_objects={
        'pearson_loss': pearson_loss, 
        'pearson_metric': pearson_metric
    }
)

# Or, load the model from HDF5 format
decoder = tf.keras.models.load_model(
    'pretrained_models/vlaai.h5', 
    custom_objects={
        'pearson_loss': pearson_loss, 
        'pearson_metric': pearson_metric
    }
)

It is also possible to just load the weights from the HDF file:

# Import the model
from model import vlaai
decoder = vlaai()
# Load the weights
decoder.load_weights('pretrained_models/vlaai.h5')

Running the examples

The examples directory contains code for evaluating the VLAAI network in Jupyter notebook format. These notebooks can be run on Google Colab using their github integration

The example notebook where the pre-trained VLAAI network is evaluated on the DTU dataset by Fuglsang et al. can be executed on Google Colab using the following link:

.