Lightweight Motion Forecasting

A lightweight graph-convolutional model for skeletal human motion forecasting on the Human3.6M (H3.6M) dataset. The paper is available here: https://www.esann.org/sites/default/files/proceedings/2021/ES2021-145.pdf

Setup

• Install the python libraries pip install -r requirements.txt (This file contains the GPU libs for tensorflow and tensorflow_graphics, remove '-gpu' to use the cpu versions)

Usage

• Get the H3.6M Dataset
• The CLI is located in main.py, it consists of two subprograms train and eval for training and evaluation of models, respectively.
• Calling python main.py --help prints an overview of the CLI arguments

To train a model, call

python main.py train

This will train a model with the default configuration (s. configs.py) To evaluate a model, call

python main.py eval --checkpoint <path_to_checkpoint>

This will run the default evaluation on a model with the default configuration (s. configs.py), restored from the checkpoint thats passed in path_to_checkpoint A checkpoint to run the model with default configuration is located in the ckpts folder Alternatively, you can alter the defaults by passing additional cli arguments or directly modify the configs.py file.

Model Architecture

• The model is based on Graph-WaveNet, a spatio-temporal extension to the original WaveNet.
• It consists of N identical blocks that consist of two different layers
  • Spatio-temporal convolution (ST-Conv; Implementation)
    • ST-Conv replaces the 1D convolution in the original Graph-WaveNet
  • Graph convolution that respects joint hierarchy (K-GCN; Implementation)
    • K-GCN replaces the diffusion GCN in the original Graph-WaveNet
• WaveNet-style skip connections accumulate the outputs of the blocks
• A ReLU-activated MLP computes the final output
• This is an autoregressive model, hence it computes 1-step predictions that are the input to the model for the next prediction step.

Qualitative Results

Prediction and ground truth from the test set performing the walking action.

---

Prediction (solid) and ground truth (dashed) individual quaternion dimensions.

Cite: BibTex for this work :)

@inproceedings{LightGNN4HumanMotion2021,
  title={Application of Graph Convolutions in a Lightweight Model for Skeletal Human Motion Forecasting},
  author={Hermes, Luca and Hammer, Barbara and Schilling, Malte},
  url={https://www.esann.org/sites/default/files/proceedings/2021/ES2021-145.pdf},
  year={2021},
  booktitle={European Symposium on Artificial Neural Networks (ESANN)},
  pages={111-116}
}