PSL-GAN

Generative Adversarial Graph Convolutional Networks for Peruvian Sign Language Synthesis.

System requirements

• Python 3.8.11

• Create a new environment using conda create -y -n psl-gan python=3.8.11

• N_CUDA environment variable is defined to choose the GPU in case of having more than one

  pip3 install -r requirements.txt

Dataset

This repository contains the implementantion of Kinetic GAN applied to Peruvian Sign Language. It is used as a data augmentation method.

The Peruvian Sign Language dataset was created by https://github.com/gissemari/PeruvianSignLanguage. This folder contains the following data:

• segmented_signs.zip: zip file with the segmented signs in MP4 format
• raw_data_cocopose.json: located in zip file. Contains the landmarks got by COCO-WholeBody pose estimation model
• raw_data_mediapipe.json: Contains the landmarks got by MediaPipe pose estimation model (body and hands)

To generate the pickle file with the shape of N x C x T x V, where N is the number of samples, C the number of coordinates, T the number of frames, and V the number of joints, run the following code

• 27 number of joints

python3 create_dataset.py --minframes <MIN_FRAMES> --mininstances <MIN_FRAMES>  --leftHandLandmarks --rightHandLandmarks --rawCocoDataset <PATH_JSON_COCOPOSE> --inputPath <PATH_SEGMENTED_SIGNS> --nLandmarks 27 --useCoco --rawDataset <PATH_JSON_MEDIAPIPE>

• 24 number of joints

python3 create_dataset.py --minframes <MIN_FRAMES> --mininstances <MIN_FRAMES>  --leftHandLandmarks --rightHandLandmarks --rawCocoDataset <PATH_JSON_COCOPOSE> --inputPath <PATH_SEGMENTED_SIGNS> --nLandmarks 24 --useCoco --rawDataset <PATH_JSON_MEDIAPIPE> --addExtraJoint

Training the Neural Network

1. Run the following code for training without wandb tracking.

python3 train.py