Temporally-language-grounding

A Pytorch implemention for some state-of-the-art models for "Temporally language grounding in untrimmed videos"

Requirements

• Python 2.7
• Pytorch 0.4.1
• matplotlib
• The code is for Charades-STA dataset.

Three Models for this task

Supervised Learning based methods

• TALL: Temporal Activity Localization via Language Query
• MAC: MAC: Mining Activity Concepts for Language-based Temporal Localization.

Reinforcement Learning based method

• A2C: Read, Watch, and Move: Reinforcement Learning for Temporally Grounding Natural Language Descriptions in Videos.

Performance

Methods	R@1, IoU0.7	R@1, IoU0.5	R@5, IoU0.7	R@5, IoU0.5
TALL	8.63	24.09	29.33	59.60
MAC	12.31	29.68	37.31	64.14
A2C	14.25	32.66	None	None

Features Download

• visual features
• visual activity concepts (for MAC)
• ref_info
• RL_pickle (for A2C)

Training and Testing

Training and Testing for TALL, run

python main_charades_SL.py --model TALL

Training and Testing for MAC, run

python main_charades_SL.py --model MAC

Training and Testing for A2C, run

python main_charades_RL.py

Acknowledgements

Thanks the original TALL, MAC and awesome PyTorch team.