Discussion on training issues I have encountered

[zmy1116]

Thank you for the implementation for the paper. This is the first time I'm dealing with transformer model, I tried to train over Kinetics700 dataset using this model. and I just want to share some of the issues I have encountered:

The paper suggested that the model works better with pretrained weights. Although this is a direct extension from image transformer, most of vision transformer's weights should apply directly, there are 2 places that are different:

1. Positional encoding: we have H x W x T instead of HxW. so I copied the same positional encoding for every frame, sort of like how we inflate imagenet weights on I3D without dividing by T. One alternative way I'm thinking is use angular initialization to generate an 1XT positional encoding and then add to the HxW image positional encoding to form the HxWxT positional encoding.
2. We are doing two self-attentions now instead of one per block, so there are double amount of weights for qkv and output fcs. For now per block I use the same weights for the first and second self-attention if I use the same number of heads as the pretrained image model. Alternatively, in a different model I have half number of heads so the time attention and spatial attention will each use half of the heads weights.

Since it is the first time I'm dealing with Transformer, I want to reproduce what the paper claimed so I started with the "original" basic vision transformer setup:

• 12 heads 12 blocks
• GELU instead of GEGLU
• embedding size 768
• Image size 224, divide to 16x16 patches

With this setup, on a V100 GPU we can only squeeze in 4 videos (4x8x3x224x224) for training even with torch.amp , this means if I'm doing an experiment on an p3x8 machine with 4 V100 gpus (~ 12$/h normally), it would take 39 days to do 300 epochs. Of course it may not need to train for 300 epochs, but intuitively, training with batch size = 16 is not usually not optimal.

So alternatively, I tried a new model with 6 heads and 8 blocks, Now I can put 16 videos per GPU, so in total batch size = 64. The model started to train smoothly then training error increases after 7-8 epochs. The training accuracy peaked around 55% and I didn't even bother to run validation because I know clearly it's not working. Below list the relevant configuration I was using.

DATA:
  NUM_FRAMES: 8
  SAMPLING_RATE: 16
  TRAIN_JITTER_SCALES: [256, 320]
  TRAIN_CROP_SIZE: 224
  # TEST_CROP_SIZE: 224 # use if TEST.NUM_SPATIAL_CROPS: 1
  TEST_CROP_SIZE: 224 # use if TEST.NUM_SPATIAL_CROPS: 3
  INPUT_CHANNEL_NUM: [3]
  DECODING_BACKEND: torchvision
  MEAN: [0.5, 0.5, 0.5]
  STD: [0.5, 0.5, 0.5]
  WEIGHT_DECAY: 0.0
SOLVER:
  BASE_LR: 0.1 # 1 machine
  BASE_LR_SCALE_NUM_SHARDS: True
  LR_POLICY: cosine
  MAX_EPOCH: 300
  WEIGHT_DECAY: 5e-5
  WARMUP_EPOCHS: 35.0
  WARMUP_START_LR: 0.01
  OPTIMIZING_METHOD: sgd
TRANSFORMER:
  TOKEN_DIM: 768
  PATCH_SIZE: 16
  DEPTH: 8
  HEADS: 6
  HEAD_DIM: 64
  FF_DROPOUT: 0.1
  ATTN_DROPOUT: 0.0

So these are the issues are I have encountered for now. I want to share these because hopefully some of you are actually working with video model and maybe we can have a discussion. I think probably my next thing to try is to increase number of depth.

Regards

[zmy1116]

The original paper said they use pre-trained ViT, and I think training on K400 or K700 without pre-training weights is not a good choice. I also want to reproduce the results of this work, but many details I can't handle.

I did use pretrained weight from ViT

[Hanqer]

Hi, I also meet the question about training acc. I train model using pertrained weight from ViT, but the acc is lower than a baseline model (only ViT backbone, without time attention). I train on 4V100 GPU, so the batch size can be set as 48=32. I randomly initialize the weight of model that not contained in ViT (HWT position embeding, time attention).

[Tonyfy]

I got lower train/val performance too.

[RaivoKoot]

@zmy1116 , @Hanqer
Can I ask how you two got the pretrained ImageNet weights from ViT? I am training this repo's implementation for a dataset of mine too, but as far as I can tell, there are no pretrained weights available for this repo. Am I wrong?

[zmy1116]

@zmy1116 , @Hanqer
Can I ask how you two got the pretrained ImageNet weights from ViT? I am training this repo's implementation for a dataset of mine too, but as far as I can tell, there are no pretrained weights available for this repo. Am I wrong?

https://github.com/google-research/vision_transformer

[mckinziebrandon]

@zmy1116 perhaps you could share/PR the code you used for initializing with pre-trained ViT?

[zmy1116]

@zmy1116 perhaps you could share/PR the code you used for initializing with pre-trained ViT?

https://arxiv.org/pdf/2103.15691.pdf
This is the google's similar paper, their model 3 is exactly TimeSformer
initialize the weights based on section 3.4.

[mckinziebrandon]

Yes, I understand that. However, initializing TimeSformer with ViT weights does require some additional code, primarily for mapping the pre-trained weight names to the TimeSformer weight names. If you've already done this, it would be nice of you to share so we can avoid reinventing the wheel.

I'm doing it right now, and will post what I do if you don't want to share.

[zmy1116]

Yes, I understand that. However, initializing TimeSformer with ViT weights does require some additional code, primarily for mapping the pre-trained weight names to the TimeSformer weight names. If you've already done this, it would be nice of you to share so we can avoid reinventing the wheel.

I'm doing it right now, and will post what I do if you don't want to share.

please do

[mckinziebrandon]

Hm, posts an issue requesting help from others on an open-source project, but won't contribute/share code themselves. Fascinating!

[zmy1116]

Hm, posts an issue requesting help from others on an open-source project, but won't contribute/share code themselves. Fascinating!

won't contribute seems harsh, I did share some of my findings, I shared where the original VIT weights are and I did read multiple papers and shared the google paper and showed the section on weights initialization. I would think these count something. I would like to think I did provide more information than most of people who posts issues in this repo.

[RaivoKoot]

I'm doing it right now, and will post what I do if you don't want to share.

@mckinziebrandon
That would be great, thanks! I’ve also found a similar implementation which says it includes pretrained weights, but i have not yet tested it myself to see if it works as expected (planning to soon) (https://github.com/m-bain/video-transformers)

[justinrmiller]

I'm still very new to the ML space, but would like to contribute as well. I'd be happy to contribute some documentation if we could hop on a Discord/Slack chat and discuss. Really interested in seeing how this model performs and how I can apply it to some videos. Thanks!

[zmy1116]

@Hanqer @Tonyfy

With multiple rounds of changes and testing, I am able to reproduce similar (not better) result on Kinetics700_2020 with video transformer.

I did the following:

• based on google's paper https://arxiv.org/pdf/2103.15691.pdf, I implemented their model 2, their model 3 is exactly TimeSformer and the model 2 use spatial attention and time attention into 2 separate stages. They claim that doing this is better , also it is now a smaller model, we can fit a batch of 64 with 8 v100. The model modification is very straight forward based on this repo.

• It is important to find good learning rate and I have to do learning rate warm up, otherwise training diverges from very beginning, for kinetics I use the following

SOLVER:
 BASE_LR: 0.05 # 1 machine
 BASE_LR_SCALE_NUM_SHARDS: True
 LR_POLICY: cosine
 MAX_EPOCH: 30
 WEIGHT_DECAY: 5e-5
 WARMUP_EPOCHS: 1.0
 WARMUP_START_LR: 0.01
 OPTIMIZING_METHOD: sgd

• I included color jittering in augmentation, not sure how much this helped

After 30 epochs I'm getting ~62% accuracy on kinetics700_2020 with multi-views. My best model (with X3D-M) on this dataset was ~63.4% with multi-views. I don't think it's good but I don't see any results for this dataset online. The closest public model I could find on this is from SenseTime's lab on K700, they get 64% with multi-views.

So I would say with video transformer i can get a reasonable model , and the training time is around 30h on an 8GPU machine, which I find very interesting.

[Hanqer]

@zmy1116 Thanks for sharing the training configuration. I do the same thing that finding a good learning rate with warm up. In my setting, I use lr=0.01 and warm up for 2 epochs, model 3 (TimeSformer). But in Kinetics400, I can't get reasonable results. I will test the configuration you shared.
By the way, is your model using 8 frames sampling with sampling rate x32? Have you trained on Kinetics400?