Reproducing the results on ActivityNet

[tobbysong]

Hi @Pilhyeon, thanks for your great work! Now I'm following your BMUE and have some trouble in reproducing the results on ActivityNet dataset.

I have tried to do some experiments on ActivityNet v1.2. I downloaded the I3D features provided by this link and adapt them to BMUE format. The following are some of my results. All experiments run for 6k epochs, the results are showed in the form of "(average_mAP, Test_acc)":

• [no params changed] (0.0390, 0.43)

According to Sec 4.1 in your arXiv paper, T=50, so I set num_segments as 50 and do the following experiments:

• [num_segments: 50] (0.0275, 0.31)
  • [class_th: 0.1] (0.0332, 0.36)
  • [class_th: 0.1, act_thresh_cas: np.arange(0.0, 0.15, 0.015)] (0.0400, 0.36)

Besides, I also tried to change "act_thresh_magnitudes", "NMS thresh", "alpha", "_lambda & gamma in get_proposal_oic()", etc. The results don't seem to get better: the test accuracy is around 0.4 and the average_mAP is very low. It's hard for me to find the best settings. Could you please share your params settings on ActivityNet 1.2 & 1.3 datasets? Or give me some advice on which params to change?

Looking forward to your reply and I'd be glad to cite your excellent work. Thanks!

[Pilhyeon]

First of all, I apologize for the missing details about the experimental settings on ActivityNet.
Since I extracted the features for ActivityNet by myself, they might be different from those you used.
For example, the sampling rate of the videos or the pre-trained I3D models could be different.
I used the original fps of the videos and the pre-trained models from this repo.
For the hyperparameters, the batch size, the margin, alpha, (r_act, r_bkg), and the NMS threshold are set to 512, 250, 1e-4, (5, 50), and 0.8, respectively.
I hope this would help.

[xumh-9]

Can you share the feature of act1.2?

[cecilia930426]

Thanks for your wonderful work. Could you please share your features in Activitynet 1.3?

[P-DX]

Hi, have you got the better result after you set the hyperparameters as mentioned?
Thanks.

[sauradip]

Hi @Pilhyeon, thanks for your great work! Now I'm following your BMUE and have some trouble in reproducing the results on ActivityNet dataset.

I have tried to do some experiments on ActivityNet v1.2. I downloaded the I3D features provided by this link and adapt them to BMUE format. The following are some of my results. All experiments run for 6k epochs, the results are showed in the form of "(average_mAP, Test_acc)":

• [no params changed] (0.0390, 0.43)

According to Sec 4.1 in your arXiv paper, T=50, so I set num_segments as 50 and do the following experiments:

• [num_segments: 50] (0.0275, 0.31)

  • [class_th: 0.1] (0.0332, 0.36)
  • [class_th: 0.1, act_thresh_cas: np.arange(0.0, 0.15, 0.015)] (0.0400, 0.36)

Besides, I also tried to change "act_thresh_magnitudes", "NMS thresh", "alpha", "_lambda & gamma in get_proposal_oic()", etc. The results don't seem to get better: the test accuracy is around 0.4 and the average_mAP is very low. It's hard for me to find the best settings. Could you please share your params settings on ActivityNet 1.2 & 1.3 datasets? Or give me some advice on which params to change?

Looking forward to your reply and I'd be glad to cite your excellent work. Thanks!

Hi toby ,

Can you kindly share your version of the repo for ActivityNet dataset in this email : sauradipnag95@gmail.com if possible ?

[liming-ai]

Hi @Pilhyeon, thanks for your great work! Now I'm following your BMUE and have some trouble in reproducing the results on ActivityNet dataset.

I have tried to do some experiments on ActivityNet v1.2. I downloaded the I3D features provided by this link and adapt them to BMUE format. The following are some of my results. All experiments run for 6k epochs, the results are showed in the form of "(average_mAP, Test_acc)":

• [no params changed] (0.0390, 0.43)

According to Sec 4.1 in your arXiv paper, T=50, so I set num_segments as 50 and do the following experiments:

• [num_segments: 50] (0.0275, 0.31)

  • [class_th: 0.1] (0.0332, 0.36)
  • [class_th: 0.1, act_thresh_cas: np.arange(0.0, 0.15, 0.015)] (0.0400, 0.36)

Besides, I also tried to change "act_thresh_magnitudes", "NMS thresh", "alpha", "_lambda & gamma in get_proposal_oic()", etc. The results don't seem to get better: the test accuracy is around 0.4 and the average_mAP is very low. It's hard for me to find the best settings. Could you please share your params settings on ActivityNet 1.2 & 1.3 datasets? Or give me some advice on which params to change?

Looking forward to your reply and I'd be glad to cite your excellent work. Thanks!

hi @tobbysong, have you got the better result after you set the hyperparameters as mentioned?

[kiwiJ]

@xumh-9 @cecilia930426 @sauradip
ActivityNet1.2 from W-TALC github.
ActivityNet1.3 from A2CL-PT github.

[hawklucky]

how about the settings of 'class_th', 'act_thresh_cas', 'act_thresh_magnitudes'?
Could you provide a complete configure file for ActivityNet1.2 and ActivityNet1.3?
That will very helpful. @Pilhyeon

[liming-ai]

how about the settings of 'class_th', 'act_thresh_cas', 'act_thresh_magnitudes'?
Could you provide a complete configure file for ActivityNet1.2 and ActivityNet1.3?
That will very helpful.@kiwiJ

Hi, could you reproduce the result of 41.8 mAP about THUMOS14 in paper without their pre-trained model?

[hawklucky]

how about the settings of 'class_th', 'act_thresh_cas', 'act_thresh_magnitudes'?
Could you provide a complete configure file for ActivityNet1.2 and ActivityNet1.3?
That will very helpful.@kiwiJ

Hi, could you reproduce the result of 41.8 mAP about THUMOS14 in paper without their pre-trained model?

Unfortunately, it's only around 40.0 mAP

[liming-ai]

how about the settings of 'class_th', 'act_thresh_cas', 'act_thresh_magnitudes'?
Could you provide a complete configure file for ActivityNet1.2 and ActivityNet1.3?
That will very helpful.@kiwiJ

Hi, could you reproduce the result of 41.8 mAP about THUMOS14 in paper without their pre-trained model?

Unfortunately, it's only around 40.0 mAP

The best result I got is 40.2, I sent email to author, but he cannot provide any training log and told me the random seed is unknown for the best result...I have tried many random seed but it did not work...

[Richard-61]

how about the settings of 'class_th', 'act_thresh_cas', 'act_thresh_magnitudes'?
Could you provide a complete configure file for ActivityNet1.2 and ActivityNet1.3?
That will very helpful.@kiwiJ

Hi, could you reproduce the result of 41.8 mAP about THUMOS14 in paper without their pre-trained model?

Unfortunately, it's only around 40.0 mAP

The best result I got is 40.2, I sent email to author, but he cannot provide any training log and told me the random seed is unknown for the best result...I have tried many random seed but it did not work...

Me too. I run the code and only get the result of 40.2 mAP about THUMOS14.

[Jockey721]

how about the settings of 'class_th', 'act_thresh_cas', 'act_thresh_magnitudes'?
Could you provide a complete configure file for ActivityNet1.2 and ActivityNet1.3?
That will very helpful.@kiwiJ

Hi, could you reproduce the result of 41.8 mAP about THUMOS14 in paper without their pre-trained model?

Unfortunately, it's only around 40.0 mAP

The best result I got is 40.2, I sent email to author, but he cannot provide any training log and told me the random seed is unknown for the best result...I have tried many random seed but it did not work...

Me too. I run the code and only get the result of 40.2 mAP about THUMOS14.

Me too. Has anyone reproduced the reported performance?