|
1 | 1 | # PyTorch implementation of Deformable Convolution |
2 | 2 | * By Wei OUYANG @ Institut Pasteur |
3 | | - * The original implementation in Keras/TensorFlow: https://github.com/felixlaumon/deform-conv |
| 3 | + * Thanks to Felix Lau's Keras/TensorFlow implementation: https://github.com/felixlaumon/deform-conv |
4 | 4 |
|
5 | 5 | ### TODO List |
6 | 6 | - [x] implement offsets mapping in pytorch |
|
13 | 13 | - [ ] support input image with different width and height |
14 | 14 | - [ ] benchmark with tensorflow implementation |
15 | 15 |
|
16 | | -## Understanding Deformable Convolution |
| 16 | +## Deformable Convolutional Networks |
17 | 17 | > Dai, Jifeng, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen |
18 | 18 | Wei. 2017. “Deformable Convolutional Networks.” arXiv [cs.CV]. arXiv. |
19 | 19 | http://arxiv.org/abs/1703.06211 |
20 | 20 |
|
21 | | - |
22 | | - |
23 | | -Check out |
24 | | -https://medium.com/@phelixlau/notes-on-deformable-convolutional-networks-baaabbc11cf3 |
25 | | -for my summary of the paper. |
26 | | - |
27 | | -## Experiment on MNIST and Scaled Data Augmentation |
28 | | - |
29 | | -To demonstrate the effectiveness of deformable convolution with scaled images, |
30 | | -we show that by simply replacing regular convolution with deformable convolution |
31 | | -and fine-tuning just the offsets with a scale-augmented datasets, deformable CNN |
32 | | -performs significantly better than regular CNN on the scaled MNIST dataset. This |
33 | | -indicates that deformable convolution is able to more effectively utilize |
34 | | -already learned feature map to represent geometric distortion. |
35 | | - |
36 | | -First, we train a 4-layer CNN with regular convolution on MNIST without any data |
37 | | -augmentation. Then we replace all regular convolution layers with deformable |
38 | | -convolution layers and freeze the weights of all layers except the newly added |
39 | | -convolution layers responsible for learning the offsets. This model is then |
40 | | -fine-tuned on the scale-augmented MNIST dataset. |
| 21 | +The following animation is generated by Felix Lau (with his tensorflow implementation): |
41 | 22 |
|
42 | | -In this set up, the deformable CNN is forced to make better use of the learned |
43 | | -feature map by only changing its receptive field. |
44 | | - |
45 | | -Note that the deformable CNN did not receive additional supervision other than |
46 | | -the labels and is trained with cross-entropy just like the regular CNN. |
47 | | - |
48 | | -| Test Accuracy | Regular CNN | Deformable CNN | |
49 | | -| --- | --- | --- | |
50 | | -| Regular MNIST | 98.74% | 97.27% | |
51 | | -| Scaled MNIST | 57.01% | 92.55% | |
52 | | - |
53 | | -Please refer to `scripts/scaled_mnist.py` for reproducing this result. |
54 | | - |
55 | | -## Notes on Implementation |
| 23 | + |
56 | 24 |
|
57 | | -- This implementation is not efficient. In fact a forward pass with deformed |
58 | | - convolution takes 260 ms, while regular convolution takes only 10 ms. Also, |
59 | | - GPU average utilization is only around 10%. |
60 | | -- This implementation also does not take advantage of the fact that offsets and |
61 | | - the input have similar shape (in `tf_batch_map_offsets`). (So STN-style |
62 | | - bilinear sampling will help) |
63 | | -- The TensorFlow Keras backend must be used (channel-last) |
64 | | -- You can check ensure the TensorFlow implementation is equivalent to its scipy |
65 | | - counterpart by running unit tests (e.g. `py.test -x -v --pdb`) |
| 25 | +Also Check out Felix Lau's summary of the paper: https://medium.com/@phelixlau/notes-on-deformable-convolutional-networks-baaabbc11cf3 |
0 commit comments