Does FINN have the potential to deploy a Transformer-based network on an FPGA/PYNQ?

[Mengzou01]

Hi, everyone.
So we might be able to adapt the self-attention unit to the MVTU?
Regarding the model, can MobileViTv3(http://arxiv.org/abs/2209.15159) or binaryBERT:(http://arxiv.org/abs/2012.15701) be used as references?
I'm a beginner with FINN, so please forgive me if I make any ridiculous mistakes.
All possibilities and any help will be welcomed with open arms. Thank you, everyone!
L.

[fpjentzsch]

Hi,

there are a number of open PRs by @iksnagreb in this repository that will add Transformer support to FINN. In our fork, FINN+, we have already added this functionality to the dev branch and are currently in the process of fixing remaining minor issues.

The best documentation we currently have is probably this repository: https://github.com/eki-project/FINN-T
The corresponding paper will be published very soon.

[Mengzou01]

Hi, fpjentzsch.
Thank you so much for the detailed and fresh insight! I really appreciate your help with this.
So excited to hear that Trsansformer is supported! I will try to deploy my own Transformer model and keep an eye on the FINN+ paper at FPT 2024.
L.