Dealing with ConvTranspose Layers while streamlining and converting to hw?

[Mounice97]

Is there a specific way to do the streamlining for ConvTranspose layers on an autoencoder while working with FINN? I am currently working with an autoencoder for the purpose of reducing noise in images. At the moment, it seems that brevitas has the possibility to work with QuantConvTranspose layers and train with them. However I have faced the problem that while working with FINN, there might be some poblems in the export that might not allow me to do the complete streamlining of the file along with the conversion to hw layers. Since Im2Col only works with regular convolutional layers and later these can be easily converted into ConvolutionalInputGenerators. Has anyone had any experience on how to deal with this kind of layers?
I have heard on other forums about the possibility of turning the problem around by working with upsampling+Conv layers to substitute ConvTranspose layers, but has there been any new update that would avoid this procedure?

[fpjentzsch]

Maybe @hleblevec can help you, he contributed this PR for supporting transpsed convolutions: #835

I believe a more efficient implementation with a modified ConvolutionInputGenerator instead of upsampling/padding is also on the roadmap.

[hleblevec]

Hi @Mounice97 and thanks @fpjentzsch for tagging me.
For the moment there is a support of transposed convolutions implemented as a fractionnaly-strided convolution. The streamlining and convertion to hw layers can be done through the InferPixelPaddingDeconv transformation. You can find here an example of using this transformation in a custom conversion to hw layers step: https://github.com/Xilinx/finn-examples/blob/7a672f89553882854fedb0c864272ff8f0f9975d/build/espcn/custom_steps.py#L134

It works, but it's not very hardware efficient. A more efficient implementation is currently being developped, but you can use the current implementation in the meantime. Please don't hesitate to ask for help if you encounter issues with using the transformation.

[Mounice97]

I have changed to the dev branch, and I'm still getting the same error for the missing connection with some clock pins. Thus, I have tried to open the created file with Vivado to make the connections manually. However, is this supposed to be the way to address this problem, or is there a way inside of FINN's functions to solve this without going directly to Vivado?

[fpjentzsch]

The error referred to the clock pins of "system_ila_*". I wonder why you have ILAs in your design. Did you enable enable_hw_debug in your build config? Maybe that feature is buggy and you can simply disable the ILAs.

[Mounice97]

Thanks, that seem to solve the issue, indeed, that feature seems to be buggy. Now, after adjusting some characteristics on the folding factors and the ram styles on certain nodes, I can now create a build. Just one more question, I am using the 'KV260_SOM' from the supported PYNQ boards in FINN. However, I am currently using the Kria KR260, and if I open the project in Vivado, then the project part is obviously for the Kria KV260. I can change it to the Kria KR260 and then regenerate the bitfiles with the corrected IPs. However, should I run again the MakePYNQDriver function after making such changes, or would this not work? In principle, how should I proceed to address my FPGA correctly?

[fpjentzsch]

I don't think you'll need to run MakePYNQDriver again in this situation.

You can see here what changes need to be made to add native support for a new ZYNQ device (KV260 as example): https://github.com/Xilinx/finn/pull/692/files

[Mounice97]

Thank you very much, @fpjentzsch. Now I have been able to complete the hardware build and deployment. Now It's just a matter of refining some details to reduce the number of clock cycles for my project.