Does FINN not like ReLU layers? Help please! (Fixed)

[rothej]

How are we supposed to transform the RELU layers into a form that FINN likes? I am having trouble with the following code (model was exported using export_qonnx):

import finn
from finn.util.visualization import showSrc, showInNetron
from finn.util.basic import make_build_dir
import os
import onnx
file_path = str('../workspace/jh_fpga_amr/src/py/models')
file_name = str('/vgglike_5f_5c_4re_4mp_pr0_quant8.onnx')
onnx_model = onnx.load(file_path + file_name)
showInNetron(file_path + file_name)

At this point, model displays fine. I can see conv, maxpool, relu, and quant layers.

from qonnx.util.cleanup import cleanup_model
from qonnx.core.modelwrapper import ModelWrapper
import os
os.makedirs(os.path.dirname(file_path + '/cleanup' + file_name), exist_ok=True)
export_onnx_path_cleaned = file_path + '/cleanup' + file_name
model = ModelWrapper(onnx_model)
model = cleanup_model(model)
model.save(export_onnx_path_cleaned)
showInNetron(export_onnx_path_cleaned)

Dont notice much of a difference here

from finn.transformation.qonnx.convert_qonnx_to_finn import ConvertQONNXtoFINN
from qonnx.core.modelwrapper import ModelWrapper
model_for_sim = ModelWrapper(export_onnx_path_cleaned)

from qonnx.core.datatype import DataType

finnonnx_in_tensor_name = model_for_sim.graph.input[0].name
finnonnx_out_tensor_name = model_for_sim.graph.output[0].name
print("Input tensor name: %s" % finnonnx_in_tensor_name)
print("Output tensor name: %s" % finnonnx_out_tensor_name)
finnonnx_model_in_shape = model_for_sim.get_tensor_shape(finnonnx_in_tensor_name)
finnonnx_model_out_shape = model_for_sim.get_tensor_shape(finnonnx_out_tensor_name)
print("Input tensor shape: %s" % str(finnonnx_model_in_shape))
print("Output tensor shape: %s" % str(finnonnx_model_out_shape))
finnonnx_model_in_dt = model_for_sim.get_tensor_datatype(finnonnx_in_tensor_name)
finnonnx_model_out_dt = model_for_sim.get_tensor_datatype(finnonnx_out_tensor_name)
print("Input tensor datatype: %s" % str(finnonnx_model_in_dt.name))
print("Output tensor datatype: %s" % str(finnonnx_model_out_dt.name))
print("List of node operator types in the graph: ")
print([x.op_type for x in model_for_sim.graph.node])

model = model_for_sim#.transform(ConvertQONNXtoFINN())

export_onnx_path_converted = file_path + '/conv' + file_name
os.makedirs(os.path.dirname(export_onnx_path_converted), exist_ok=True)
model.save(export_onnx_path_converted)
showInNetron(export_onnx_path_converted)

I can see the shapes look good, and layer names are fine.

from qonnx.transformation.general import GiveReadableTensorNames, GiveUniqueNodeNames, RemoveStaticGraphInputs
from qonnx.transformation.infer_shapes import InferShapes
from qonnx.transformation.infer_datatypes import InferDataTypes
from qonnx.transformation.fold_constants import FoldConstants
from qonnx.transformation.change_3d_tensors_to_4d import Change3DTo4DTensors
import finn.transformation.streamline.absorb as absorb
import finn.transformation.fpgadataflow.convert_to_hw_layers as to_hw
from finn.transformation.qonnx.convert_qonnx_to_finn import ConvertQONNXtoFINN

from qonnx.transformation.base import Transformation
from finn.transformation.qonnx.qonnx_activation_handlers import QuantReluHandler

class ApplyQuantReluHandler(Transformation):
    def apply(self, model):
        graph = model.graph
        node_ind = 0
        graph_modified = False
        while node_ind < len(graph.node):
            n = graph.node[node_ind]
            if n.op_type == "Relu":
                if node_ind + 1 < len(graph.node):
                    quantize_node = graph.node[node_ind + 1]
                    if quantize_node.op_type in ["Quant", "BipolarQuant"]:
                        qrh = QuantReluHandler(model, quantize_node, node_ind + 1)
                        model = qrh.replace_quant_node()
                        graph_modified = True
                        continue
            node_ind += 1
        if graph_modified:
            model = model.transform(InferShapes())
            model = model.transform(InferDataTypes())
        return (model, graph_modified)

model = model.transform(InferShapes())
model = model.transform(FoldConstants())
model = model.transform(GiveUniqueNodeNames())
model = model.transform(GiveReadableTensorNames())
model = model.transform(InferDataTypes())
model = model.transform(RemoveStaticGraphInputs())
model = model.transform(ApplyQuantReluHandler())
model = model.transform(ConvertQONNXtoFINN())
model = model.transform(Change3DTo4DTensors())
model = model.transform(absorb.AbsorbScalarMulAddIntoTopK())
model = model.transform(to_hw.InferChannelwiseLinearLayer())
model = model.transform(to_hw.InferLabelSelectLayer())
model = model.transform(GiveUniqueNodeNames())

os.makedirs(os.path.dirname(file_path + '/verif' + file_name), exist_ok=True)
verif_model_filename = file_path + '/verif' + file_name
model.save(verif_model_filename)
showInNetron(verif_model_filename)

Heres the part Ive been playing with and trying to debug. I know i need to use 3d->4d since the tensors on this model are 1d which FINN by default does not like. However! It does not like Relu.

The QuantReluHandler transform is a FINN function, but I think it needs to be called from a model.transform - however it wants nodes and node indexes as inputs, so I was throwing together something custom. It (meaning my current code) is, at this current time, garbage. I am pretty sure I am just doing the Relu handling wrong, but I cant find an example that does it properly.

Does anyone know of one? Change3Dto4DTensors will yell at me if I try to run a model with RELU layers through it, so I was trying to fix prior. Ive tried different orders of qonntofinn as well (it used to be much earlier in the code).

Below is the warning I get which is likely what causes build to fail.

.../change_3d_tensors_to_4d.py:93: UserWarning: Transformation is not applied,                 found unsupported nodes in the graph: ['Relu', 'Relu', 'Relu', 'Relu', 'Relu', 'Relu', 'Relu', 'Relu', 'Relu'].
  warnings.warn(

[Phu1234567]

I believe the Relu nodes should no longer be in the graph after you converting from ONNX to FINN (ConvertQONNXtoFINN).

Im also working on a 3D model, and before I convert from 3D to 4D, my 'tidied up' model no longer have the RELU node, but instead replaced by a lower operation nodes. Heres my example intermediate models:

After exporting my model to brevitas qonnx:

Then, transform(ConvertQONNXtoFINN):

Notice how the RELU between the BatchNorm and MaxPool nodes is no longer there. This all happened before I convert to 4D.

[rothej]

I've been barking up the wrong tree (suspected as much) because my conversion isn't working as expected. I figured the ReLU conversion should be handled internally.

import finn
from finn.util.visualization import showSrc, showInNetron
from finn.util.basic import make_build_dir
import os
import onnx
file_path = str('../workspace/jh_fpga_amr/src/py/models')
file_name = str('/vgglike_5f_5c_4mp_pr0_quant8.onnx')
onnx_model = onnx.load(file_path + file_name)
showInNetron(file_path + file_name)

from qonnx.util.cleanup import cleanup

os.makedirs(os.path.dirname(file_path + '/cleanup' + file_name), exist_ok=True)
export_onnx_path_cleaned = file_path + '/cleanup' + file_name
cleanup(file_path + file_name, out_file=export_onnx_path_cleaned)
showInNetron(export_onnx_path_cleaned)

from finn.transformation.qonnx.convert_qonnx_to_finn import ConvertQONNXtoFINN
from qonnx.core.modelwrapper import ModelWrapper

model = ModelWrapper(export_onnx_path_cleaned)

model = model.transform(ConvertQONNXtoFINN())

export_onnx_path_converted = file_path + '/conv' + file_name
os.makedirs(os.path.dirname(export_onnx_path_converted), exist_ok=True)
model.save(export_onnx_path_converted)
showInNetron(export_onnx_path_converted)

I am not sure why the RELU layer is still there, but I guess it must be because I am exporting the onnx model with some features that the transform doesn't like? Maybe I shouldn't be exporting qonnx in my code and instead importing the pytorch/brevitas model and converting it to qonnx within the FINN docker. I'll give that a shot.

[rothej]

So, fixed this issue in model gen by using QuantReLU from brevitas instead of relu from pytorch. Now everything flows just fine. Commenting so others know what my fix was if they search for this issue