Cannot reshape array error on the last Dense Layer

[MatteoAbrate]

Hi to everyone,
I'm an Italian astrophysics researcher and i started working with FINN last year.
Following the tutorials i managed to build a Bitfile for implementing a pretty small CNN for binary classification of 1x1x48x48 images using the cybersecurity end2endexample on the Xilinx ZCU-104.
Right now i come back to FINN for upgrading the model with the new training we managed to achieve and i was not able to use the old environment so i built a new one.

In this one the code i used to built the bitfile doesn't work anymore,

This is the code I'm using:

import finn.builder.build_dataflow as build
import finn.builder.build_dataflow_config as build_cfg
import os
import shutil

model_file = "./onnx/cnv_adapted.onnx"


rtlsim_output_dir = "output_ipstitch_ooc_rtlsim"

#Delete previous run results if exist
if os.path.exists(rtlsim_output_dir):
    shutil.rmtree(rtlsim_output_dir)
    print("Previous run results deleted!")

cfg_stitched_ip = build.DataflowBuildConfig(
    output_dir          = rtlsim_output_dir,
    mvau_wwidth_max     = 1000,
    target_fps          = 1000000,
    synth_clk_period_ns = 5.4,
    fpga_part           = "xczu7ev-ffvc1156-2-i",
    folding_config_file     = "folding_dsp.json",
    generate_outputs=[
        build_cfg.DataflowOutputType.STITCHED_IP,
        build_cfg.DataflowOutputType.RTLSIM_PERFORMANCE,
        build_cfg.DataflowOutputType.OOC_SYNTH,
    ]
)

and this is my folding_dsp.json:

{
  "Defaults": {},
  "Thresholding_Batch_0": {
    "PE": 1,
    "ram_style": "distributed",
    "mem_mode": "const",
    "runtime_writeable_weights": 0
  },
  "FMPadding_Batch_0": {
    "SIMD": 1
  },
  "ConvolutionInputGenerator_0": {
    "SIMD": 1,
    "ram_style": "distributed"
  },
  "MatrixVectorActivation_0": {
    "PE": 8,
    "SIMD": 9,
    "ram_style": "auto",
    "resType": "dsp",
    "mem_mode": "decoupled",
    "runtime_writeable_weights": 0
  },
  "StreamingMaxPool_Batch_0": {
    "PE": 1
  },
  "FMPadding_Batch_1": {
    "SIMD": 1
  },
  "ConvolutionInputGenerator_1": {
    "SIMD": 5,
    "ram_style": "distributed"
  },
  "MatrixVectorActivation_1": {
    "PE": 8,
    "SIMD": 9,
    "ram_style": "auto",
    "resType": "dsp",
    "mem_mode": "decoupled",
    "runtime_writeable_weights": 0
  },
  "StreamingMaxPool_Batch_1": {
    "PE": 1
  },
  "FMPadding_Batch_2": {
    "SIMD": 1
  },
  "ConvolutionInputGenerator_2": {
    "SIMD": 1,
    "ram_style": "distributed"
  },
  "MatrixVectorActivation_2": {
    "PE": 4,
    "SIMD": 1,
    "ram_style": "auto",
    "resType": "dsp",
    "mem_mode": "decoupled",
    "runtime_writeable_weights": 0
  },
  "MatrixVectorActivation_3": {
    "PE": 4,
    "SIMD": 1,
    "ram_style": "auto",
    "resType": "dsp",
    "mem_mode": "decoupled",
    "runtime_writeable_weights": 0
  },
  "MatrixVectorActivation_4": {
    "PE": 4,
    "SIMD": 1,
    "ram_style": "auto",
    "resType": "dsp",
    "mem_mode": "decoupled",
    "runtime_writeable_weights": 0
  },
  "MatrixVectorActivation_5": {
    "PE": 4,
    "SIMD": 1,
    "ram_style": "auto",
    "resType": "dsp",
    "mem_mode": "decoupled",
    "runtime_writeable_weights": 0
  }
}

in the step_hw_ipgen [12/19] i get:

ValueError: cannot reshape array of size 144 into shape (1,12,12,0,4)

my net is structured as follows:

class brevitas_model(Module):
    def __init__(self):
        super(brevitas_model, self).__init__()
        self.quant_inp = qnn.QuantIdentity(bit_width=8, return_quant_tensor=True)
        self.conv1 = qnn.QuantConv2d(in_channels=1, out_channels=10, kernel_size=(3,3), stride=(1,1), padding=(1,1), groups=1, dilation=1, bias=True, weight_bit_width=8, bias_quant=Int8Bias)
        self.relu1 = qnn.QuantReLU(bit_width=8, return_quant_tensor=True)
        self.conv2 = qnn.QuantConv2d(in_channels=10, out_channels=5, kernel_size=(3,3), stride=(1,1), padding=(1,1), groups=1, dilation=1, bias=True, weight_bit_width=8, bias_quant=Int8Bias)
        self.relu2 = qnn.QuantReLU(bit_width=8, return_quant_tensor=True)
        self.conv3 = qnn.QuantConv2d(in_channels=5, out_channels=1, kernel_size=(3,3), stride=(1,1), padding=(1,1), groups=1, dilation=1, bias=True, weight_bit_width=8, bias_quant=Int8Bias)
        self.relu3 = qnn.QuantReLU(bit_width=8, return_quant_tensor=True)
        self.flatten = nn.Flatten()
        self.fc1 = qnn.QuantLinear(144, 72, bias=True, weight_bit_width=8, bias_quant=Int8Bias)
        self.relu4 = qnn.QuantReLU(bit_width=8, return_quant_tensor=True)
        self.fc2 = qnn.QuantLinear(72, 72, bias=True, weight_bit_width=8, bias_quant=Int8Bias)
        self.relu5 = qnn.QuantReLU(bit_width=8, return_quant_tensor=True)
        self.fc3 = qnn.QuantLinear(72, 1, bias=True, weight_bit_width=8, bias_quant=Int8Bias)
    

def forward(self, x): 
    out = self.quant_inp(x)
    out = self.conv1(out)
    out = self.relu1(out)
    out = F.max_pool2d(out, kernel_size=(2, 2), stride=(2, 2))
    out = self.conv2(out)
    out = self.relu2(out)
    out = F.max_pool2d(out, kernel_size=(2, 2), stride=(2, 2))
    out = self.conv3(out)
    out = self.relu3(out)
    out = self.flatten(out)
    out = self.fc1(out)
    out = self.relu4(out)
    out = self.fc2(out)
    out = self.relu5(out)
    out = self.fc3(out)


    return out

model = brevitas_model()

I also tried the method in the cnv_end2end_example.ipynb following the transformations and adapting it to my code. Here i managed to get to the following dense layer, the one with dimension 72 but i get almost the same error:

     # Load the adapted model
  model = ModelWrapper("./onnx/cnv_adapted_model.onnx")
  
  # Fetch fully connected layers
  fc_layers = model.get_nodes_by_op_type("MVAU_hls")
  
  # Folding parameters: each tuple is (PE, SIMD, in_fifo_depth)
  folding = [
      (16, 1, [128]),  # conv1
      (32, 10, [128]), # conv2
      (16, 5, [128]),  # conv3
      (8, 9, [64]),    # fc1
      (8, 9, [64]),    # fc2
      (4, 9, [32]),    # fc3
  ]
  
  # Apply folding parameters to fully connected layers
  for fcl, (pe, simd, ififodepth) in zip(fc_layers, folding):
      fcl_inst = getCustomOp(fcl)
      fcl_inst.set_nodeattr("PE", pe)
      fcl_inst.set_nodeattr("SIMD", simd)
      fcl_inst.set_nodeattr("inFIFODepths", ififodepth)
      # Debug: Verify parameters are set correctly
      print(f"{fcl.name} -> PE: {pe}, SIMD: {simd}, inFIFOdepths: {ififodepth}")
  
  # Use same SIMD values for the sliding window operators
  swg_layers = model.get_nodes_by_op_type("ConvolutionInputGenerator_rtl")
  for i, swg in enumerate(swg_layers):
      if i < len(folding):
          swg_inst = getCustomOp(swg)
          simd = folding[i][1]
          swg_inst.set_nodeattr("SIMD", simd)
          # Debug: Verify SIMD is set correctly
          print(f"{swg.name} -> SIMD: {simd}")
      else:
          print(f"No folding parameters available for {swg.name}")
  
  # Save the transformed model
  model = model.transform(GiveUniqueNodeNames())
  model.save("./onnx/cnv_adapted_folded.onnx")
  
  # Debug: Print the shapes to ensure compatibility
  for node in model.graph.node:
      node_inst = getCustomOp(node)
      try:
          print(f"Checking folded input shape for node: {node.name}")
          folded_shape = node_inst.get_folded_input_shape()
          print(f"{node.name}: Folded input shape = {folded_shape}")
      except Exception as e:
          print(f"Error getting folded input shape for {node.name}: {e}")
  
  # Hardware generation parameters
  test_pynq_board = "Pynq-Z1"
  target_clk_ns = 10
  
  # Transform and verify if parameters are applied
  try:
      model = model.transform(ZynqBuild(platform=test_pynq_board, period_ns=target_clk_ns))
      print("Hardware generation successful")
  except Exception as e:
      print(f"Error during hardware generation: {e}")

In this code a brief debug snippet was added to try to understand what is happening but the results are confusing, it gives me this output/error:

-> PE: 16, SIMD: 1, inFIFOdepths: [128]
 -> PE: 32, SIMD: 10, inFIFOdepths: [128]
 -> SIMD: 1
 -> SIMD: 10
 -> SIMD: 5
Checking folded input shape for node: FMPadding_rtl_0
FMPadding_rtl_0: Folded input shape = (1, 48, 48, 1, 1)
Checking folded input shape for node: ConvolutionInputGenerator_rtl_0
ConvolutionInputGenerator_rtl_0: Folded input shape = (1, 50, 50, 1, 1)
Checking folded input shape for node: MVAU_rtl_0
MVAU_rtl_0: Folded input shape = (1, 48, 48, 9, 1)
Checking folded input shape for node: Thresholding_rtl_0
Thresholding_rtl_0: Folded input shape = (1, 48, 48, 10, 1)
Checking folded input shape for node: StreamingMaxPool_hls_0
StreamingMaxPool_hls_0: Folded input shape = (1, 48, 48, 1, 10)
Checking folded input shape for node: FMPadding_rtl_1
FMPadding_rtl_1: Folded input shape = (1, 24, 24, 1, 10)
Checking folded input shape for node: ConvolutionInputGenerator_rtl_1
ConvolutionInputGenerator_rtl_1: Folded input shape = (1, 26, 26, 1, 10)
Checking folded input shape for node: MVAU_rtl_1
MVAU_rtl_1: Folded input shape = (1, 24, 24, 90, 1)
Checking folded input shape for node: Thresholding_rtl_1
Thresholding_rtl_1: Folded input shape = (1, 24, 24, 5, 1)
Checking folded input shape for node: StreamingMaxPool_hls_1
StreamingMaxPool_hls_1: Folded input shape = (1, 24, 24, 1, 5)
Checking folded input shape for node: FMPadding_rtl_2
FMPadding_rtl_2: Folded input shape = (1, 12, 12, 1, 5)
Checking folded input shape for node: ConvolutionInputGenerator_rtl_2
ConvolutionInputGenerator_rtl_2: Folded input shape = (1, 14, 14, 1, 5)
Checking folded input shape for node: MVAU_rtl_2
MVAU_rtl_2: Folded input shape = (1, 12, 12, 45, 1)
Checking folded input shape for node: Thresholding_rtl_2
Thresholding_rtl_2: Folded input shape = (1, 12, 12, 1, 1)
Checking folded input shape for node: MVAU_hls_0
MVAU_hls_0: Folded input shape = (1, 144, 1)
Checking folded input shape for node: MVAU_hls_1
MVAU_hls_1: Folded input shape = (1, 7, 10)
Checking folded input shape for node: MVAU_rtl_3
MVAU_rtl_3: Folded input shape = (1, 72, 1)
Checking folded input shape for node: LabelSelect_hls_0
LabelSelect_hls_0: Folded input shape = (1, 1, 1)
/home/abrate/Desktop/finn/src/finn/transformation/fpgadataflow/floorplan.py:107: UserWarning: 26 nodes have no entry in the provided floorplan, SLR was set to -1
  warnings.warn(
Error during hardware generation: cannot reshape array of size 72 into shape (1,4,16)

It seems that the last layer is trying to use a wrong SIMD, i tried also using 1 and 12 but nothing changes, I can't understand what is happening and the reason why the same identical process mentioned above following the simple end2end flow worked about 10 month ago and now it is completely different.

Thanks in advance to everyone that can help me and thanks for the beautiful work you have done with FINN!

Matteo

[fpjentzsch]

Hi,

it looks like your folding config has no effect because it refers to the old node names (e.g. "MatrixVectorActivation_0"). FINN recently underwent major refactoring for release v0.10, which introduces "specialization" of layers into _hls and _rtl variants to target different back-end implementations (e.g. "MVAU_hls_0" or "MVAU_rtl_0", depending on which back-end was selected by you (manually via a specialize_layers_config_file json) or the compiler (automatically)).

See this guide for further information on how to update your build flow: #1020

[MatteoAbrate]

Thanks a lot!