FINN model gives array of 0's at output

[B-Willems]

When trying to run my custom model on pynq-z2 using FINN I'm encountering a weird issue. I made a small CNN to classify a sattelite image as "iceberg" or "ship". When i run my model on the PL I always get 0 (ship) on the output whatever the input is. To debug this i tried to remove the TopK layer so i can see what the output of my model is before executing the labelselect_batch. However instead of the expected 1D array containing 2 values, i get a 2D (2x2) array with all 0's as can be seen in the screenshot below.

My quantized brevitas model however does work as expected when I test it:

My model is shown below

`class Network(nn.Module):
def init(self):
# Define all the parameters of the ModelOne
super(Network, self).init()

      self.quant_inp = qnn.QuantIdentity(bit_width=4, return_quant_tensor=True)

      self.conv1 = qnn.QuantConv2d(in_channels=4, out_channels=8, kernel_size=3, return_quant_tensor=True, weight_bit_width=3, bias_quant=BiasQuant)
      self.relu1 = qnn.QuantReLU(return_quant_tensor=True, bit_width=4)
      self.maxpool1 = qnn.QuantMaxPool2d(kernel_size=2, return_quant_tensor=True)

      self.drop1 = qnn.QuantDropout(0.1)

      #self.conv2 = qnn.QuantConv2d(in_channels=8, out_channels=16, kernel_size=3, return_quant_tensor=True, weight_bit_width=3, bias_quant=BiasQuant)
      #self.relu2 = qnn.QuantReLU(return_quant_tensor=True, bit_width=4)
      #self.maxpool2 = qnn.QuantMaxPool2d(kernel_size=2, return_quant_tensor=True)

      #self.drop2 = qnn.QuantDropout(0.1)

      #self.conv3 = qnn.QuantConv2d(in_channels=16, out_channels=32, kernel_size=3, return_quant_tensor=True, weight_bit_width=3, bias_quant=BiasQuant)
      #self.relu3 = qnn.QuantReLU(return_quant_tensor=True, bit_width=4)
      #self.maxpool3 = qnn.QuantMaxPool2d(kernel_size=2, return_quant_tensor=True)

      #self.drop3 = nn.Dropout(0.1)

      self.conv4 = qnn.QuantConv2d(in_channels=8, out_channels=16, kernel_size=3, return_quant_tensor=True, weight_bit_width=3, bias_quant=BiasQuant)
      self.conv5 = qnn.QuantConv2d(in_channels=16, out_channels=16, kernel_size=3, return_quant_tensor=True, weight_bit_width=3, bias_quant=BiasQuant)
      self.relu4 = qnn.QuantReLU(return_quant_tensor=True, bit_width=4)
      self.maxpool4 = qnn.QuantMaxPool2d(kernel_size=2, return_quant_tensor=True)
      self.drop4 = qnn.QuantDropout(0.1)
      self.flat1 = nn.Flatten()

      self.line1 = qnn.QuantLinear(in_features=4624, out_features=16, bias=True, return_quant_tensor=True, bias_quant=BiasQuant, weight_bit_width=3)
      self.relu5 = qnn.QuantReLU(bit_width=4)

      self.quant_inp2 = qnn.QuantIdentity(bit_width=4, return_quant_tensor=True)

      self.line2 = qnn.QuantLinear(in_features=16, out_features=16, bias=True, return_quant_tensor=True, bias_quant=BiasQuant, weight_bit_width=3)
      self.relu6 = qnn.QuantReLU(bit_width=4)

      self.quant_inp3 = qnn.QuantIdentity(bit_width=4, return_quant_tensor=True)

      self.line3 = qnn.QuantLinear(in_features=16, out_features=2, bias=False, weight_bit_width=3, return_quant_tensor=True)

  def forward(self, x):
      out = x
      #out = self.pad(out)
      #out = out.permute(0, 3, 2, 1)

      out = self.quant_inp(out)

      out = self.conv1(out)
      out = self.relu1(out)
      out = self.maxpool1(out)
      out = self.drop1(out)

      out = self.conv4(out)
      out = self.conv5(out)
      out = self.relu4(out)
      out = self.maxpool4(out)
      out = self.drop4(out)

      out = self.flat1(out)

      out = self.line1(out)
      out = self.relu5(out)

      out = self.quant_inp2(out)

      out = self.line2(out)

      out = self.relu6(out)
      out = self.quant_inp3(out)
      out = self.line3(out)
      return out

`

Any help is welcome as i have been debugging this for over a week now and can't seem to get anywhere.

[fpjentzsch]

Hi, when you say "the quantized brevitas model works", do you run inference still within PyTorch or do you execute the exported .onnx model?

To find out if the error is introduced somewhere during the transformation steps, you can use FINN's cppsim or rtlsim infrastructure to simulate the .onnx at different steps.

If everything looks fine up until RTL simulation, but it's still broken on hardware, then you might have an error in your driver (e.g. input data packing and output data unpacking).

[B-Willems]

I did verification using the provided notebook. The exported .onnx model works just fine, but the RTLsim of the stitched IP always gives the same output no matter what input image is provided.

Do you have any idea what might be causing this issue?

[JPPalacios]

@B-Willems I am facing something similar on my pynq-buffer outputs from my accelerator. Did you end up figuring it out?

[eveneha]

@B-Willems @JPPalacios @fpjentzsch
Also facing the same issue with only zero-outputs from accelerator. Anyone figured something out?

[eveneha]

UPDATE: Issue was with the quantization layers and input datatype. I lined up quantization correctly through the layers and changed the datatype of the input to match what were made in driver.py, and it worked.