InferQuantizedStreamingFCLayer: Signed output requres actval < 0

[rbcarlos]

I am working on a simple draft for sentiment classification using Conv1D layers. When running through the step_convert_to_hls I get the following error in the InferQuantizedStreamingFCLayer transformation:

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-7-5bb031d2ee74> in <module>
      3 model = model.transform(to_hls.InferBinaryStreamingFCLayer(mem_mode))
      4 # needed for non-bipolar MatMul layers
----> 5 model = model.transform(to_hls.InferQuantizedStreamingFCLayer(mem_mode))
      6 # TopK to LabelSelect
      7 model = model.transform(to_hls.InferLabelSelectLayer())

/workspace/finn-base/src/finn/core/modelwrapper.py in transform(self, transformation, make_deepcopy, cleanup, fix_float64)
    139         model_was_changed = True
    140         while model_was_changed:
--> 141             (transformed_model, model_was_changed) = transformation.apply(
    142                 transformed_model
    143             )

/workspace/finn/src/finn/transformation/fpgadataflow/convert_to_hls_layers.py in apply(self, model)
    775                             + ": out_scale=1 or bipolar output needed for conversion."
    776                         )
--> 777                         assert (not odt.signed()) or (actval < 0), (
    778                             consumer.name + ": Signed output requres actval < 0"
    779                         )

AssertionError: MultiThreshold_1: Signed output requres actval < 0

Here is the definition of the network in Brevitas:

class TextClassificationModel(nn.Module):

    def __init__(self, pretrained_embeddings, num_class, weight_bit_width, input_bit_width, act_bit_width, freeze_embeddings = False ):
        super(TextClassificationModel, self).__init__()

        self.quant_inp = qnn.QuantIdentity(
            bit_width=4, return_quant_tensor=True, act_quant=CommonActQuant)
        
        self.conv_1 = qnn.QuantConv1d(in_channels = pretrained_embeddings.shape[1], 
                                              out_channels = 200, 
                                              kernel_size = 2,
                                              bias = False,
                                              weight_bit_width = weight_bit_width,
                                              return_quant_tensor=True,
                                              weight_quant=CommonWeightQuant)
        self.conv_2 = qnn.QuantConv1d(in_channels = pretrained_embeddings.shape[1], 
                                              out_channels = 200, 
                                              kernel_size = 3,
                                              bias = False,
                                              weight_bit_width = weight_bit_width,
                                              return_quant_tensor=True,
                                              weight_quant=CommonWeightQuant)
        self.conv_3 = qnn.QuantConv1d(in_channels = pretrained_embeddings.shape[1], 
                                              out_channels = 200, 
                                              kernel_size = 4,
                                              bias = False,
                                              weight_bit_width = weight_bit_width,
                                              return_quant_tensor=True,
                                              weight_quant=CommonWeightQuant)
        
        self.relu_1 = qnn.QuantReLU(bit_width = act_bit_width,return_quant_tensor=True,act_quant=CommonActQuant)
        
        self.relu_2 = qnn.QuantReLU(bit_width = act_bit_width,return_quant_tensor=True,act_quant=CommonActQuant)
        
        self.relu_3 = qnn.QuantReLU(bit_width = act_bit_width,return_quant_tensor=True,act_quant=CommonActQuant)
        
        self.relu_4 = qnn.QuantReLU(bit_width = act_bit_width,
                                              return_quant_tensor=True,act_quant=CommonActQuant)
        
        self.relu_5 = qnn.QuantReLU(bit_width = act_bit_width,
                                              return_quant_tensor=True, act_quant=CommonActQuant)
        
        self.fc1 = qnn.QuantLinear(1*200, 512, bias = False,
                                              weight_bit_width = weight_bit_width,
                                              return_quant_tensor=True,
                                              weight_quant=CommonWeightQuant)
        self.fc2 = qnn.QuantLinear(512, 64, bias = False,
                                              weight_bit_width = weight_bit_width,
                                              return_quant_tensor=True,
                                              weight_quant=CommonWeightQuant)
        self.fc3 = qnn.QuantLinear(64, 1, bias = False,
                                              weight_bit_width = weight_bit_width,
                                              weight_quant=CommonWeightQuant)
        
        self.dropout = nn.Dropout(0.2)

        self.kernel_sizes = [200 - x + 1 for x in [2,3,4]]

    def forward(self, embedded):
        embedded = embedded.permute(0, 2, 1)

        x = self.quant_inp(embedded)
        
        conved = []
        conved.append(self.relu_1(self.conv_1(x)))
        conved.append(self.relu_2(self.conv_2(x)))
        conved.append(self.relu_3(self.conv_3(x)))
            
        pooled = [F.max_pool1d(conv, self.kernel_sizes[i]).reshape(-1, 200) for i,conv in enumerate(conved)]

        cat = self.dropout(pooled[0])
            
        x = self.relu_4(self.fc1(cat))
        x = self.relu_5(self.fc2(x))
        x = self.fc3(x)

        return x

I have also included the absorb.AbsorbSignBiasIntoMultiThreshold() transform before the streamline, as suggested in a thread on gitter.

I am also including the onnx file.
end2end_sentiment-8.onnx.zip

On another note. As you can see, I have multiple conv layers defined, which I would like to concatenate before the dropout layer. Is there any way to do that? I tried but it did not work, as the Change3DTo4DTensors() transformation does not support this type of node.

[rbcarlos]

I have fixed this issue by using the QuantIdentity layer instead of the QuantReLU.