Conv2D is no longer supporting Masking in TF v2.17.0

[kavjayawardana]

Dear Keras team,

Conv2D layer no longer supports Masking layer in TensorFlow v2.17.0. I've already raised this issue with TensorFlow. However, they requested that I raise the issue here.

Due the dimensions of our input (i.e. (timesteps, width, channels)), size of the input shape (i.e. (2048, 2000, 3)) and size of the dataset (i.e. over 1 million samples), it is not practical to use LSTM, GRU, RNN or ConvLSTM1D layers, and therefore, Conv2D layers worked sufficiently well in our applications. The gaps in the dataset was handled with the Masking layer, and the masking layer was compatible with the Conv layers (among other layers, such as Cropping and Padding) from all TF versions up to (and including) TF v2.16. However, in TF v2.17.0, we get the following user warning "Layer 'conv2d' (of type Conv2D) was passed an input with a mask attached to it. However, this layer does not support masking and will therefore destroy the mask information. Downstream layers will not see the mask".

Is this a bug in TF v2.17.0?
Or is this feature now depreciated in TF v2.17.0?
Would you be able to reintroduce this feature in future versions?

Best

Kav

LINK TO THE CODE ON COLAB NOTEBOOK:
https://colab.research.google.com/drive/102k6UNSKb-d03DcmcUtCxmV9Qz9bjZoD?usp=drive_link

STANDALONE CODE:
from tensorflow.keras.layers import Conv2D, Masking, Flatten
from tensorflow.keras import Model, Input

batch = 1
timesteps = 10
width = 10
channels = 2
filters = 4
kernel_size = 3
mask_value = -1

x_input = Input(shape=(timesteps, width, channels))
x_masking = Masking(mask_value)(x_input)
x_conv2d = Conv2D(filters, kernel_size)(x_masking)
x_flatten = Flatten()(x_conv2d)

model = Model(x_input, x_flatten)
model.compile(loss='mse')

RELEVANT LOG OUTPUT
/usr/local/lib/python3.10/dist-packages/keras/src/layers/layer.py:915: UserWarning: Layer 'conv2d' (of type Conv2D) was passed an input with a mask attached to it. However, this layer does not support masking and will therefore destroy the mask information. Downstream layers will not see the mask.
warnings.warn(

LINK TO THE ORIGINAL RAISED ISSUE ON TENSORFLOW REPO
tensorflow/tensorflow#73531

[fchollet]

I do not believe that masking was ever supported in Conv2D layers (your mask may have been accepted by the layer, but it would not have been taken into account by the convolution computation).

Q: what do you need the mask for? Maybe you can just pass it to the layer that needs it.

[kavjayawardana]

It was never supported in the previous version? Interesting. The user warning never appeared in the previous TF versions, and thus, I naturally assumed Conv2d supported the Masking layer. I’m currently using TF v2.16 in my local machine, and the warning never appeared in it or in the previous versions.

Regarding your question: we have an array of dim-(2000, 3) observations coming in very second. We are using 2048 consecutive seconds of data to predict a label, and we are using tf.keras.preprocessing.timeseries_dataset_from_array to feed the data during training. However, there are gaps in our dataset (due to mechanical failures and such), and thus, we need masking to remove such samples during the training process. Therefore, the only option available for us is use Masking.

As for using Conv2d (instead of using ConvLSTM1D), Conv2D is significantly faster to train and to run than ConvLSTM1D

[kavjayawardana]

@fchollet, thank you for informing me that Masking was never supported in Conv2D. I think I now found a solution to the problem
best

kav

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No