How to Perform Full Cross-Attention with CP Enabled and Padding Mask #1915
Description
I have a query q and a pair of key-value kv, and I want to perform full cross-attention on them with CP enabled. However, the lengths of the tensors are not multiples of cp * 2, so I applied padding and passed in a mask. I also set AttnMaskType to padding. However, I encountered an issue: enabling CP and selecting AttnMaskType as padding is not supported simultaneously. How should I set the parameters to achieve this?