With the default config, hugging face text encoder does not add casual mask, which may lead to information leakage in the decoder side. If using text transformers in open_clip, the problem does not exist since casual mask was added.