How to use different text encoder from the default config?

[Edwardmark]

Hi, could you please tell me how to use a custum text encoder?

For example, I want to change the text_encoder in open_clip/src/open_clip/model_configs/ViT-B-16-SigLIP2-256.json to https://huggingface.co/OFA-Sys/chinese-clip-vit-base-patch16/blob/main/config.json.
I change the ViT-B-16-SigLIP2-256.json as follows:
"text_cfg": { "context_length": 64, "vocab_size": 21128, "hf_model_name": "OFA-Sys/chinese-clip-vit-large-patch14", "hf_tokenizer_name": "OFA-Sys/chinese-clip-vit-large-patch14", "tokenizer_kwargs": { "clean": "canonicalize", "byte_fallback": false }, "width": 768, "heads": 12, "layers": 12, "no_causal_mask": true, "proj_bias": true, "pool_type": "last" } }

but error says as:
TypeError: ChineseCLIPModel.init() got an unexpected keyword argument 'add_pooling_layer'.

Could you help me with that? Thanks

[Edwardmark]

@lucidrains @fabito @mehdidc @Separius