Adding image decoder to CoCa

[iejMac]

No description provided.

[iejMac]

Ok current state is bare minimum version to get things kind of working. That means:

• We get the image tokenized using VQGAN (I think this is correct, still need to write some decoding code to check if we're tokenizing correctly
• We create a image decoder transformer which is just like the text decoder transformer and predict the next image token autoregressively
• We calculate the loss
• Code is as decent as I could make it in one sitting. Still needs improvement

[iejMac]

@gpucce @rom1504 wandb = https://wandb.ai/iejmac/open-clip/reports/MAGIC--VmlldzozODE4NzM3

[iejMac]

TODO:

Code:

• CoCa generation code should be modality-agnostic - it should be able to generate images and text based on the shape (or parameters) of the input
• create some start_of_image token !!!
• BIG Cleanup. Can we go without making a dependency on taming-transformers and omegaconf?
• Config cleanup + update old coca configs

Model:

• Train something at B/32 scale
• dropout text conditioning 10% of the time as suggested by Katherine (either put nothing in cross attention or some learned sequence)
• axial positional embeddings suggested by lucidrains

[iejMac]

https://arxiv.org/abs/2303.13455