Thanks for the curated list of instruction dataset. We have a related work that we hope is added to this awesome repository.
Accountable Textual-Visual Chat Learns to Reject Human Instructions in Image Re-creation (https://arxiv.org/pdf/2303.05983.pdf)
Project Page: https://matrix-alpha.github.io/