Awesome-Visual-Instruction-Tuning

• Awesome-Visual-Instruction-Tuning
  • Visual Instruction Tuning
  • Instruction Tuning
  • Awesome Awesomeness

Visual Instruction Tuning

Title	Venue	Date	Code	Demo	Dataset
LEGO:Language Enhanced Multi-modal Grounding Model	arXiv	2024-01-11	Github	Coming soon	Coming soon
VILA: On Pre-training for Visual Language Models	arXiv	2023-12-12	-	-	-
LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning	arXiv	2023-12-06	-	-	-
InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Models	arXiv	2023-12-04	-	-	-
LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning	arXiv	2023-11-30	Github	Demo	-
Continual Instruction Tuning for Large Multimodal Models	arXiv	2023-11-27	-	-	-
Contrastive Vision-Language Alignment Makes Efficient Instruction Learner	arXiv	2023-11-23	Coming soon	-	-
DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback	arXiv	2023-11-16	-	-	Dataset
To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning	arXiv	2023-11-13	Github	-	Dataset
OtterHD: A High-Resolution Multi-modality Model	arXiv	2023-11-07	Github	Demo	Dataset
What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning	arXiv	2023-11-02	Github	-	-
REVO-LION: Evaluating and Refining Vision-Language Instruction Tuning Datasets	arXiv	2023-10-10	Coming soon	-	Coming soon
Improved Baselines with Visual Instruction Tuning	arXiv	2023-10-05	Github	Demo	Dataset
Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning	arXiv	2023-09-29	Github	Demo	Dataset
An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models	arXiv	2023-09-18	-	-	-
ImageBind-LLM: Multi-modality Instruction Tuning	arXiv	2023-09-07	Github	-	-
LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding	arXiv	2023-06-29	Github	Demo	Dataset
MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning	ACL 2023	2023-06-10	Github	-	-
MIMIC-IT: Multi-Modal In-Context Instruction Tuning	arXiv	2023-06-08	Github	Demo	-
M$^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning	arXiv	2023-06-07	-	Demo	Dataset
Visual Instruction Tuning	NeurIPS 2023, Oral	2023-04-17	Github	Demo	Dataset

Instruction Tuning

Title	Venue	Date	Code	Demo
MoDS: Model-oriented Data Selection for Instruction Tuning	arXiv	2023-11-27	Github	-
Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks	EMNLP 2023	2023-11-01	Coming soon	-
Distort, Distract, Decode: Instruction-Tuned Model Can Refine its Response from Noisy Instructions	NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following	2023-11-01	-	-
Dynamics of Instruction Tuning: Each Ability of Large Language Models Has Its Own Growth Pace	arXiv	2023-10-30	Github	-
Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning	NeurIPS 2023 Workshop	2023-10-18	Github	-
LoBaSS: Gauging Learnability in Supervised Fine-tuning Data	arXiv	2023-10-16	-	-
How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition	arXiv	2023-10-09	-	-
Instruction Tuning for Large Language Models: A Survey	arXiv	2023-10-04	Github	-
From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning	arXiv	2023-08-23	Github	-
#InsTag: Instruction Tagging for Diversity and Complexity Analysis	arXiv	2023-08-14	Github	Demo
Orca: Progressive Learning from Complex Explanation Traces of GPT-4	arXiv	2023-06-05	-	-
LIMA: Less Is More for Alignment	arXiv	2023-05-18	-	-
Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuning	arXiv	2023-05-16	-	-
Instruction Tuning with GPT-4	arXiv	2023-04-06	Github	Demo
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning	arXiv	2023-01-31	Github	-
How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources	NeurIPS 2023	2023-01-07	Github	-
Self-Instruct: Aligning Language Models with Self-Generated Instructions	ACL 2023	2022-12-20	Github	-
Scaling Instruction-Finetuned Language Models	arXiv	2022-10-20	-	-

Awesome Awesomeness

Repositories	Stars
Awesome-LLM-Survey
Instruction-Tuning-Papers