LittleLanguageModel

This repo is a tiny language model mimicking LLMs. I would like to create LLM-like model from scratch. To some degree LLM could also be abbreviated to little language model. :D

Todolist

• Collect Data
• Clean & Augment Data
• Make Tokenizer
• Make Model Based on Transformer
• Pretrain Model
• LLM Alignment

Optional

• Transfer to downstream tasks.
• SFT Training
• Distribute Data Parallel
• Distill model
• RAG
• Agent

Recommend to read

1. LLM数据工程3——数据收集魔法：获取顶级训练数据的方法
2. 掌握 LLM 技术：数据预处理