Puro-2B: How Can a Poor Lab Pretrain a Model from Scratch?
Matching Qwen2-1.5B on RTX 5090 GPUs—with a $5,090 budget
On RTX 5090 GPUs, Puro-2B matches Qwen2-1.5B on the average score across 15 tasks with about $4,400 in pretraining compute—under $5,090. We cover the hardware, FP8, optimizer, and data recipe, and provide the code, data, training configurations, and model weights needed to reproduce the experiments.
Kairong Luo, Jiarui Cui, Yaorui Yin, Shengqi Chen et al.Instructors: Wenguang Chen, Kaifeng LyuTranslated from Chinese with AI assistance.