Insights, findings and technical deep-dives from the Pico community. We believe that training language models should be a science not an art.
We embedded a well-known "learn-to-learn" algorithm (MAML) directly into the pretraining loop of Llama-style decoder models, ranging from 11M to 570M parameters. Here's what we found, and why it matters.
David Africa
ReadParameter-efficient training methods have transformed how we work with large language models. LoRA, which fine-tunes only a small set of low-rank matrix updates while keeping the rest of the model frozen, has become a staple of the modern ML toolkit. Its extension, **ReLoRA**, pushed this idea one step further: instead of just fine-tuning with low-rank updates, why not *pretrain* with them? Turns out it's not that simple.
Yuval Weiss
Read