- https://huggingface.co/SII-GAIR-NLP/davinci-llm-model

Overview

daVinci-LLM-3B is a 3B-parameter base language model presented in daVinci-LLM: Towards the Science of Pretraining. This project aims to make the pretraining process a transparent and reproducible scientific endeavor.

We release not only the final weights but also training trajectories, intermediate checkpoints, data processing decisions, and 200+ ablation studies covering data quality, mixture design, training dynamics, and evaluation validity.

• GitHub: GAIR-NLP/daVinci-LLM
• Paper: arXiv:2603.27164
• Dataset: davinci-llm-data

The model follows a two-stage curriculum over ~8T tokens:

• Stage 1 (6T tokens): broad pretraining over diverse web-scale corpora.
• Stage 2 (2T tokens): structured QA and reasoning-heavy data to amplify math and code reasoning.