Large-Scale Open Datasets: The Foundation
RedPajama (1.2T tokens)
A reproduction of LLaMA's training data. Built from CommonCrawl, GitHub, arXiv, and other sources, it ships under the Apache 2.0 license and is cleared for commercial use. Version 2 expands the corpus to 30 trillion tokens.
The Pile (825GB)
A high-quality dataset released by EleutherAI that combines 22 sources. It is rich in technical documents from arXiv, PubMed, and similar venues, and was used to train the Pythia models.
C4 (Colossal Clean Crawled Corpus, 750GB)
Built to train Google's T5, C4 is a heavily filtered version of CommonCrawl. It carries the ODC-BY license, so commercial use is unrestricted.
Dolma (3T tokens) & OLMo
Built by Allen AI to train the OLMo models. It leans heavily on AI/ML technical material — Semantic Scholar papers, math content, and more — which makes it especially well suited for training in those domains.
Real-World LLM Training Cases: Proven Strategies
LLaMA (Meta)
Key takeaway: Open data alone is enough to reach state-of-the-art performance. The winning move is to pair large-scale web data (67% CommonCrawl) with high-quality specialist sources such as GitHub, Wikipedia, and arXiv.
Pythia (EleutherAI)
Key takeaway: Trained on The Pile, with every training checkpoint and the underlying data released openly. It shows that thorough documentation and transparency are essential to scientific reproducibility and to earning the community's trust.
OLMo (Allen AI)
Key takeaway: A two-stage training strategy works well. Stage one trains broadly on large-scale general data; stage two shores up specific capabilities with high-quality targeted data such as math and Q&A.
Falcon (TII)
Key takeaway: Proved that web data alone (RefinedWeb) can reach SOTA. Its extremely aggressive deduplication suggests that data quality can matter more than diversity.
Dataset Summary
| Dataset | Size | Tokens | License | Languages |
|---|---|---|---|---|
| RedPajama-v1 | 1.2T | 1.2T | Apache 2.0 | EN + 20 |
| The Pile | 825GB | 300B | MIT + per-source | EN |
| C4 | 750GB | ~750B | ODC-BY | 100+ |
| Dolma v1.7 | 4.5TB | 2.3T | ODC-BY | Mostly EN |
| FineWeb | 44TB | 15T | ODC-BY | EN (v2: multilingual) |
| The Stack v2 | 32.1TB | 900B | Permissive licenses | 658 |
| Wikipedia (EN) | ~58GB | ~2.24B | CC-BY-SA 4.0 | 338 |
Recommended Starter Mix (7B model, 1T tokens)
General-Purpose Model
- CommonCrawl (filtered): 65%
- C4: 10%
- The Stack (code): 5%
- GitHub (code): 5%
- Wikipedia: 5%
- Books (Gutenberg): 5%
- arXiv (papers): 3%
- StackExchange (Q&A): 2%
Data Composition
Code-Centric Model
- The Stack: 70%
- GitHub: 15%
- StackExchange (coding tags): 5%
- Documentation: 5%
- arXiv (CS papers): 3%
- General web (filtered): 2%
Data Composition
Conclusion and Recommendations
The track record of leading AI labs makes one thing clear: you can build a state-of-the-art LLM entirely on openly licensed data. What separates success from failure is the strategic blending of proven sources and pushing data quality to its limit through rigorous preprocessing and deduplication.
For a mix you can start with today, we recommend combining FineWeb for general web knowledge, Dolma or SlimPajama for technical documents, and The Stack v2 for code.
Recent 2025 case law suggests that training on lawfully acquired data is likely to qualify as "fair use," but pirated repositories such as Books3 and LibGen carry serious legal risk and must be avoided at all costs.
Related Downloads
Legally Defensible Data Curation Strategy Guide
Download the full legal and strategy guide (PDF) for training LLMs in AI and data science.
Resource 1 (Coming Soon)
A new guide or report is in the works and will be posted here soon.
Resource 2 (Coming Soon)
A new guide or report is in the works and will be posted here soon.
Disclaimer
This research was conducted as part of Pebblous's AADS (Agentic AI Data Scientist) technology development program. AI was used in the underlying research and in producing this web page.