Llama 3 8B: Training Notes
Project record. Background on the base model under llama3-8b-darkidol-2.0-uncensored-q4_k_m-imat.gguf. Compiled 31 October 2026 from Meta's published material (sources below).
Summary
Meta released Llama 3 8B on 18 April 2024. It is a small model that was trained for an unusually long time on heavily filtered data, and that is the main documented reason it performs well for its size.
Key facts
| Item | Detail |
|---|---|
| Parameters | 8 billion (8.03B) |
| Pretraining data | Over 15 trillion tokens from publicly available sources, about seven times Llama 2's data, including four times as much code |
| Non-English data | About 5% of pretraining data, covering more than 30 languages |
| Knowledge cutoff | March 2023 |
| Context window | 8,192 tokens |
| Tokenizer | 128K-token vocabulary (up from 32K in Llama 2), encoding the same text in up to about 15% fewer tokens |
| Attention | Grouped query attention (GQA), which reduces memory use during inference |
| Training compute | About 1.3 million H100-80GB GPU-hours |
| Post-training | Supervised fine-tuning, rejection sampling, PPO and DPO, using over 10 million human-annotated examples |
Why the 8B model performs above its size
Data volume. Earlier scaling research (Chinchilla, 2022) put the compute-optimal training data for an 8B model at roughly 200 billion tokens. Meta trained on more than 15 trillion, about 75 times that. Meta reported that the 8B model was still improving roughly log-linearly at 15 trillion tokens. Training a small model far past the "optimal" point costs more at training time but gives a better model at inference time, which suits local use.
Data quality. Meta built a filtering pipeline using heuristic filters, NSFW filters, semantic deduplication, and text-quality classifiers. Llama 2 was used to generate the training data for those quality classifiers.
Tokenizer efficiency. The larger vocabulary packs more text into each token, so the model sees more content for the same compute and context length.
Relevance to DarkIdol
Almost all of a model's factual knowledge comes from pretraining. DarkIdol is a community merge of roleplay and creative-writing fine-tunes on top of Llama 3 8B (made by Hugging Face user aifeifei798). The merge mainly changes style and behaviour: conversational tone, story writing and removal of refusals. The knowledge base is Meta's 15-trillion-token pretraining.
Sources
Meta AI, "Introducing Meta Llama 3: The most capable openly available LLM to date," 18 April 2024: ai.meta.com/blog/meta-llama-3/
Llama 3 model card: github.com/meta-llama/llama3/blob/main/MODEL_CARD.md
Llama Team, AI @ Meta, "The Llama 3 Herd of Models," July 2024: arxiv.org/abs/2407.21783
Hoffmann et al., "Training Compute-Optimal Large Language Models" (Chinchilla), 2022: arxiv.org/abs/2203.15556