Chapter 2 Pre-training and Scaling Large Language Models

After scoping a use case, the next lifecycle decision (Section 1.7) is which model to build on. Almost always the answer is an existing pre-trained model. This chapter explains why, covering:

  • how pre-training works;
  • why training large models strains GPU memory, and the techniques that make it feasible (number formats and distributed training);
  • the scaling laws that link model size, data, and compute; and
  • the narrow case where pre-training your own domain model can be worth it.

2.1 Choosing a starting model

Open models are distributed through model hubs such as Hugging Face. Each model ships with a model card (Mitchell et al. 2019). The card records the intended uses, training data and procedure, evaluation results, and known limitations. Read it before anything else. It tells you whether the model’s architecture, licence, languages, and context length fit your task.

In short.

  • Start from an existing model on a hub, and read its model card first: intended use, data, evaluations, limits, licence.

2.2 Pre-training objectives and architecture families

Pre-training is self-supervised learning on a very large text corpus. The labels come from the text itself, so no human annotation is needed. What the model learns to do depends on its training objective, and the objective is tied to its architecture:

Family Pre-training objective What the model sees Good at Examples
Encoder-only (autoencoding) Masked language modelling: predict randomly masked tokens Both sides of each token (bidirectional) Classification, named-entity recognition, embeddings BERT (Devlin et al. 2019), RoBERTa (Liu et al. 2019)
Decoder-only (autoregressive) Causal language modelling: predict the next token Only earlier tokens (left-to-right) Text generation; at scale, zero-shot and few-shot on many tasks GPT (Brown et al. 2020), BLOOM, Llama
Encoder–decoder (sequence-to-sequence) Varies. T5 uses span corruption: masked spans are replaced by sentinel tokens, and the decoder reconstructs them Encoder: whole input; decoder: earlier outputs Translation, summarization, input-to-output tasks T5 (Raffel et al. 2020), BART

The two main losses, for a token sequence \(x_1, \dots, x_T\):

\[ \mathcal{L}_{\text{causal}} = -\sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t}), \qquad \mathcal{L}_{\text{masked}} = -\sum_{t \in \mathcal{M}} \log p_\theta(x_t \mid x_{\setminus \mathcal{M}}) \]

Here \(\mathcal{M}\) is the set of masked positions. BERT masks about 15% of tokens (Devlin et al. 2019).

Example: one sentence, three objectives. Take “The spring campaign lifted online sales by twelve percent.”

Objective Input the model sees What it must predict
Masked LM (BERT) The spring campaign [MASK] online sales by twelve [MASK]. “lifted” and “percent”, using words on both sides
Causal LM (GPT) The spring campaign lifted online “sales”, then each later word in turn, seeing only what came before
Span corruption (T5) The spring campaign <X> by twelve percent. <X> lifted online sales

The data are identical; the objective decides whether the model learns to fill gaps from both sides, to continue text, or to rebuild missing spans.

Today’s general-purpose LLMs are almost all decoder-only. Encoder models remain the workhorses for classification and embeddings.

Data. Pre-training corpora contain trillions of tokens drawn from web crawls, books, code, and curated collections. Raw web text needs heavy processing: language identification, quality filtering, deduplication, removal of personal data, and toxicity filtering. This usually discards a large share of the raw crawl. If you ever plan to pre-train, budget data collection for that loss. Filtering reduces some risks but does not guarantee unbiased or safe data.

In short.

  • Pre-training is self-supervised: the text supplies its own labels.
  • Masked LM (encoders) for understanding and embeddings; causal LM (decoders) for generation; span corruption (encoder–decoders) for input-to-output tasks.
  • Corpora have trillions of tokens, after heavy filtering and deduplication.

2.3 Why training does not fit on one GPU

Storing a model is cheap compared with training it. At 32-bit precision each parameter takes 4 bytes, so 1 billion parameters need 4 GB just for the weights. Training also needs, for every parameter:

Component FP32 training (bytes/param) Mixed-precision training (bytes/param)
Weights used in forward/backward 4 2 (16-bit copy)
Gradients 4 2
Adam optimizer: momentum + variance 8 8 (kept in FP32)
FP32 master copy of weights — 4
Total, before activations 16 16

Activations come on top. These are the intermediate outputs saved for the backward pass, and they grow with batch size and sequence length. Activation checkpointing saves memory by storing only some activations and recomputing the rest during the backward pass (Chen et al. 2016).

Rules of thumb:

  • Training needs roughly 4–6× the memory of the FP32 weights. That is about 16 GB for a 1B-parameter model before activations, and around 24 GB with a typical allowance for activations.
  • A 175B-parameter model needs about 2.8 TB for weights, gradients, and optimizer states alone. No single GPU comes close.
  • Fine-tuning has the same profile when all weights are updated. That is the main motivation for PEFT (Section 3.5).

Two levers reduce the problem: smaller number formats (next section) and splitting the work across GPUs (the section after).

In short.

  • Training needs about 16 bytes per parameter before activations: weights, gradients, and Adam states.
  • A 7B model therefore needs about 112 GB, more than one 80 GB GPU, and a 175B model about 2.8 TB.
  • Full fine-tuning has the same profile, which is why PEFT exists.

2.4 Number formats and quantization

A floating-point number has three parts: a sign bit, exponent bits that set the range, and fraction (mantissa) bits that set the precision.

Format Sign / exponent / fraction bits Bytes Largest value \(\pi\) stored as
FP32 1 / 8 / 23 4 \(\approx 3.4 \times 10^{38}\) 3.1415927
FP16 1 / 5 / 10 2 65,504 3.140625
BF16 1 / 8 / 7 2 \(\approx 3.4 \times 10^{38}\) 3.140625
INT8 8-bit integer, \(-128\) to \(127\) 1 Set by a scale factor Depends on the scale (see below)

FP16 vs. BF16. Both use 2 bytes and, for \(\pi\), give the same rounded value. The difference is range. FP16 overflows above 65,504, so FP16 training needs loss scaling to keep small gradients from underflowing to zero (Micikevicius et al. 2018). BF16 (brain floating point, from Google Brain) keeps FP32’s 8 exponent bits. It trades precision for the full FP32 range, which makes training more stable. That is why it became the default 16-bit training format on hardware that supports it.

Mixed-precision training runs the forward and backward passes in 16-bit but keeps an FP32 master copy of the weights for the optimizer update (Micikevicius et al. 2018). This gives most of the speed and activation-memory benefit of 16-bit arithmetic while preserving update accuracy.

Integer quantization maps real values to integers through a scale factor \(s\). With symmetric quantization, \(q = \text{round}(x / s)\) and \(s = \max|x| / 127\). Precision therefore depends on the range of the values being quantized. For example, if the largest weight magnitude in a tensor is 4.0, then \(s = 4/127 \approx 0.0315\). \(\pi\) is stored as \(q = 100\) and recovered as \(100 \times 0.0315 \approx 3.150\). Choosing \(s\) well is the central problem. Quantization-aware training learns the scale factors during training. Post-training quantization estimates them afterwards (Section 5.1.2).

Current practice (October 2026). BF16 mixed precision is the baseline for training. Recent accelerators also support 8-bit floating-point (FP8) formats. Large models have been trained with FP8 mixed precision, for example DeepSeek-V3 (DeepSeek-AI 2024). For inference, 8-bit and 4-bit weight quantization are routine.

In short.

  • Exponent bits set range; fraction bits set precision.
  • BF16 keeps FP32’s range at half the size, so it trains stably; FP16 needs loss scaling.
  • Mixed precision computes in 16-bit but updates an FP32 master copy; integer quantization depends on a well-chosen scale factor.

2.5 Distributed training

Multiple GPUs serve two purposes: speed (process more data in parallel) and capacity (fit a model that one GPU cannot hold).

2.5.1 Data parallelism (DDP)

Distributed data parallel (PyTorch DDP) puts a full copy of the model on every GPU and gives each GPU a different slice of every batch. After the backward pass, gradients are averaged across GPUs (an all-reduce), so every copy applies the same update and stays identical. DDP speeds up training almost linearly, but each GPU must hold the full model, its gradients, and its optimizer states. It adds speed, not capacity.

2.5.2 Sharding: ZeRO and FSDP

Under DDP, every GPU stores identical copies of the same states, which is pure redundancy. ZeRO (Zero Redundancy Optimizer) (Rajbhandari et al. 2020) shards these states across the \(N_d\) data-parallel GPUs instead, in three progressive stages. With \(\Psi\) parameters and mixed-precision Adam (2 + 2 + 12 bytes per parameter), the memory per GPU is:

Stage What is sharded Memory per GPU 7.5B params, 64 GPUs
Baseline (DDP) Nothing \(16\Psi\) 120 GB
ZeRO-1 Optimizer states \(4\Psi + 12\Psi / N_d\) 31.4 GB
ZeRO-2 + gradients \(2\Psi + 14\Psi / N_d\) 16.6 GB
ZeRO-3 + parameters \(16\Psi / N_d\) 1.9 GB

Stages 1 and 2 cut memory by up to about 4× and 8×. Stage 3 cuts it in proportion to the number of GPUs. The example column is the paper’s own illustration.

Fully Sharded Data Parallel (PyTorch FSDP) (Zhao et al. 2023) implements the same idea natively in PyTorch:

  1. Before computing a layer, each GPU gathers that layer’s full parameters from the other GPUs (all-gather).
  2. It runs the forward or backward computation, then frees the gathered copy.
  3. After the backward pass, gradients are reduced and re-sharded (reduce-scatter), so each GPU updates only its own shard.

The cost is extra communication. FSDP exposes this trade-off as a sharding strategy:

  • Full sharding: most memory saved, most communication.
  • No sharding: equivalent to DDP.
  • Hybrid sharding: shard within a machine and replicate across machines, keeping heavy traffic on fast local links.
  • CPU offload: parameters or optimizer states can be moved to CPU memory, at a further speed cost.

2.5.3 Model parallelism

Sharding still runs each layer whole on one GPU at a time. The largest models also split the computation itself:

  • Tensor parallelism splits individual weight matrices across GPUs. This is the Megatron-LM approach (Shoeybi et al. 2019). It needs very fast interconnects, so it is usually kept within one machine.
  • Pipeline parallelism places different layers on different GPUs and streams micro-batches through them to keep all stages busy (Huang et al. 2019).

Frontier training combines data, tensor, and pipeline parallelism (“3D parallelism”) with sharding. As GPU count grows, communication overhead lowers the throughput per GPU. That is a real cost when planning a training run.

Which strategy when:

Situation Usual choice
The model, gradients, and optimizer states fit on one GPU; you want speed DDP
They do not fit on one GPU, but each layer does FSDP or ZeRO (stage 2 or 3)
A single layer’s computation is too large, or the model is very large Add tensor parallelism within a machine and pipeline parallelism across machines
Fine-tuning a large model on few GPUs PEFT (LoRA or QLoRA) first; shard only if still needed

In short.

  • DDP copies the model to every GPU: more speed, no more capacity.
  • ZeRO and FSDP shard optimizer states, gradients, and parameters: capacity at a communication cost.
  • Tensor and pipeline parallelism split the computation itself; frontier runs combine all three.

2.6 Scaling laws

Pre-training minimizes next-token loss. Three quantities determine how low the loss goes: model size \(N\) (parameters), dataset size \(D\) (training tokens), and compute \(C\). For dense transformers, training compute is approximately

\[ C \approx 6ND \quad \text{FLOPs.} \]

Large budgets are often quoted in petaFLOP/s-days. One petaFLOP/s-day is \(10^{15}\) operations per second sustained for a day, which is \(8.64 \times 10^{19}\) FLOPs. GPT-3 175B, trained on 300B tokens, used about \(3.14 \times 10^{23}\) FLOPs, or roughly 3,640 petaFLOP/s-days (Brown et al. 2020). That matches \(6 \times 175\text{B} \times 300\text{B}\).

2.6.1 Kaplan et al.: loss follows power laws

Kaplan et al. (2020) found that test loss falls as a power law in each of \(N\), \(D\), and \(C\), provided the other two are not the bottleneck. For example, \(L(N) \approx (N_c / N)^{\alpha_N}\) with \(\alpha_N \approx 0.076\). On log–log axes these are straight lines over many orders of magnitude. That makes loss predictable: small, cheap runs forecast the loss of large ones.

In practice compute is the hard constraint (hardware, time, money). The practical question is therefore: for a fixed compute budget, how should it be split between a bigger model and more data? Kaplan et al. recommended putting most extra compute into model size. That reasoning shaped the very large models of 2020–21.

2.6.2 Chinchilla: scale data and parameters together

Hoffmann et al. (2022) revisited the question with more careful experiments, including learning-rate schedules matched to each run’s length. They fitted the loss as

\[ L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}. \]

Here \(E\) is the irreducible loss of the data, and the two other terms are the penalties for finite model size and finite data. Minimizing this subject to \(C \approx 6ND\) gives the main result: \(N\) and \(D\) should grow in roughly equal proportion, which works out to about 20 training tokens per parameter. This is a historical empirical estimate for their model family and setup, not a universal constant.

Example: sizing a model for a budget. With \(C = 6ND\) and \(D = 20N\), compute is \(C = 120N^2\), so \(N = \sqrt{C / 120}\). For a budget of \(10^{23}\) FLOPs, \(N \approx 2.9 \times 10^{10}\) (about 29B parameters) and \(D \approx 580\)B tokens. A team with that budget that instead trained a 175B model would get only about 95B tokens, far short of the roughly 3.5T that size would need.

The test: Chinchilla (70B parameters, 1.4T tokens) used the same compute as Gopher (280B parameters, 300B tokens). It outperformed Gopher, GPT-3, and other much larger models on a wide range of downstream tasks. Measured against the 20× rule, many earlier models were over-parameterized and under-trained:

Model Parameters Chinchilla-optimal tokens (~20×) Actual training tokens
GPT-3 (Brown et al. 2020) 175B ~3.5T 300B
BLOOM 176B ~3.5T ~366B
Chinchilla (Hoffmann et al. 2022) 70B ~1.4T 1.4T
LLaMA-65B 65B ~1.3T 1.4T

Interpreting scaling laws:

  • They predict pre-training loss, not specific downstream abilities. Individual skills can improve unevenly.
  • They hold for the architecture, data mixture, and recipe they were fitted on. Change those and you must re-fit.
  • “Compute-optimal” optimizes training compute only. It ignores the cost of serving the model afterwards.

Current practice (October 2026). The last point changed practice. A model that will serve billions of requests is cheaper overall if it is smaller and trained on far more than 20 tokens per parameter (Sardana et al. 2024). Llama 3’s 8B model, for example, was trained on about 15T tokens, close to 1,900 tokens per parameter (Llama Team, AI @ Meta 2024). “Over-training” small models is now normal. Chinchilla remains the reference point for the training-compute trade-off.

In short.

  • Training compute is about \(6ND\) FLOPs; loss falls as a power law in parameters, data, and compute.
  • Chinchilla: for a fixed training budget, grow parameters and tokens together, about 20 tokens per parameter.
  • Serving costs now favour smaller models trained on far more tokens.

2.7 Domain adaptation and pre-training your own model

You rarely need to pre-train. The exception is a domain whose language is poorly represented in general corpora:

  • Rare terms: legal terms such as mens rea and res judicata.
  • Everyday words with special meanings: consideration in contract law means the value exchanged that makes an agreement binding, not kindness.
  • Shorthand: a prescription note like “1 tab po qid pc & hs” means one tablet by mouth four times a day, after meals and at bedtime.

A model learns its vocabulary and representations in pre-training. Later stages build on those, so a domain the model barely saw may be handled poorly. Before committing to from-scratch training, climb this ladder of options, from cheapest to most expensive:

  1. Prompting and retrieval (Chapter 5).
  2. Fine-tuning or PEFT on domain tasks (Chapter 3).
  3. Continued (domain-adaptive) pre-training. Keep pre-training an existing model on unlabelled domain text. Gururangan et al. (2020) showed consistent downstream gains from this across biomedical, computer-science, news, and review domains.
  4. Pre-training from scratch on a mixed domain and general corpus.

2.7.1 Case study: BloombergGPT

BloombergGPT (S. Wu et al. 2023) is the best-known from-scratch domain model:

  • Model and data. A 50.6B-parameter decoder trained on a corpus of about 51% financial text (Bloomberg’s own and public financial data) and 49% general public text. The general data preserved broad language ability.
  • Compute. About 1.3 million A100 GPU-hours.
  • Scaling choices. The authors used Chinchilla scaling laws to size the model for their budget, and the parameter count landed close to compute-optimal. The training tokens (569B) fell short of the compute-optimal amount, because there was not enough high-quality financial text. Real data limits forced a trade-off.
  • Result. The paper reports that BloombergGPT outperformed similarly sized open models on financial tasks by clear margins while staying competitive on general-purpose benchmarks.

The lesson is that from-scratch domain pre-training is justified only when the domain language is distinctive, high-quality domain data exists at scale, and the compute budget is large.

Current practice (October 2026). Given how capable general models now are, combined with continued pre-training, fine-tuning, and retrieval, from-scratch domain pre-training is a rare choice. Justify it with an evaluation showing that the cheaper options fall short.

In short.

  • Pre-train only when the domain’s language is distinctive, high-quality domain data exist at scale, and the budget is large.
  • First climb the cheaper ladder: prompting and retrieval, fine-tuning, continued pre-training.
  • BloombergGPT shows the trade-offs: Chinchilla-sized, but data-limited.

2.8 Key takeaways

The section summaries above hold the details. The chapter in five lines:

  • Start: from an existing model; the objective (masked, causal, span) matches the architecture and the task.
  • Memory: about 16 bytes per parameter to train, so large models need BF16, mixed precision, and sharding.
  • Distribution: DDP for speed; ZeRO/FSDP for capacity; tensor and pipeline parallelism for the largest models.
  • Scaling: loss follows power laws; Chinchilla’s ~20 tokens per parameter for training compute, more when serving costs count.
  • Domain models: the expensive end of a ladder; justify them with evidence that cheaper options fall short.

2.9 Interview checkpoints

  • Masked vs. causal language modelling? Masked LM predicts hidden tokens using both sides of the context. Causal LM predicts the next token using only earlier ones.
  • How much GPU memory does training a 7B-parameter model need? With mixed precision and Adam, about 16 bytes × 7B ≈ 112 GB before activations. That calls for sharding, or PEFT for fine-tuning.
  • Why BF16 over FP16? The same 8-bit exponent as FP32 gives the full range, so there is less overflow or underflow and no need for loss scaling. The cost is lower precision.
  • DDP vs. FSDP? DDP replicates everything and adds speed only. FSDP shards parameters, gradients, and optimizer states, which adds capacity at a communication cost.
  • The three ZeRO stages? Shard optimizer states; then also gradients; then also parameters.
  • What is \(C \approx 6ND\)? The approximate training FLOPs for a dense transformer with \(N\) parameters trained on \(D\) tokens.
  • The Chinchilla result in one line? For a fixed training-compute budget, scale parameters and tokens together, at about 20 tokens per parameter. Many earlier models were under-trained.
  • Why train beyond Chinchilla-optimal? Serving costs dominate over a model’s life. A smaller model trained on more data is cheaper to run at the same quality.
  • When would you pre-train a domain model? When the domain language is distinctive, domain data is plentiful, the budget is large, and prompting, RAG, fine-tuning, and continued pre-training have been shown to fall short.

References

Brown, Tom B., Benjamin Mann, Nick Ryder, et al. 2020. “Language Models Are Few-Shot Learners.” Advances in Neural Information Processing Systems. https://arxiv.org/abs/2005.14165.
Chen, Tianqi, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. “Training Deep Nets with Sublinear Memory Cost.” arXiv Preprint arXiv:1604.06174. https://arxiv.org/abs/1604.06174.
DeepSeek-AI. 2024. “DeepSeek-V3 Technical Report.” arXiv Preprint arXiv:2412.19437. https://arxiv.org/abs/2412.19437.
Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.” Proceedings of NAACL-HLT 2019. https://arxiv.org/abs/1810.04805.
Gururangan, Suchin, Ana Marasović, Swabha Swayamdipta, et al. 2020. “Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks.” Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. https://arxiv.org/abs/2004.10964.
Hoffmann, Jordan, Sebastian Borgeaud, Arthur Mensch, et al. 2022. “Training Compute-Optimal Large Language Models.” Advances in Neural Information Processing Systems. https://arxiv.org/abs/2203.15556.
Huang, Yanping, Youlong Cheng, Ankur Bapna, et al. 2019. “GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism.” Advances in Neural Information Processing Systems. https://arxiv.org/abs/1811.06965.
Kaplan, Jared, Sam McCandlish, Tom Henighan, et al. 2020. “Scaling Laws for Neural Language Models.” arXiv Preprint arXiv:2001.08361. https://arxiv.org/abs/2001.08361.
Liu, Yinhan, Myle Ott, Naman Goyal, et al. 2019. “RoBERTa: A Robustly Optimized BERT Pretraining Approach.” arXiv Preprint arXiv:1907.11692. https://arxiv.org/abs/1907.11692.
Llama Team, AI @ Meta. 2024. “The Llama 3 Herd of Models.” arXiv Preprint arXiv:2407.21783. https://arxiv.org/abs/2407.21783.
Micikevicius, Paulius, Sharan Narang, Jonah Alben, et al. 2018. “Mixed Precision Training.” International Conference on Learning Representations. https://arxiv.org/abs/1710.03740.
Mitchell, Margaret, Simone Wu, Andrew Zaldivar, et al. 2019. “Model Cards for Model Reporting.” Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*). https://arxiv.org/abs/1810.03993.
Raffel, Colin, Noam Shazeer, Adam Roberts, et al. 2020. “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.” Journal of Machine Learning Research 21 (140): 1–67. https://arxiv.org/abs/1910.10683.
Rajbhandari, Samyam, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models.” SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. https://arxiv.org/abs/1910.02054.
Sardana, Nikhil, Jacob Portes, Sasha Doubov, and Jonathan Frankle. 2024. “Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws.” Proceedings of the 41st International Conference on Machine Learning. https://arxiv.org/abs/2401.00448.
Shoeybi, Mohammad, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.” arXiv Preprint arXiv:1909.08053. https://arxiv.org/abs/1909.08053.
Wu, Shijie, Ozan Irsoy, Steven Lu, et al. 2023. “BloombergGPT: A Large Language Model for Finance.” arXiv Preprint arXiv:2303.17564. https://arxiv.org/abs/2303.17564.
Zhao, Yanli, Andrew Gu, Rohan Varma, et al. 2023. “PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel.” Proceedings of the VLDB Endowment 16 (12). https://arxiv.org/abs/2304.11277.