Chapter 1 Foundations of Generative AI and LLMs
This part of the book builds an understanding of large language models from the ground up, following the lifecycle of an LLM project:
| Chapter | Question it answers |
|---|---|
| This chapter | What is an LLM, how does a transformer generate text, and how do prompts and generation settings shape the output? |
| Chapter 2 | How are LLMs pre-trained, and what do model size, data, and compute buy? |
| Chapter 3 | How do we adapt a model to our tasks, efficiently, and measure the result? |
| Chapter 4 | How do we make a model helpful, honest, and harmless? |
| Chapter 5 | How do we serve a model and connect it to data and tools responsibly? |
These chapters began as study notes from the DeepLearning.AI / AWS course Generative AI with Large Language Models (Coursera). They have since been rewritten as a reference, checked against the primary papers cited in each chapter, and extended with later developments.
1.1 What generative AI is
Generative models learn the statistical structure of their training data and produce new samples from it: text, images, audio, video, or code. This book focuses on large language models (LLMs): neural networks trained on very large text corpora to predict the next token. Chatbots, code assistants, and writing tools are applications built on top of that one capability.
Key terms:
- Foundation model. A model trained broadly enough to be adapted to many downstream tasks. Examples include BERT, GPT, T5/FLAN-T5, BLOOM, and Llama. Its capabilities must be measured on each task; they cannot be assumed from its size.
- Parameters. The learned weights, currently up to hundreds of billions for large models. Parameters encode distributed statistical patterns, not a database of retrievable facts. More parameters add capacity, but data, training recipe, and adaptation matter as much.
- Prompt, completion, inference. The text you send is the prompt. Producing text is inference. The output is the completion.
- Context window. The maximum number of tokens the model can attend to at once: instructions, conversation, documents, tool results, and its own output. The limit varies by model; check current documentation.
You interact with an LLM in natural language, not in a formal programming language. You can use it as is, or adapt it (Chapter 3) without training a new model.
1.2 What LLMs are used for
Many tasks reduce to next-token prediction under the right prompt:
- Generation: drafting essays, emails, or marketing copy.
- Summarization: condensing dialogues, documents, or reviews.
- Translation: between natural languages, or from natural language to code or SQL.
- Information extraction: pulling named entities (people, places, organizations), fields, or labels out of text.
- Augmented tasks: answering from documents or calling APIs. These are not abilities of the model alone; the surrounding application supplies the data and tools (Chapter 5).
Larger models generally handle more tasks without examples. Smaller or specialized models can still be the better choice on cost, latency, privacy, or a narrow task. Choose by evaluating on your own use case.
1.3 Before transformers: recurrent networks
Earlier language models used recurrent neural networks (RNNs). An RNN reads tokens one at a time and carries a hidden state forward, so in principle it summarizes everything seen so far. In practice it has three problems:
- Long-range dependencies. Gradients shrink (or explode) as they flow back through many time steps, so plain RNNs struggle to learn from distant context (Bengio et al. 1994). LSTMs reduced but did not remove the problem (Hochreiter and Schmidhuber 1997).
- Sequential computation. Step \(t\) needs the result of step \(t-1\). Training cannot be parallelized across positions, which limits model and data scale.
- A fixed-size bottleneck. The whole history must be squeezed into one state vector.
Language needs wide context. Words have several meanings: a bank can be a financial institution or a riverside. Sentences can be syntactically ambiguous: in “The teacher taught the students with the book”, whose book is it? Resolving such cases often requires the whole sentence or document.
Attention was the fix. It was first added to RNN translation models: the decoder learned to look back at all encoder states and weight the most relevant ones (Bahdanau et al. 2015). The transformer took the next step and dropped recurrence entirely.
1.4 The transformer
“Attention Is All You Need” (Vaswani et al. 2017) introduced the transformer. It is built from attention and position-wise feed-forward layers, with no recurrence. Two properties explain its success:
- Parallelism. All positions in a sequence are processed at once. That makes training efficient on GPUs and allows much larger models and datasets.
- Direct context. Every token can attend to every other token in one step, however far apart they are.
1.4.1 From text to vectors
- Tokenization. Text is split into tokens from a fixed vocabulary. Tokens are usually subword units learned by algorithms such as byte-pair encoding (Sennrich et al. 2016), so rare words split into pieces. Each token maps to an integer ID. A model must always be used with the tokenizer it was trained with.
- Embedding. Each token ID is mapped to a learned vector of dimension \(d_{\text{model}}\) (512 in the original transformer). Tokens with related meanings end up close together in this space.
- Positional information. Attention by itself ignores order, so position must be added. The original paper added fixed sinusoidal positional encodings to the embeddings. Many modern LLMs instead use rotary position embeddings (RoPE), which encode relative position inside the attention computation (Su et al. 2024).
1.4.2 Self-attention
Each token’s vector \(x_i\) is projected into three vectors with learned matrices. The query \(q_i = x_i W^Q\) says what the token is looking for. The key \(k_j = x_j W^K\) says what each token offers. The value \(v_j = x_j W^V\) is what gets passed along if the token is attended to. Stacking these vectors into matrices \(Q, K, V\) gives scaled dot-product attention:
\[ \text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{Q K^\top}{\sqrt{d_k}}\right) V \]
- \(QK^\top\) scores how relevant every token is to every other token. The softmax turns each row of scores into weights that sum to one; plotted as a grid, these weights form an attention map. Each output is a weighted average of the value vectors.
- Why divide by \(\sqrt{d_k}\)? Dot products grow with the vector dimension \(d_k\). Large scores push the softmax towards a one-hot output, where gradients become very small. Scaling keeps training stable.
- Multi-head attention runs \(h\) attention operations in parallel, each with its own projections (the original used \(h = 8\) heads of size \(d_k = 64\)), and concatenates the results. Different heads learn different relations, for example syntactic agreement, coreference, or topic. Which relation a head learns is not chosen in advance.
- Cost. The score matrix is \(n \times n\) for a sequence of \(n\) tokens, so attention compute and memory grow quadratically with context length. That is why long contexts are expensive.
1.4.3 The rest of a transformer block
Each layer (block) wraps attention with:
- a position-wise feed-forward network: two linear layers with a nonlinearity, applied to each token separately (inner width 2,048 in the original);
- residual connections and layer normalization around both sub-layers, which make deep stacks trainable.
The original model stacked six such layers in the encoder and six in the decoder. A final linear layer maps each output vector to one logit per vocabulary token, and a softmax turns the logits into a probability distribution over the next token.
1.4.4 Encoder, decoder, and the three model families
- The encoder reads the whole input with bidirectional self-attention. It produces a contextual representation of each input token.
- The decoder generates output one token at a time. Its self-attention is masked (causal): each position sees only earlier positions, so the model cannot peek at the token it is predicting. In an encoder–decoder model, the decoder also uses cross-attention, where its queries attend over the encoder’s outputs.
Using one or both halves gives three families: encoder-only (BERT), decoder-only (GPT, Llama), and encoder–decoder (T5, BART). They differ in pre-training objective and in which tasks they suit. Chapter 2 compares them in detail.
1.4.5 Generating text, end to end
Take translation with an encoder–decoder model, from “J’aime l’apprentissage automatique” to “I love machine learning”:
- Tokenize the French sentence with the model’s tokenizer.
- The encoder processes all input tokens in parallel into contextual representations.
- The decoder starts from a start-of-sequence token. Attending to its own previous outputs and (through cross-attention) to the encoder’s representations, it produces a distribution over the next token, and one token is chosen.
- The chosen token is appended and fed back in. Step 3 repeats until an end-of-sequence token is produced or a length limit is reached.
- Detokenize the output tokens into text.
A decoder-only model works the same way without the encoder. The prompt and the tokens generated so far form one sequence, and the model extends it one token at a time. This loop is autoregressive generation. How the next token is chosen from the distribution is set by the generation configuration (Section 1.6).
1.4.6 Landmark models
| Model | Year | Family | Size | Why it matters |
|---|---|---|---|---|
| BERT (Devlin et al. 2019) | 2018 | Encoder | 110M (Base), 340M (Large) | Bidirectional pre-training with masked language modelling; set the pattern of pre-train then fine-tune for understanding tasks |
| T5 (Raffel et al. 2020) | 2019 | Encoder–decoder | Up to 11B | Framed every task as text-to-text; FLAN-T5 is its instruction-tuned version |
| GPT-3 (Brown et al. 2020) | 2020 | Decoder | 175B | Showed strong few-shot in-context learning at scale |
| BLOOM (BigScience Workshop 2022) | 2022 | Decoder | 176B | Open-science, multilingual (46 natural and 13 programming languages) |
| LLaMA (Touvron et al. 2023) | 2023 | Decoder | 7B–65B | Smaller models trained on more tokens; led a wave of open-weight models |
Model families move quickly. Use this table for the historical milestones, and check current model cards for anything newer.
1.5 Prompting and in-context learning
Prompt engineering is designing the input so the model produces the desired completion. Its most important effect, discovered at scale, is in-context learning (ICL): the model picks up a task from examples placed in the prompt, with no weight updates (Brown et al. 2020).
- Zero-shot: an instruction and the input, no examples. For example, “Classify this review: … Sentiment:”.
- One-shot: one solved example before the real input.
- Few-shot: several solved examples, ideally covering the label space (for example, both a positive and a negative review).
What to know:
- ICL depends on scale. Large models often do well zero-shot. Small models may fail even with several examples.
- Examples cost context. They use up the context window and add tokens to every call.
- There is a ceiling. If a handful of examples does not get the performance you need, more examples rarely help. That is the signal to fine-tune (Chapter 3).
The Prompt Engineering part of this book (starting with Chapter 6) covers prompt structure, techniques, and evaluation in depth.
1.6 Configuring generation
At each step the model outputs a probability distribution over the vocabulary. Inference-time settings (as opposed to training parameters) decide how the next token is picked from it, and when to stop.
- Max new tokens. A cap on the number of generated tokens. Generation can stop earlier at an end-of-sequence token.
- Greedy decoding. Always pick the most probable token. It is deterministic and fine for short factual outputs, but prone to repetitive loops in longer text. (Beam search, which keeps the best few partial sequences, is a deterministic alternative common in translation.)
- Random sampling. Draw the next token in proportion to its probability.
This adds variety but can wander into low-probability, incoherent tokens.
In Hugging Face Transformers, sampling is enabled with
do_sample=True. - Top-k sampling. Sample only from the \(k\) most probable tokens, renormalized.
- Top-p (nucleus) sampling. Sample from the smallest set of tokens whose cumulative probability reaches \(p\). The candidate set adapts to how confident the model is (Holtzman et al. 2020).
- Temperature. Rescale the logits before the softmax: \(p_i \propto \exp(z_i / T)\). \(T < 1\) sharpens the distribution (more predictable). \(T > 1\) flattens it (more varied). \(T = 1\) leaves it unchanged.
Worked example. Suppose the next-token logits are apple \(= -1.0\), banana \(= 0.5\), cake \(= 2.0\), and donut \(= 1.2\).
| Next token | \(T = 0.5\) | \(T = 1.0\) | \(T = 2.0\) |
|---|---|---|---|
| apple | 0.002 | 0.029 | 0.094 |
| banana | 0.040 | 0.130 | 0.200 |
| cake | 0.797 | 0.581 | 0.423 |
| donut | 0.161 | 0.261 | 0.283 |
At \(T = 1\):
- Top-k with \(k = 3\) keeps cake, donut, and banana and renormalizes them to 0.598, 0.269, and 0.133.
- Top-p with \(p = 0.8\) keeps cake and donut, the smallest set whose cumulative probability (0.842) reaches 0.8, and renormalizes them to 0.690 and 0.310.
Apple can never be sampled under either filter.
In practice, use low temperature or greedy decoding for extraction, classification, and code. Use moderate temperature with top-p for open-ended writing. Hosted APIs may expose only some of these settings, and reasoning models may fix them.
1.7 The generative AI project lifecycle
An LLM project moves through four stages, and later results often send you back to an earlier one:
| Stage | Key decisions | Covered in |
|---|---|---|
| Scope | Define the use case narrowly: which tasks, which inputs, what counts as success | This section |
| Select | Use an existing foundation model (the usual choice) or pre-train your own | Chapter 2 |
| Adapt and align | Prompting first, then fine-tuning or PEFT, then alignment with human feedback; evaluate at every step | Chapters 3 and 4 |
| Integrate | Optimize for inference; add retrieval and tools to overcome model limits; deploy, monitor, and manage risks | Chapter 5 |
Scoping matters most. A model that must do many things needs to be larger and more general than one that only extracts named entities from invoices. A narrow scope makes model choice, evaluation, and risk assessment easier and cheaper.
The inner loop is adapt → evaluate → adapt. Start with prompt engineering and in-context learning. Measure against a held-out evaluation set. Fine-tune only if prompting falls short. Then return to prompting with the tuned model.
Even a well-adapted model invents information when unsure and is unreliable at exact computation. It also knows nothing after its training cut-off. Production systems therefore add retrieval, tools, and verification around the model; see Chapter 5.
1.8 Key takeaways
- An LLM is a next-token predictor trained on huge text corpora. Many tasks, from summarization to translation to extraction, are next-token prediction under the right prompt.
- RNNs struggled with long-range context and could not parallelize. Transformers replaced recurrence with attention, which made large-scale training practical.
- Scaled dot-product attention, \(\text{softmax}(QK^\top/\sqrt{d_k})V\), lets every token weight every other token. Multi-head attention learns several relations at once. The cost is quadratic in sequence length.
- Encoder (bidirectional), decoder (causal, autoregressive), and encoder–decoder (cross-attention) models suit different tasks.
- In-context learning (zero-, one-, few-shot) adapts behaviour without training but is limited by scale and context.
- Greedy, top-k, top-p, and temperature control the trade-off between determinism and diversity at inference.
- The lifecycle is scope → select → adapt and align → integrate, with evaluation throughout.
1.9 Interview checkpoints
- Why did transformers replace RNNs? Parallel training across positions, and direct attention between any two tokens instead of a long chain of recurrent steps.
- Write the attention formula. Why the \(\sqrt{d_k}\)? \(\text{softmax}(QK^\top/\sqrt{d_k})V\). Without scaling, large dot products saturate the softmax and shrink the gradients.
- What are Q, K, and V? Learned projections of each token: what it is looking for (query), what it offers to be matched on (key), and the content it passes on when attended to (value).
- Why multi-head attention? Several heads can capture different relations in parallel at the same total cost.
- Why does attention need positional information? Attention treats its inputs as an unordered set; positional encodings or RoPE inject word order.
- What does causal masking do? It stops each decoder position from seeing future tokens, so training matches left-to-right generation.
- What is the complexity of self-attention in sequence length? Quadratic, \(O(n^2)\).
- Top-k vs. top-p? Top-k keeps a fixed number of tokens. Top-p keeps a variable number whose cumulative probability reaches \(p\).
- What does temperature do? It divides the logits before the softmax. Lower values sharpen the distribution; higher values flatten it.
- Zero-shot vs. few-shot? When do you move to fine-tuning? Few-shot includes solved examples in the prompt. Move on when a handful of examples does not reach the target performance, or when context or cost limits bite.