Chapter 3 Instruction Fine-Tuning and Evaluation

1. Prompt Engineering and Inference:

   • Zero-shot: Models respond to prompts without examples.

   • One/Few-shot: Including example outputs improves performance, but it uses context window space and may not work for smaller models.

Drawbacks: - In context learning may not work for smaller LLM models

- Examples take up valuable space in the context window

2. Fine-tuning Basics:

   • Unlike pre-training on vast data using self-supervised learning, fine-tuning uses labeled examples (prompt-completion pairs) via supervised learning to optimize model weights for specific tasks.

   • Instruction fine-tuning trains models with explicit instruction-based examples for tasks (e.g., “Summarize the following text”).

   • Data sets of task-specific examples consists of PROMPT + COMPLETION couples

   • Full fine-tuning updates all parameters.

3. Instruction Fine-Tuning Process:

   • Prepare labeled data with templates (e.g., using Amazon reviews to create classification or summarization prompts).

   • There are prompt template libraries turn existing unstructured datasets into instruction prompt datasets for fine tuning

   • Split data into training, validation, and test sets.

   • Train the model using cross-entropy loss and backpropagation.

   • Evaluate using validation and test datasets for accuracy.

4. Outcome:

   • The result is an updated model (instruct model) fine-tuned for your desired tasks.

This approach is highly common in LLM optimization and essential for building task-specific solutions.

3.1 Fine-tuning on a single task

Single-task fine-tuning can improve performance on a well-defined task when the training data are representative and evaluation matches the intended use. The required sample size is empirical: it depends on the model, task diversity, label quality, adaptation method, and acceptable error. Full fine-tuning can also produce catastrophic forgetting—performance improves on the adapted task while degrading on capabilities learned previously.

3.1.1 Key Points:

1. Single-task Fine-tuning:

   • Focuses on one task (e.g., summarization or sentiment analysis) using task-specific examples.

   • Effective for applications where only one task is required.

2. Catastrophic Forgetting:

   • Occurs because full fine-tuning modifies the original model’s weights.

   • Catastrophic forgetting occurs when a machine learning model forgets previously learned information as it learns new information.

   • Catastrophic forgetting is a common problem in machine learning, especially in deep learning models.

   • One way to mitigate catastrophic forgetting is by using regularization techniques to limit the amount of change that can be made to the weights of the model during training.

   • Example: A model fine-tuned for sentiment analysis may lose its ability to do named entity recognition.

3. Options to Mitigate Catastrophic Forgetting:

   • Evaluate use case needs: If only one task is needed, catastrophic forgetting may not be an issue.

   • Multitask Fine-tuning: Train on a balanced mixture of relevant tasks and evaluate both target and retained capabilities. Data and compute needs vary substantially by model and objective.

   • Parameter Efficient Fine-tuning (PEFT): Modify only a small number of task-specific layers while preserving most of the original LLM weights. PEFT helps maintain robustness to catastrophic forgetting.

These approaches enable tailoring LLMs while balancing task-specific optimization with multitask generalization.

3.2 Multi-task, instruction fine-tuning

• Multitask fine-tuning trains a model on examples for multiple tasks (e.g., summarization, review rating, entity recognition).

• Avoids catastrophic forgetting by improving performance across all tasks simultaneously.

• Requirements:

  • Needs sufficiently diverse, high-quality examples across the intended tasks; there is no universal minimum sample count.

  • Produces a general-purpose model capable of handling diverse tasks well.

• Example:

  • FLAN Models (Fine-Tuned Language Net):

    • Family of models like FLAN-T5 and FLAN-PaLM are fine-tuned using 473 datasets across 146 tasks.

    • SAMSum dataset (dialogue summaries) is one of the datasets used for fine-tuning FLAN-T5.

3.2.1 Custom Fine-Tuning for Specific Use Cases

1. Scenario:
   • Example: Customer service teams using summaries of support chat conversations.
     
   • FLAN-T5 may struggle with domain-specific tasks (e.g., customer support chats) not covered in its training.
2. Solution:
   • Use a representative, permissioned domain dataset to adapt and evaluate the model. A public dialogue-summarization dataset may help with prototyping, but it is not a substitute for deployment-domain data.
3. Benefits of Domain-Specific Fine-Tuning:
   • Improves task performance on unique datasets (e.g., internal customer support chats).
     
   • May improve domain performance, but hallucination, safety, privacy, and distribution-shift risks still require separate evaluation.

3.2.2 Evaluation and Next Steps

• Evaluate the quality of fine-tuned model outputs using metrics and benchmarks.
  
• Internal, domain-specific datasets (e.g., customer service transcripts) can provide even better results tailored to the business needs.

3.3 Evaluating Model Performance in Language Tasks

When assessing language model performance, statements like “the model showed improvement” need formalization through evaluation metrics. While traditional machine learning metrics (e.g., accuracy) are straightforward, evaluating language-based models involves additional challenges, particularly when outputs are non-deterministic.

No single metric establishes overall quality. A defensible evaluation combines representative held-out cases, task-appropriate automatic metrics, structured human review where judgment is required, error analysis, and checks for safety, latency, cost, and subgroup behavior that matter in the deployment context.

Key Metrics:

1. ROUGE (Recall-Oriented Understudy for Gisting Evaluation):

   • Used for summarization tasks.

   • Measures overlap between model-generated and reference text using:

     • ROUGE-1: Focus on unigram (single word) matches.

     • ROUGE-2: Incorporates bigram (two-word sequence) matches to account for word order.

     • ROUGE-L: Uses the longest common subsequence for recall, precision, and F1-score calculations.

   • Challenges include susceptibility to artificially high scores from repeated words or incorrect word orders. Solutions include unigram clipping and experimenting with n-gram sizes based on sentence structure and use cases.

2. BLEU (Bilingual Evaluation Understudy):

   • Designed for machine translation tasks.

   • Calculates precision over multiple n-gram sizes and averages them.

   • Example: A BLEU score of 0.495 reflects a moderate match, improving as generated text resembles the reference more closely.

Best Practices:

• ROUGE is ideal for diagnosing summarization performance.

• BLEU is suited for translation evaluation.

• Both are simple and computationally inexpensive but should not be the sole evaluation tools for large language models. Use standardized benchmarks for comprehensive assessment.

Next steps often include benchmarking models against established datasets for a holistic evaluation. Many libraries (e.g., Hugging Face) provide pre-built implementations of these metrics for ease of use.

3.4 Benchmarks

Large language models (LLMs) are complex, and traditional metrics like ROUGE and BLEU provide limited insights into their capabilities. To evaluate LLMs more holistically, researchers rely on specialized datasets and benchmarks, which assess various skills, risks, and limitations. Selecting appropriate datasets for evaluation is crucial for understanding an LLM’s performance, particularly on unseen data.

3.4.1 Key Benchmarks for LLM Evaluation:

1. GLUE (General Language Understanding Evaluation):

   • Introduced in 2018.

   • Focuses on generalization across tasks like sentiment analysis and question-answering.

   • Encourages the development of versatile models.

2. SuperGLUE:

   • Introduced in 2019 to address GLUE’s limitations.

   • Adds tasks like multi-sentence reasoning and reading comprehension.

   • Features more challenging tests.

Both GLUE and SuperGLUE offer leaderboards for comparing systems on fixed task suites. Strong benchmark performance should not be interpreted as general human equivalence: datasets can saturate, become contaminated, or fail to represent a deployment setting.

3. MMLU (Massive Multitask Language Understanding):

   • Tests modern LLMs on diverse topics such as mathematics, history, law, and computer science.

   • Emphasizes advanced world knowledge and problem-solving.

4. BIG-bench:

   • Includes 204 tasks across disciplines like linguistics, common sense reasoning, software development, and biology.
     
   • Offers scalability with multiple size options to manage costs.

5. HELM (Holistic Evaluation of Language Models):

   • Focuses on model transparency and suitability for specific tasks.

   • Measures seven metrics (including fairness, bias, and toxicity) across 16 scenarios.

   • Goes beyond accuracy with multidimensional evaluation to reveal trade-offs.

HELM continuously evolves with new models, metrics, and scenarios, making it a dynamic resource for tracking progress in LLM capabilities.

3.4.2 Summary:

Modern LLM benchmarks like GLUE, SuperGLUE, MMLU, BIG-bench, and HELM provide essential tools for evaluating language models. They offer valuable insights into models’ strengths, weaknesses, and performance across a variety of tasks and contexts, aiding both research and practical applications.