Loading lesson...
Loading lesson...
This is the third of 8 Practice & Strategy modules. Module 5 taught you to evaluate classification models with precision, recall, and cross-validation. Those methods assume a clear ground truth. But when a model generates text, code, or images, what does "correct" even mean? This module covers the evaluation methods designed for generative AI: human judgement protocols, automated judges, adversarial testing, and the benchmark infrastructure that the industry uses to compare models.
Evaluation innovation · 2024
In early 2024, the LMSYS Chatbot Arena became the most-cited leaderboard in AI. The concept was simple: show a user the same prompt answered by two anonymous models, let them pick the better response, and compute Elo ratings from the votes. No benchmark suite, no automated scorer, just raw human preference at scale.
The results were revealing. Models that dominated traditional benchmarks like MMLU sometimes ranked poorly on Arena. Users cared about factors that benchmarks did not measure: helpfulness, tone, ability to follow nuanced instructions, and willingness to say "I don't know." Conversely, some models that scored modestly on MMLU ranked highly because they were more useful in practice.
But human evaluation does not scale. Chatbot Arena required hundreds of thousands of votes. For internal model comparisons at companies running dozens of experiments per week, waiting for human votes is impractical. This created demand for LLM-as-judge: using a strong model (like GPT-4) to evaluate the outputs of weaker models. The question at the heart of this module is whether and when you can trust each approach.
If two language models both produce fluent, plausible answers to the same question, how do you decide which one is better?
Chatbot Arena exposed a gap between what benchmarks measure and what users value. This module walks through the evaluation methods available, from human protocols through automated methods to adversarial testing, so you can choose the right evaluation strategy for your specific use case.
If you have already designed evaluation harnesses for generative models, use the knowledge checks to confirm your understanding and skip to Module 20: AI agents and tool use.
With the learning outcomes established, this module begins by examining human evaluation: the gold standard and its costs in depth.
For generative tasks, human evaluation remains the gold standard. A human reader can assess whether a summary captures the key points, whether generated code is idiomatic, whether a translation preserves nuance, and whether a response is actually helpful. No automated metric reliably captures all of these dimensions.
A rigorous human evaluation protocol requires: a clear rubric defining what "good" means (helpfulness, accuracy, fluency, safety), multiple independent annotators per item to measure agreement, randomised presentation order to avoid position bias, and a sufficient sample size for statistical power. The standard measure of annotator agreement is Cohen's kappafor two annotators or Krippendorff's alpha for multiple annotators.
The limitations are practical: human evaluation is slow (days to weeks), expensive (paid annotators or diverted engineering time), non-reproducible (different annotators, different results), and does not scale to the hundreds of experiments a fast-moving team runs per month. This is why human evaluation is typically reserved for high-stakes decisions: product launches, model releases, and safety evaluations.
“Human evaluation, despite its limitations, remains the most reliable way to assess the quality of generated text. No automated metric has been shown to correlate perfectly with human judgement across all dimensions of quality.”
van der Lee, C. et al., 'Best Practices for the Human Evaluation of Automatically Generated Text', Journal of Artificial Intelligence Research (2021) - Section 2: Current Practices
This thorough survey of human evaluation methods in NLP identified systematic problems in how the field conducts human evaluation and proposed standardised protocols. It remains the definitive reference for designing evaluation studies.
With an understanding of human evaluation: the gold standard and its costs in place, the discussion can now turn to llm-as-judge: automated evaluation with language models, which builds directly on these foundations.
LLM-as-judge uses a strong language model (typically GPT-4 or Claude) to evaluate the outputs of other models. The judge receives the prompt, the model's response, and a rubric, then produces a score or a pairwise preference. The appeal is obvious: it is fast, cheap, reproducible, and scales to thousands of evaluations per hour.
Research has shown that strong LLM judges agree with human annotators at rates comparable to human-human agreement (roughly 80% on pairwise preferences). This makes LLM-as-judge viable for rapid iteration during development, where the goal is to detect regressions and compare candidates, not to produce a final quality assessment.
The known biases are well documented. Position bias: judges tend to prefer the first response in a pairwise comparison. Verbosity bias: longer responses are rated higher even when they add no substance. Self-preference bias: a model used as a judge may prefer outputs from its own family. Mitigation strategies include randomising presentation order, using multiple judges, and calibrating against a held-out set of human annotations.
Common misconception
“LLM-as-judge can fully replace human evaluation.”
LLM judges are useful for rapid iteration and regression detection, but they have systematic biases (position, verbosity, self-preference) and cannot evaluate dimensions that require real-world knowledge or cultural context. The recommended practice is to use LLM judges for development-time evaluation and reserve human evaluation for high-stakes decisions: model releases, safety assessments, and product launches. Treat LLM-as-judge as a fast proxy, not a ground truth.
With an understanding of llm-as-judge: automated evaluation with language models in place, the discussion can now turn to red teaming: adversarial evaluation for safety, which builds directly on these foundations.
Red teaming is the practice of systematically probing a model for failure modes, harmful outputs, and safety violations. The term comes from military and cybersecurity practice, where a "red team" plays the adversary to test an organisation's defences. In AI evaluation, red teamers attempt to make the model produce dangerous, biased, or misleading content.
Effective red teaming requires diversity of approach. Automated red teaming uses other language models to generate adversarial prompts at scale. Human red teaming brings domain expertise: a medical professional can identify subtly dangerous health advice that an automated system would miss. The most thorough evaluations combine both.
Anthropic, OpenAI, Google DeepMind, and Meta all conduct red teaming before model releases. The findings inform both model fine-tuning (RLHF adjustments to reduce harmful outputs) and system-level mitigations (content filters, refusal classifiers). Red teaming is not a one-time event; it is a continuous process because adversarial techniques evolve and new failure modes are discovered as models are deployed in new contexts.
With an understanding of red teaming: adversarial evaluation for safety in place, the discussion can now turn to benchmark suites: mmlu, humaneval, and beyond, which builds directly on these foundations.
MMLU (Massive Multitask Language Understanding) tests a model across 57 academic subjects from elementary mathematics to professional law. It became the de facto benchmark for LLM general knowledge. Its limitation is that it measures multiple-choice question answering, not the open-ended generation that most users care about.
HumanEval tests code generation by presenting function signatures and docstrings and asking the model to implement the function. Solutions are verified by running them against test cases. Pass@k measures the probability that at least one of k generated samples passes all tests.
MT-Bench tests multi-turn conversation quality using GPT-4 as a judge across categories like writing, reasoning, and mathematics. It was designed to complement Chatbot Arena with a reproducible, automated alternative.
The fundamental problem with all static benchmarks is saturation: as models improve, they approach or exceed human performance on the benchmark, and the benchmark stops discriminating between models. MMLU scores above 90% are now common among frontier models, making it less useful for comparing them. The field responds by creating harder benchmarks (MMLU-Pro, GPQA), but the cycle repeats.
“Benchmarks are like thermometers: they tell you the temperature, but they do not tell you why the patient is sick. A model that scores 90% on MMLU may still fail catastrophically on your specific use case.”
Liang, P. et al., 'Holistic Evaluation of Language Models (HELM)', Stanford CRFM (2022) - Section 1: Motivation
HELM was the first systematic attempt to evaluate LLMs across multiple dimensions (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency) rather than reducing evaluation to a single score. It demonstrated that models ranked differently depending on which dimension was measured.
With an understanding of benchmark suites: mmlu, humaneval, and beyond in place, the discussion can now turn to elo ratings: ranking models through competition, which builds directly on these foundations.
The Elo rating system, originally designed for chess, assigns each model a numerical rating based on pairwise comparisons. When Model A beats Model B (a human prefers A's response), A's rating increases and B's decreases. The magnitude of the change depends on the expected outcome: an upset (low-rated model beats high-rated model) causes a larger rating swing.
Chatbot Arena uses Elo ratings computed from over a million human votes. The system is attractive because it naturally handles the transitivity problem: if A beats B and B beats C, A should (on average) beat C, and the ratings reflect this without requiring direct A-vs-C comparisons for every pair.
Limitations include: Elo assumes a single dimension of quality, but model quality is multidimensional (a model can be better at code and worse at creative writing). Elo is sensitive to the population of prompts: a model optimised for coding will rank higher if the prompt distribution skews toward code. And Elo ratings are relative, not absolute: a rating of 1200 means nothing without knowing the ratings of other models in the pool.
Common misconception
“The model at the top of the leaderboard is the best model for my use case.”
Leaderboard rankings reflect aggregate performance across the leaderboard's prompt distribution, which may not resemble your use case. A model ranked fifth overall might be first for medical question answering or first for code generation in your target language. Always evaluate on your own data, with your own rubric, before making deployment decisions. Leaderboards are useful for shortlisting candidates, not for making final choices.
With an understanding of elo ratings: ranking models through competition in place, the discussion can now turn to evaluation harnesses: infrastructure for systematic testing, which builds directly on these foundations.
An evaluation harness is the software infrastructure that automates the process of running models against benchmark suites, collecting results, and reporting scores. The two most widely used are EleutherAI's lm-evaluation-harness (open source, supports hundreds of benchmarks) and Stanford's HELM (Holistic Evaluation of Language Models, which evaluates across multiple quality dimensions).
A well-designed harness handles: model loading and inference, prompt formatting for each benchmark, answer extraction and scoring, aggregation across examples, and result storage for comparison. It should be deterministic: running the same model on the same benchmark twice should produce the same score (controlling for randomness in sampling by fixing seeds).
For production teams, the evaluation harness is not optional. It is the mechanism that gates deployment: a model must pass a defined set of benchmarks before it can be promoted from staging to production. This is the ML equivalent of a CI/CD pipeline's test suite: if the tests fail, the deployment is blocked.
A team uses GPT-4 as a judge to compare outputs from two fine-tuned models. They always present Model A's response first and Model B's second. After 500 evaluations, GPT-4 prefers Model A 62% of the time. Should they trust this result?
A model achieves 92% on MMLU, placing it in the top 5 on the leaderboard. The team plans to deploy it for medical question answering in a clinical setting. What additional evaluation is necessary?
An AI startup claims their model 'outperforms GPT-4' based on an internal benchmark with 200 questions. What should a careful evaluator ask?
Zheng, L. et al., 'Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena', NeurIPS (2023)
Full paper
Introduced both MT-Bench and the Chatbot Arena evaluation methodology. Demonstrated that strong LLM judges agree with humans at rates comparable to human-human agreement, establishing LLM-as-judge as a viable evaluation approach.
van der Lee, C. et al., 'Best Practices for the Human Evaluation of Automatically Generated Text', JAIR (2021)
Sections 2-5
thorough survey of human evaluation practices in NLP. Identified systematic problems in how the field conducts human evaluation and proposed standardised protocols for reproducible, reliable assessment.
Liang, P. et al., 'Holistic Evaluation of Language Models (HELM)', Stanford CRFM (2022)
Sections 1-3
First systematic multi-dimensional evaluation of LLMs covering accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. Demonstrated that models rank differently depending on which dimension is measured.
Hendrycks, D. et al., 'Measuring Massive Multitask Language Understanding', ICLR (2021)
Full paper
Introduced MMLU, the most widely used LLM benchmark. Covers 57 academic subjects and established the practice of testing models across diverse knowledge domains.
Perez, E. et al., 'Red Teaming Language Models with Language Models', EMNLP (2022)
Sections 2-4
Demonstrated automated red teaming using language models to generate adversarial test cases at scale. Established the methodology for systematic adversarial evaluation that is now standard at major AI labs.
You now understand how to evaluate AI systems rigorously: human protocols for ground truth, LLM judges for rapid iteration, red teaming for safety, and benchmarks for shortlisting. The next frontier is models that do not just answer questions but take actions. Module 20 covers AI agents, tool use, the ReAct pattern, and multi-agent orchestration.
Module 19 of 24 · AI Practice & Strategy
