Module E1 of 3 · Exam and certification

Course revision guide

45 min 3 stages distilled 3 figures

The whole course in one place, arranged for revision. Each stage below gives you its figure, the distinctions worth holding in your head, and the fastest routes back into the full material. Skim the figures first, then test yourself against the distinctions: anything you cannot explain from memory points at the stage to revisit.

Stage 1: What learning from data actually is, and what counts as evidence that it worked

Where the model output ends and the system decision begins

Both lanes end at an output arrow entering the same system boundary, and only that boundary, holding policy, guardrails, review and audit, emits the action, so swapping a written rule for a learned model moves none of the accountability.

A learned model's output is one input to the decision. The surrounding system, with its policy, guardrails, and human review, is what makes the decision and carries the accountability.

Hold these distinctions

Artificial intelligence, machine learning and deep learning are nested, not interchangeable. A rule-based system with no learning in it is still AI; a system that learns from data is machine learning; a multi-layer neural network is deep learning. Naming the wrong one is usually how a claim gets inflated.
Narrow AI performs one task and cannot transfer the ability. Artificial general intelligence would perform any intellectual task a person can, and does not exist. Every system discussed in this stage is narrow.
In traditional software the programmer writes the rules and the machine applies them. In machine learning the programmer supplies data and an algorithm and the machine derives the rules, which is why quality follows the data and why behaviour cannot be read off the source.
Bias is a systematic mismatch between the training data and the population served. Noise is random variation with no pattern behind it. More data reduces noise; more data of the same kind does nothing about bias.
The validation set guides decisions about the model and can be used repeatedly. The test set estimates performance on unseen data and can be used once. Confusing them turns an unbiased estimate into a flattering one.
A loss function is what the model is optimised against during training. A metric is what you judge it by afterwards. They are frequently different, and choosing a loss that ignores the error type your metric cares about is a common and quiet mistake.
Precision is the share of flagged items that were real. Recall is the share of real items that were flagged. Moving the threshold trades one for the other, so a system cannot be described by either alone.
Overfitting means the model learned the training set's noise and does not generalise: training performance high, validation performance low. Underfitting means the model is too simple to capture the pattern: both low. The gap between the two curves tells you which you have.
Demographic parity equalises the rate of positive predictions across groups. Equalized odds equalises the error rates across groups. When base rates differ, satisfying one violates the other, which is a proven result rather than an implementation problem.
LIME explains one prediction by fitting an interpretable model to local perturbations and is model-agnostic but unstable. SHAP attributes contributions using Shapley values, giving theoretical guarantees and global as well as local explanations at a higher computational cost.
The model is the component that produces a prediction. The system is what acts on it, and the system is what carries the accountability. Evaluating the model and evaluating the deployment are different exercises with different questions.

Full stage summary Scenario practice

Stage 2: How the current generation of models is built, prompted, deployed and governed

A transformer block runs attention and a residual path in parallel

Three projections carry the same embedding into scores, softmax and weighted values, while a dashed residual runs around the block into the Add and forward card, so the next block still reads the original signal when attention contributes little.

A transformer block runs two paths at once: an attention path that mixes tokens through Q, K, V projections, and a residual path that preserves the original signal. The next block reads the sum of both, cited to Vaswani et al. 2017 §3.2.

Hold these distinctions

Encoder-only versus decoder-only versus encoder-decoder: the difference is what a token may attend to. Bidirectional attention suits understanding tasks, causal masking is what autoregressive generation requires, and keeping both halves suits work where the whole source must be read before the target is produced
Parallel attention versus sequential recurrence: recurrence needs one step per token however much hardware is available, while self-attention computes every position in one matrix multiplication. The price is that attention cost grows with the square of the sequence length
Token versus word: models operate on sub-word units produced by byte-pair encoding, so a rare term or a sentence in an under-represented language consumes more tokens than its length suggests and eats context that appeared to be free
Context window versus memory: the window is how much can be supplied in one pass, not how much is attended to. Enlarging it does not make attention uniform across positions
Pre-training versus supervised fine-tuning versus reinforcement learning from human feedback: the first produces language structure and knowledge, the second teaches the shape of a helpful answer, the third optimises against a reward model learned from pairwise human preferences while a divergence penalty keeps the model near its fine-tuned checkpoint
Zero-shot versus few-shot prompting: zero-shot supplies instruction and constraints only, few-shot supplies worked examples so the pattern can be inferred with no weight update. Example selection and class balance matter more than example count
Retrieval precision versus answer faithfulness versus citation accuracy: the first is a retrieval-stage measure, the second and third are generation-stage measures, and citation accuracy can fail while the answer is correct and the retrieved chunks were right
Classification versus detection versus segmentation: one label per image, a set of boxes with classes and confidences, and a label for every pixel. Instance segmentation adds object identity and panoptic segmentation combines both, and each rung has its own architecture, loss and metric
Intersection over union versus mean average precision: the first measures overlap between one predicted region and the ground truth, the second averages precision across classes at stated overlap thresholds. A detection score without its threshold cannot be compared with anything
Batch versus real-time serving: batch computes predictions on a schedule for work that can be precomputed, real-time computes them at request time when a decision gates something else. Hybrid serving combines batch-computed stable features with real-time volatile signals
Data drift versus concept drift: the first is a change in the input distribution and is visible without labels, the second is a change in what the inputs predict and is invisible to input monitoring because the inputs look unchanged
Feature store versus model registry: the feature store holds one definition of each feature for both training and serving and prevents skew, the registry holds versioned model artefacts with their lineage and makes rollback and promotion gates possible
Direct versus indirect prompt injection: direct injection is typed into the interface by the attacker, indirect injection is planted in content the system will later retrieve, so the user never sees it and the model cannot distinguish the document's content from its own instructions
Model extraction versus membership inference: extraction rebuilds a functional copy from interface responses and attacks intellectual property, inference determines whether a specific record was in the training set and attacks the privacy of the data
Provider versus deployer: the provider builds and places the system on the market and supplies documentation and declared intended uses, the deployer puts it to work in a specific context and answers for whether it is appropriate there. Both carry obligations
High risk versus unacceptable risk: high-risk systems are permitted subject to risk management, data governance, documentation, human oversight, accuracy requirements and conformity assessment; unacceptable-risk practices are prohibited outright. The tier follows the deployment context, not the architecture

Full stage summary Scenario practice

Stage 3: Designing, costing and evaluating AI systems, and the safety case around them

Three concentric rings absorb the model in a production AI system

Each rectangle wholly contains the one inside it, and the request crosses all three before it reaches the model, so the audit trail, escalation and ownership drawn on the outer ring cover every inner control, the model included.

A production AI system has at least three concentric boundaries around the model. The model alone is never the system; each outer ring absorbs the responsibilities of the rings it contains.

Hold these distinctions

Data drift versus concept drift: data drift means the input distribution has moved away from the training distribution; concept drift means the relationship between inputs and outputs has changed while the inputs may look entirely normal. The second is harder to detect and needs outcome data, not input statistics
Training-serving skew versus drift: skew is a defect present from the first day, where the feature computed during training differs from the one computed at serving time; drift is a change over time in a system that was consistent at launch. A feature store removes the first and a monitor detects the second
Post-training quantisation versus quantisation-aware training: the first lowers precision on an already-trained model and costs nothing but accuracy; the second simulates the reduced precision during training so the weights adapt to it, preserving more accuracy at the cost of retraining
Quantisation versus distillation versus pruning: quantisation reduces bytes per parameter, distillation reduces the parameter count by training a smaller student on a larger teacher's output probabilities, and pruning removes weights or structures that contribute little. They compose, which is why edge deployment usually needs more than one
Structured versus unstructured pruning: unstructured pruning zeroes individual weights and can reach high sparsity that ordinary hardware cannot exploit; structured pruning removes whole neurons, heads or layers and produces a model that is genuinely faster on standard hardware, usually after a fine-tuning pass to recover accuracy
Human evaluation versus automated judging versus static benchmarks: the first is the reference standard and does not scale, the second scales and carries position, verbosity and self-enhancement biases, the third is reproducible and saturates. A release decision should rest on all three rather than on the most flattering one
Benchmark saturation versus benchmark irrelevance: a saturated benchmark no longer separates strong models from each other, while an irrelevant one never measured your task in the first place. A high score on a multiple-choice suite is not evidence about open-ended generation in your domain
Workflow versus agent: in a workflow, models and tools run through predefined code paths written by a developer; in an agent, the model decides the control flow dynamically from what it has observed. The loop is what makes it an agent, and the loop is also what needs a budget gate to stop it
Q-learning versus policy gradient methods: Q-learning estimates the value of state-action pairs and derives the policy from it, which requires a discrete and enumerable action space; policy gradient methods parameterise the policy directly and therefore handle continuous actions, at the cost of higher variance in their estimates
Outer alignment versus inner alignment: outer alignment asks whether the specified objective is the right one, inner alignment asks whether the model internally pursues that objective rather than a correlated proxy that diverges outside the training distribution. A system can fail either one while passing the other
Visible rationale versus hidden reasoning tokens: both add inference-time work and both can improve results, but neither is a guarantee that the stated reasoning caused the answer. Ask for evidence and assumptions that can be checked, not for a narrative
Impact versus feasibility in use case selection: quick wins are feasible with moderate impact and exist to generate evidence and confidence, strategic bets are high impact and need planning, and foundations enable later work while delivering no direct value. Confusing a foundation with a quick win is how a data platform ends up carrying a business case it cannot support

Full stage summary Scenario practice

Exam technique for the two papers

The Foundation paper is 40 single-answer questions in 60 minutes, closed book, and it tests precise recall. Revise the pairs that sound alike and are not: supervised against unsupervised, against recall, validation against test, a rule written by an engineer against a pattern inferred from data, retrieval against fine-tuning. In the sitting, read the stem twice and decide what is actually being asked before looking at the options, because many questions turn on one qualifying word.

The options are written so that measuring them does not help. The correct answer is as often the short one as the long one, and no option is there to be eliminated for being obviously silly, so eliminate on meaning rather than on shape. A precise claim beats a sweeping one, and an option that would be true of a different method is the commonest way to be wrong.

The Practitioner paper is 8 workplace scenarios in 90 minutes, open book, scored five, three, one and zero. Rank the options rather than hunting a single right answer. Across this course the five-point answer almost always establishes what the system is actually being asked to do, and what evidence would settle it, before reaching for a model: it names the cost of each kind of mistake, or the measurement that would show whether the thing works at all. The three-point answer is normally real work aimed slightly wrong, most often sequenced too early or answering a question nobody asked. Use the reference panel to confirm a distinction, not to learn the topic inside 90 minutes.

The primary sources behind the course

Version-sensitive claims across the course anchor to primary, dated sources, and every stage summary cites the ones its teaching rests on. They are collected here so revision starts from the authority rather than from a summary of it.

NIST AI Risk Management Framework
Northcutt, Athalye and Mueller, 'Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks'
Chouldechova, 'Fair prediction with disparate impact: A study of bias in recidivism prediction instruments'
Angwin, Larson, Mattu and Kirchner, 'Machine Bias', ProPublica
Mitchell, M. et al. Model Cards for Model Reporting, FAccT (2019)
Grinsztajn, Oyallon and Varoquaux, 'Why do tree-based models still outperform deep learning on tabular data?'
Tom Mitchell, Machine Learning (1997), Chapter 1
Rumelhart, Hinton and Williams, 'Learning representations by back-propagating errors', Nature (1986)
Hochreiter and Schmidhuber, 'Long Short-Term Memory', Neural Computation (1997)
Lazer and colleagues, 'The Parable of Google Flu: Traps in Big Data Analysis', Science (2014)
Vaswani, A. et al. Attention Is All You Need (2017)
Hoffmann, J. et al. Training Compute-Optimal Large Language Models (2022)
Lewis, P. et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)
Greshake, K. et al. Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023)
Goodfellow, I., Shlens, J. and Szegedy, C. Explaining and Harnessing Adversarial Examples (2014)
OWASP Top 10 for Large Language Model Applications, project page
Sculley, D. et al. Hidden Technical Debt in Machine Learning Systems, NeurIPS (2015)
Regulation (EU) 2024/1689, the Artificial Intelligence Act
Mitchell, M. et al. Model Cards for Model Reporting, FAT* (2019)
Sculley, D. et al. Hidden Technical Debt in Machine Learning Systems, NeurIPS (2015)
Frantar, E. et al. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2023)
Hinton, G., Vinyals, O. and Dean, J. Distilling the Knowledge in a Neural Network (2015)
Zheng, L. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS (2023)
Yao, S. et al. ReAct: Synergizing Reasoning and Acting in Language Models (2022)
Anthropic, Building Effective Agents
Model Context Protocol
Ouyang, L. et al. Training language models to follow instructions with human feedback, NeurIPS (2022)
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction, 2nd edition (2018)
Schaeffer, R. et al. Are Emergent Abilities of Large Language Models a Mirage?, NeurIPS (2023)
Bai, Y. et al. Constitutional AI: Harmlessness from AI Feedback (2022)
AI Security Institute (United Kingdom)
Financial Conduct Authority discussion paper on artificial intelligence and machine learning in financial services (2022)
EU Artificial Intelligence Act

When the distinctions hold from memory and the figures feel obvious, you are ready to drill the papers and then sit the mocks.

Start the exam practice Go straight to the mocks

Loading lesson...

Stage 1: What learning from data actually is, and what counts as evidence that it worked

Where the model output ends and the system decision begins

A learned model's output is one input to the decision. The surrounding system, with its policy, guardrails, and human review, is what makes the decision and carries the accountability.

Hold these distinctions

Artificial intelligence, machine learning and deep learning are nested, not interchangeable. A rule-based system with no learning in it is still AI; a system that learns from data is machine learning; a multi-layer neural network is deep learning. Naming the wrong one is usually how a claim gets inflated.
Narrow AI performs one task and cannot transfer the ability. Artificial general intelligence would perform any intellectual task a person can, and does not exist. Every system discussed in this stage is narrow.
In traditional software the programmer writes the rules and the machine applies them. In machine learning the programmer supplies data and an algorithm and the machine derives the rules, which is why quality follows the data and why behaviour cannot be read off the source.
Bias is a systematic mismatch between the training data and the population served. Noise is random variation with no pattern behind it. More data reduces noise; more data of the same kind does nothing about bias.
The validation set guides decisions about the model and can be used repeatedly. The test set estimates performance on unseen data and can be used once. Confusing them turns an unbiased estimate into a flattering one.
A loss function is what the model is optimised against during training. A metric is what you judge it by afterwards. They are frequently different, and choosing a loss that ignores the error type your metric cares about is a common and quiet mistake.
Precision is the share of flagged items that were real. Recall is the share of real items that were flagged. Moving the threshold trades one for the other, so a system cannot be described by either alone.
Overfitting means the model learned the training set's noise and does not generalise: training performance high, validation performance low. Underfitting means the model is too simple to capture the pattern: both low. The gap between the two curves tells you which you have.
Demographic parity equalises the rate of positive predictions across groups. Equalized odds equalises the error rates across groups. When base rates differ, satisfying one violates the other, which is a proven result rather than an implementation problem.
LIME explains one prediction by fitting an interpretable model to local perturbations and is model-agnostic but unstable. SHAP attributes contributions using Shapley values, giving theoretical guarantees and global as well as local explanations at a higher computational cost.
The model is the component that produces a prediction. The system is what acts on it, and the system is what carries the accountability. Evaluating the model and evaluating the deployment are different exercises with different questions.

Stage 2: How the current generation of models is built, prompted, deployed and governed

A transformer block runs attention and a residual path in parallel

Hold these distinctions

Encoder-only versus decoder-only versus encoder-decoder: the difference is what a token may attend to. Bidirectional attention suits understanding tasks, causal masking is what autoregressive generation requires, and keeping both halves suits work where the whole source must be read before the target is produced
Parallel attention versus sequential recurrence: recurrence needs one step per token however much hardware is available, while self-attention computes every position in one matrix multiplication. The price is that attention cost grows with the square of the sequence length
Token versus word: models operate on sub-word units produced by byte-pair encoding, so a rare term or a sentence in an under-represented language consumes more tokens than its length suggests and eats context that appeared to be free
Context window versus memory: the window is how much can be supplied in one pass, not how much is attended to. Enlarging it does not make attention uniform across positions
Pre-training versus supervised fine-tuning versus reinforcement learning from human feedback: the first produces language structure and knowledge, the second teaches the shape of a helpful answer, the third optimises against a reward model learned from pairwise human preferences while a divergence penalty keeps the model near its fine-tuned checkpoint
Zero-shot versus few-shot prompting: zero-shot supplies instruction and constraints only, few-shot supplies worked examples so the pattern can be inferred with no weight update. Example selection and class balance matter more than example count
Retrieval precision versus answer faithfulness versus citation accuracy: the first is a retrieval-stage measure, the second and third are generation-stage measures, and citation accuracy can fail while the answer is correct and the retrieved chunks were right
Classification versus detection versus segmentation: one label per image, a set of boxes with classes and confidences, and a label for every pixel. Instance segmentation adds object identity and panoptic segmentation combines both, and each rung has its own architecture, loss and metric
Intersection over union versus mean average precision: the first measures overlap between one predicted region and the ground truth, the second averages precision across classes at stated overlap thresholds. A detection score without its threshold cannot be compared with anything
Batch versus real-time serving: batch computes predictions on a schedule for work that can be precomputed, real-time computes them at request time when a decision gates something else. Hybrid serving combines batch-computed stable features with real-time volatile signals
Data drift versus concept drift: the first is a change in the input distribution and is visible without labels, the second is a change in what the inputs predict and is invisible to input monitoring because the inputs look unchanged
Feature store versus model registry: the feature store holds one definition of each feature for both training and serving and prevents skew, the registry holds versioned model artefacts with their lineage and makes rollback and promotion gates possible
Direct versus indirect prompt injection: direct injection is typed into the interface by the attacker, indirect injection is planted in content the system will later retrieve, so the user never sees it and the model cannot distinguish the document's content from its own instructions
Model extraction versus membership inference: extraction rebuilds a functional copy from interface responses and attacks intellectual property, inference determines whether a specific record was in the training set and attacks the privacy of the data
Provider versus deployer: the provider builds and places the system on the market and supplies documentation and declared intended uses, the deployer puts it to work in a specific context and answers for whether it is appropriate there. Both carry obligations
High risk versus unacceptable risk: high-risk systems are permitted subject to risk management, data governance, documentation, human oversight, accuracy requirements and conformity assessment; unacceptable-risk practices are prohibited outright. The tier follows the deployment context, not the architecture

Stage 3: Designing, costing and evaluating AI systems, and the safety case around them

Three concentric rings absorb the model in a production AI system

A production AI system has at least three concentric boundaries around the model. The model alone is never the system; each outer ring absorbs the responsibilities of the rings it contains.

Hold these distinctions

Data drift versus concept drift: data drift means the input distribution has moved away from the training distribution; concept drift means the relationship between inputs and outputs has changed while the inputs may look entirely normal. The second is harder to detect and needs outcome data, not input statistics
Training-serving skew versus drift: skew is a defect present from the first day, where the feature computed during training differs from the one computed at serving time; drift is a change over time in a system that was consistent at launch. A feature store removes the first and a monitor detects the second
Post-training quantisation versus quantisation-aware training: the first lowers precision on an already-trained model and costs nothing but accuracy; the second simulates the reduced precision during training so the weights adapt to it, preserving more accuracy at the cost of retraining
Quantisation versus distillation versus pruning: quantisation reduces bytes per parameter, distillation reduces the parameter count by training a smaller student on a larger teacher's output probabilities, and pruning removes weights or structures that contribute little. They compose, which is why edge deployment usually needs more than one
Structured versus unstructured pruning: unstructured pruning zeroes individual weights and can reach high sparsity that ordinary hardware cannot exploit; structured pruning removes whole neurons, heads or layers and produces a model that is genuinely faster on standard hardware, usually after a fine-tuning pass to recover accuracy
Human evaluation versus automated judging versus static benchmarks: the first is the reference standard and does not scale, the second scales and carries position, verbosity and self-enhancement biases, the third is reproducible and saturates. A release decision should rest on all three rather than on the most flattering one
Benchmark saturation versus benchmark irrelevance: a saturated benchmark no longer separates strong models from each other, while an irrelevant one never measured your task in the first place. A high score on a multiple-choice suite is not evidence about open-ended generation in your domain
Workflow versus agent: in a workflow, models and tools run through predefined code paths written by a developer; in an agent, the model decides the control flow dynamically from what it has observed. The loop is what makes it an agent, and the loop is also what needs a budget gate to stop it
Q-learning versus policy gradient methods: Q-learning estimates the value of state-action pairs and derives the policy from it, which requires a discrete and enumerable action space; policy gradient methods parameterise the policy directly and therefore handle continuous actions, at the cost of higher variance in their estimates
Outer alignment versus inner alignment: outer alignment asks whether the specified objective is the right one, inner alignment asks whether the model internally pursues that objective rather than a correlated proxy that diverges outside the training distribution. A system can fail either one while passing the other
Visible rationale versus hidden reasoning tokens: both add inference-time work and both can improve results, but neither is a guarantee that the stated reasoning caused the answer. Ask for evidence and assumptions that can be checked, not for a narrative
Impact versus feasibility in use case selection: quick wins are feasible with moderate impact and exist to generate evidence and confidence, strategic bets are high impact and need planning, and foundations enable later work while delivering no direct value. Confusing a foundation with a quick win is how a data platform ends up carrying a business case it cannot support

Exam technique for the two papers