Evaluation, metrics and failure analysis

Accuracy is an easy number to like because it feels clean. The problem is that it hides what you actually care about. In a spam filter, you can get high accuracy by declaring "not spam" for almost everything, because most email is not spam. The model looks great on paper and useless in practice.

What you want depends on the job. If you are blocking fraud, a false negative means you miss a bad transaction. If you are flagging innocent customers, a false positive means you cause real harm. Evaluation is choosing what kind of mistake is acceptable and proving the system is making the right trade.

Two scores matter. One is how the model behaves on the data it learned from, and the other is how it behaves on data it has never seen. Training performance is often optimistic because the model can memorise. Real world performance is harder because the inputs change and the environment changes. Good evaluation separates these on purpose.

For classification problems, two practical metrics are precision and recall. Precision answers the question "when the model says positive, how often is it right". Recall answers "of the real positives, how many did it catch". A spam filter with high recall catches most spam, but it might also block legitimate email. A fraud detector with high precision avoids annoying customers, but it might miss attacks.

For regression problems, you are predicting a number, like delivery time or house price. Here the question becomes "how far off are we". Metrics like mean absolute error are popular because they map to a simple story of average miss distance. Even without the formula, the idea is to measure error in the same units your users experience.

Evaluation also needs a sanity check for overfitting and underfitting. Overfitting is when training looks great and real performance drops. Underfitting is when both are poor because the model cannot learn the pattern. The fix is rarely "more metrics". It is usually better data, better representation, or a simpler model that is easier to trust.

Finally, production systems fail quietly. You can ship a model that passes every offline test and still break in the real world because the input distribution shifts. A spam campaign changes writing style. A fraud ring adapts. A new product changes customer behaviour. The model is not wrong in a dramatic way. It is just slowly less useful.

This is why evaluation is not a one time exam. It is a lifecycle. You validate before release, you test on untouched data, and you keep watching after deployment. If the system starts drifting, you want to notice early and know what to do next.

Technical people sometimes treat metrics like they are morally neutral. They are not. A false positive can be an annoyed customer, a delayed service, or a person wrongly flagged. A false negative can be money lost, harm missed, or abuse allowed through. When I say “match the metric to the decision”, I mean match it to the harm profile and the workload you are creating for humans.

I also want you to respect baselines. If a human process already catches 95 percent of bad cases, your model must beat that in a way that reduces harm, not just moves it around. If the model makes the review queue unbearable, it will be disabled. I have seen this happen more times than I would like.

Write this as a short note for your CPD log. Keep it honest and specific.