Evaluation, monitoring and governance in production AI

Evaluation in production is not a single score. It is a series of checks that answer one question: does the system still help the business without creating unacceptable harm. That requires measurement before launch, after launch, and while the world changes.

Offline evaluation is what you do on datasets you control. It is useful for comparing versions and catching obvious regressions. It also has blind spots. Offline data is usually cleaner than reality, and it rarely contains the full cost of mistakes. Online evaluation is what you do in the live system, where users, latency, and edge cases are real. A model that looks strong offline can still fail online if it changes user behaviour or breaks workflows.

Accuracy alone is not enough because the cost of mistakes is not symmetric. A fraud system that misses fraud can be catastrophic. A moderation system that over blocks can silence legitimate users. In these cases you reach for precision and recall, then connect them to real outcomes like chargebacks, review workload, or user churn.

Even strong metrics are not stable. You need to monitor performance over time because production data is a moving target. The metric you track depends on the system, but the habit is the same: measure, investigate, and learn. If you cannot explain why a metric moved, you cannot fix it.

Monitoring is the boring work that saves you. Start with inputs. Are you seeing new categories, missing fields, unusual ranges, or sudden format changes. Then monitor outputs. Are scores shifting, are confidence values drifting upward, are certain groups being flagged more often. Finally monitor system health: latency, error rate, and rate limiting. If the system is slow, it will change user behaviour and it will change your data.

Alerting is where good teams become noisy teams. Too many alerts create alert fatigue, and then real problems are ignored. You want alerts that are actionable, tied to clear owners, and paired with a playbook. False positives in monitoring are not harmless. They burn trust and time.

Drift is the quiet killer. Data drift is when inputs change. Concept drift is when the meaning of the target changes, even if inputs look similar. A credit scoring model can see stable features while repayment behaviour changes during a downturn. A moderation model can see similar text while norms and tactics change.

Retraining schedules matter because drift does not wait for your roadmap. Some systems need periodic retraining. Others need trigger based retraining when drift crosses a threshold. Either way, you should treat retraining like a release, with the same discipline: evaluation, rollout, rollback, and audit trails.

This is where governance stops being paperwork and becomes operations. Someone must own the model and the system around it. Decisions about thresholds, fallbacks, and acceptable harm are product and risk decisions, not just ML decisions.

Good governance includes documentation and traceability. You want to know which data, which features, which model version, and which thresholds produced an outcome. Human oversight is part of the design. It needs authority, not just review. And you need a shutdown plan. If the system is causing harm or you cannot understand its behaviour, you stop it, switch to a safe fallback, and investigate.

Use this as a clean CPD entry. Keep it short and specific. If you can attach an artefact, do it.