Applied Capstone

Content moderation at scale is not a single model making a single decision. It is a pipeline of specialised models, each handling a different content type and policy category, orchestrated by a decision engine that combines their outputs.

Stage 1: Content classification. Incoming content is routed to specialised classifiers based on modality. Text goes to NLP models that detect hate speech, harassment, misinformation, and self-harm content. Images go to computer vision models that detect violence, nudity, and banned organisations. Video is sampled at key frames and processed by both pipelines. Audio is transcribed and processed as text.

Stage 2: Context enrichment. Raw classifier scores are enriched with contextual signals: the poster's account age, history of violations, geographic region, the content's engagement velocity (rapidly shared content is higher priority), and whether the content is a reply to previously flagged content. Context distinguishes a news article about violence from a call to violence.

Stage 3: Decision and routing. A decision engine combines classifier scores and contextual signals. High-confidence violations (score above 0.95 for the most severe categories) are removed automatically. Medium-confidence flags (0.70-0.95) are routed to human reviewers with the model's assessment as context. Low-confidence flags (below 0.70) are logged for monitoring but not actioned.

Stage 4: Human review. Human reviewers handle appeals, ambiguous cases, and new policy categories where the model has not yet been trained. Their decisions feed back into the training pipeline as labelled data, creating a flywheel that continuously improves model accuracy.

Each stage in the pipeline requires different model architectures. Text classifiers typically use fine-tuned transformer models (BERT or RoBERTa variants) trained on labelled policy violation data. Image classifiers use convolutional networks (ResNet, EfficientNet) or vision transformers (ViT) fine-tuned on content policy violation datasets. Multimodal models (CLIP variants) handle content where text and image must be understood together (a hateful meme where neither the text nor the image alone is hateful).

Training data is the critical bottleneck. Content moderation datasets are inherently biased toward English, toward Western cultural norms, and toward policy categories that have historical precedent. Under-resourced languages receive worse protection. Emerging harms (new slang, coded language, AI-generated deepfakes) are absent from training data until they are identified by human reviewers.

The training strategy must account for class imbalance: the vast majority of content on any platform is benign. Without careful sampling, models learn that the safest prediction is "not a violation," which is correct 99% of the time and catastrophically wrong when it matters. Techniques from Module 5 (evaluation metrics on imbalanced data) and Module 12 (computer vision architectures) are directly applicable here.

Content moderation must operate in real time. A post promoting self-harm that reaches thousands of users before it is removed has already caused harm. This rules out batch serving for the initial classification stage. Models must return predictions within tens of milliseconds at the scale of millions of posts per hour.

Monitoring is continuous and multi-layered. Volume monitoringtracks the rate of flagged content by category and region; sudden spikes indicate either a real-world event or an adversarial attack. Precision monitoring samples automated removal decisions and has human reviewers audit them; if precision drops below a threshold, the automated action is suspended and content is routed to human review. Adversary adaptation monitoring tracks the rate of successful policy violations (content that was harmful but not detected), estimated from user reports and appeals.

Adversarial resilience is a design requirement, not an afterthought. Attackers use character substitution (replacing letters with visually similar Unicode characters), image steganography (embedding harmful text in images), fragmentation (splitting a harmful message across multiple replies), and code-switching (mixing languages to evade language-specific classifiers). Every evasion technique that succeeds generates training data for the next model iteration.

Content moderation systems face targeted attacks. Data poisoningcould corrupt the training pipeline if an attacker floods the platform with content designed to shift the model's decision boundary (reporting benign content as harmful to make the model over-remove, or posting harmful content that evades detection to make the model under-remove). Adversarial examples target the classifiers directly: imperceptible image perturbations that flip a "violent" classification to "benign."

Under the EU AI Act, an AI content moderation system deployed by a platform serving EU users faces transparency obligations (users must be informed that AI is involved in moderation decisions) and, depending on the deployment context, may qualify as a high-risk system requiring conformity assessment. The Digital Services Act (DSA) adds further requirements: very large online platforms must publish transparency reports on content moderation, provide appeal mechanisms, and undergo independent audits.

The model card for a content moderation system must document: which content categories it covers, which languages it supports, its performance disaggregated by language and content type, known failure modes (satire, news reporting, cultural context), and the human oversight mechanisms in place. This documentation is not optional; it is a regulatory requirement and a practical necessity for maintaining system quality over time.