Stage 5: Advanced Mastery

Stage 5: Advanced Mastery

You have built agents. You understand security. Now let us take your skills to production level. This stage covers the techniques that separate hobbyist projects from enterprise systems: fine-tuning custom models, designing scalable architectures, and deploying with confidence.

Module 5.1: Fine-Tuning Open Source Models (8 hours)

Learning Objectives

By the end of this module, you will be able to:

1. Prepare datasets for fine-tuning
2. Apply LoRA and QLoRA techniques
3. Evaluate fine-tuned models effectively
4. Choose when fine-tuning is appropriate

5.1.1 When to Fine-Tune

Fine-tuning is not always the answer. Let me be direct about when it makes sense.

5.1.2 Understanding LoRA and QLoRA

5.1.3 Preparing Your Dataset

Data quality matters more than quantity. A small, high-quality dataset beats a large, noisy one.

"""
Dataset Preparation for Fine-Tuning
====================================
How to structure training data for instruction fine-tuning.
"""

import json
from typing import List, Dict


def create_training_example(
    instruction: str,
    input_text: str,
    output: str,
    system_prompt: str = ""
) -> Dict:
    """
    Create a single training example in the standard format.
    
    Args:
        instruction: What you want the model to do
        input_text: Context or input data (can be empty)
        output: The desired response
        system_prompt: Optional system context
        
    Returns:
        Dictionary in training format
    """
    example = {
        "instruction": instruction,
        "input": input_text,
        "output": output,
    }
    
    if system_prompt:
        example["system"] = system_prompt
    
    return example


# Example: Creating a dataset for customer support fine-tuning

support_examples = [
    create_training_example(
        instruction="Respond to this customer query professionally.",
        input_text="I have not received my order after 2 weeks.",
        output="""I apologise for the delay with your order. Let me look into this for you straightaway.

Could you please provide your order number? I will check the shipping status and arrange a resolution. If the order has been lost, I will ensure we send a replacement or process a full refund.

Thank you for your patience.""",
        system_prompt="You are a helpful customer support agent. Be empathetic and solution-focused."
    ),
    create_training_example(
        instruction="Respond to this customer query professionally.",
        input_text="Your product stopped working after one day.",
        output="""I am sorry to hear your product is not working as expected. That is certainly not the experience we want you to have.

For a product that failed within 24 hours, you are entitled to a full refund or immediate replacement. Which would you prefer?

If you would like a replacement, I can arrange express delivery at no extra cost. Please let me know which option works best for you.""",
        system_prompt="You are a helpful customer support agent. Be empathetic and solution-focused."
    ),
]


def save_dataset(examples: List[Dict], filepath: str):
    """Save dataset in JSONL format (one JSON object per line)."""
    with open(filepath, "w") as f:
        for example in examples:
            f.write(json.dumps(example) + "\n")


def validate_dataset(filepath: str) -> Dict:
    """
    Validate a training dataset.
    
    Returns statistics and any issues found.
    """
    stats = {
        "total_examples": 0,
        "avg_instruction_length": 0,
        "avg_output_length": 0,
        "issues": []
    }
    
    instruction_lengths = []
    output_lengths = []
    
    with open(filepath, "r") as f:
        for line_num, line in enumerate(f, 1):
            try:
                example = json.loads(line)
                stats["total_examples"] += 1
                
                # Check required fields
                if "instruction" not in example:
                    stats["issues"].append(f"Line {line_num}: Missing instruction")
                if "output" not in example:
                    stats["issues"].append(f"Line {line_num}: Missing output")
                
                # Track lengths
                instruction_lengths.append(len(example.get("instruction", "")))
                output_lengths.append(len(example.get("output", "")))
                
                # Check for very short outputs (likely low quality)
                if len(example.get("output", "")) < 50:
                    stats["issues"].append(f"Line {line_num}: Very short output")
                    
            except json.JSONDecodeError:
                stats["issues"].append(f"Line {line_num}: Invalid JSON")
    
    if instruction_lengths:
        stats["avg_instruction_length"] = sum(instruction_lengths) / len(instruction_lengths)
    if output_lengths:
        stats["avg_output_length"] = sum(output_lengths) / len(output_lengths)
    
    return stats

Module 5.2: Enterprise Architectures (7 hours)

Learning Objectives

By the end of this module, you will be able to:

1. Design multi-tenant agent systems
2. Implement scalable infrastructure
3. Handle compliance requirements
4. Plan for high availability

5.2.1 Multi-Tenant Architecture

When building agents for multiple customers, data isolation is critical.

Key Principles:

1. Data Isolation: Each tenant's data must be completely separate
2. Resource Limits: Prevent one tenant from consuming all resources
3. Audit Trails: Track all actions by tenant for compliance
4. Customisation: Allow per-tenant configuration without code changes

5.2.2 Scaling Strategies

Module 5.3: Production Deployment (5 hours)

Learning Objectives

By the end of this module, you will be able to:

1. Deploy agents with Docker and Kubernetes
2. Implement monitoring and observability
3. Set up CI/CD pipelines
4. Handle production incidents

5.3.1 Containerisation with Docker

# Dockerfile for AI Agent
FROM python:3.12-slim

# Set working directory
WORKDIR /app

# Install system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    curl \
    && rm -rf /var/lib/apt/lists/*

# Copy requirements first (for caching)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY src/ ./src/
COPY config/ ./config/

# Create non-root user for security
RUN useradd --create-home appuser
USER appuser

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:8080/health || exit 1

# Expose port
EXPOSE 8080

# Run the agent
CMD ["python", "-m", "src.main"]

5.3.2 Monitoring and Observability

Key Metrics to Track:

Module 5.4: Research Frontiers (5 hours)

Learning Objectives

By the end of this module, you will be able to:

1. Understand emerging agent architectures
2. Evaluate new reasoning techniques
3. Contribute to open-source projects
4. Stay current with AI agent research

5.4.1 Emerging Architectures

5.4.2 Staying Current

The field moves fast. Here is how I stay updated:

Key Resources:

1. Papers: ArXiv cs.AI and cs.CL sections
2. Blogs: Anthropic, OpenAI, Google DeepMind research blogs
3. Communities: Hugging Face Discord, LangChain Slack
4. Conferences: NeurIPS, ICML, ACL for foundational work

Stage 5 Assessment

🎯 Interactive: Multi-Agent Orchestrator

Explore different orchestration patterns for multi-agent systems. Build virtual teams, simulate inter-agent communication, and understand trade-offs between supervisor, peer-to-peer, hierarchical, and round-robin patterns.

🎯 Interactive: Agent Evaluation Benchmark

Learn how to evaluate AI agents across multiple dimensions. Score your agents against standard benchmark scenarios and understand what makes a production-ready agent system.

Module 5.5: Reinforcement Learning for Agents (5 hours)

Learning Objectives

By the end of this module, you will be able to:

1. Understand how reinforcement learning improves agent behaviour
2. Explain RLHF (Reinforcement Learning from Human Feedback)
3. Implement basic reward shaping for agents
4. Evaluate when RL is appropriate for your use case

5.5.1 Why Reinforcement Learning for Agents?

Supervised learning teaches models what to say. Reinforcement learning teaches them how to act. For AI agents that need to achieve goals over multiple steps, RL provides a framework for learning optimal strategies.

5.5.2 RLHF in Practice

RLHF is the technique behind the alignment of modern LLMs. Here is how it works:

5.5.3 Reward Shaping for Agents

When building your own agents, you may not have access to RLHF infrastructure. However, you can apply reward shaping principles to improve agent behaviour.

"""
Simple Reward Shaping for AI Agents
====================================
Demonstrates how to evaluate and reward agent actions.
"""

from typing import Dict, List
from dataclasses import dataclass

@dataclass
class AgentAction:
    action_type: str  # "tool_call", "response", "clarification"
    content: str
    tool_used: str | None = None
    success: bool = True

class RewardCalculator:
    """Calculate rewards for agent actions to guide behaviour."""

    def __init__(self):
        # Reward weights (tune these for your use case)
        self.weights = {
            "task_completion": 10.0,
            "efficiency": 2.0,
            "tool_accuracy": 3.0,
            "safety": 5.0,
            "user_satisfaction": 4.0,
        }

    def calculate_reward(
        self,
        actions: List[AgentAction],
        task_completed: bool,
        user_rating: int | None = None,  # 1-5 scale
        safety_violations: int = 0
    ) -> Dict[str, float]:
        """
        Calculate reward components for an agent interaction.

        Returns:
            Dictionary of reward components and total
        """
        rewards = {}

        # Task completion reward
        rewards["task_completion"] = (
            self.weights["task_completion"] if task_completed else 0
        )

        # Efficiency reward (fewer actions = better)
        # Baseline of 5 actions, penalty for more
        action_count = len(actions)
        rewards["efficiency"] = self.weights["efficiency"] * max(0, 5 - action_count) / 5

        # Tool accuracy (successful tool calls / total tool calls)
        tool_calls = [a for a in actions if a.action_type == "tool_call"]
        if tool_calls:
            success_rate = sum(1 for a in tool_calls if a.success) / len(tool_calls)
            rewards["tool_accuracy"] = self.weights["tool_accuracy"] * success_rate
        else:
            rewards["tool_accuracy"] = self.weights["tool_accuracy"]  # No tools needed

        # Safety penalty
        rewards["safety"] = self.weights["safety"] * max(0, 1 - safety_violations * 0.5)

        # User satisfaction (if available)
        if user_rating is not None:
            rewards["user_satisfaction"] = (
                self.weights["user_satisfaction"] * (user_rating - 1) / 4  # Normalise 1-5 to 0-1
            )
        else:
            rewards["user_satisfaction"] = 0

        rewards["total"] = sum(rewards.values())

        return rewards


# Example usage
if __name__ == "__main__":
    calculator = RewardCalculator()

    # Good interaction: task completed efficiently
    good_actions = [
        AgentAction("tool_call", "search_database", "database", True),
        AgentAction("response", "Here is the information you requested...")
    ]
    good_reward = calculator.calculate_reward(
        good_actions, task_completed=True, user_rating=5
    )
    print(f"Good interaction reward: {good_reward['total']:.2f}")

    # Poor interaction: multiple failed attempts
    poor_actions = [
        AgentAction("tool_call", "wrong_query", "database", False),
        AgentAction("tool_call", "another_wrong", "database", False),
        AgentAction("tool_call", "finally_right", "database", True),
        AgentAction("clarification", "Can you be more specific?"),
        AgentAction("response", "Sorry, I couldn't find that exactly...")
    ]
    poor_reward = calculator.calculate_reward(
        poor_actions, task_completed=False, user_rating=2
    )
    print(f"Poor interaction reward: {poor_reward['total']:.2f}")

5.5.4 When to Use RL for Agents

Summary

In this stage, you have learned:

1. Fine-tuning techniques: When to use them and how to prepare data

2. Enterprise architectures: Multi-tenancy, scaling, and compliance

3. Production deployment: Docker, Kubernetes, monitoring, and CI/CD

4. Research frontiers: Emerging techniques and staying current

5. Reinforcement learning: RLHF, reward shaping, and when to apply RL to agents