Research frontiers

This module is about judgement. We will look at a few ideas that genuinely change what agents can do, then we will focus on adoption. What would you test, what would you measure, and what would make you say no.

The research landscape is shifting fast. Here are the developments I think matter most for practitioners, not just academics.

If you try to keep up with everything, you will fail and feel guilty. I use a small routine instead.

1. Skim widely once a week. Ten to twenty minutes is enough.

2. Pick one idea a month and test it in a sandbox.

3. Keep a short watchlist with a rule for adoption.

Good sources to skim

• Papers on arXiv in cs.AI and cs.CL

• Research blogs and technical reports from model providers and lab teams

• Practitioner communities such as Hugging Face and LangChain

• Conference programmes such as NeurIPS, ICML, and ACL

If you want formal credentials beyond this course, evaluate them with the same discipline you would use for a platform decision.

This is optional. I include it because RLHF underpins how modern models learn preferences, and the ideas are useful when you design agent feedback loops. You can still ship robust agents without training your own reward model.

Supervised learning teaches models what to say. Reinforcement learning teaches them how to act. For agents that need to achieve goals over multiple steps, RL gives you a framework for learning strategies under feedback.

RLHF is the technique behind the alignment of modern LLMs. Here is how it works:

When building your own agents, you may not have access to RLHF infrastructure. However, you can apply reward shaping principles to improve agent behaviour.

"""
Simple Reward Shaping for AI Agents
====================================
Demonstrates how to evaluate and reward agent actions.
"""

from typing import Dict, List
from dataclasses import dataclass

@dataclass
class AgentAction:
    action_type: str  # "tool_call", "response", "clarification"
    content: str
    tool_used: str | None = None
    success: bool = True

class RewardCalculator:
    """Calculate rewards for agent actions to guide behaviour."""

    def __init__(self):
        # Reward weights (tune these for your use case)
        self.weights = {
            "task_completion": 10.0,
            "efficiency": 2.0,
            "tool_accuracy": 3.0,
            "safety": 5.0,
            "user_satisfaction": 4.0,
        }

    def calculate_reward(
        self,
        actions: List[AgentAction],
        task_completed: bool,
        user_rating: int | None = None,  # 1-5 scale
        safety_violations: int = 0
    ) -> Dict[str, float]:
        """
        Calculate reward components for an agent interaction.

        Returns:
            Dictionary of reward components and total
        """
        rewards = {}

        # Task completion reward
        rewards["task_completion"] = (
            self.weights["task_completion"] if task_completed else 0
        )

        # Efficiency reward (fewer actions = better)
        # Baseline of 5 actions, penalty for more
        action_count = len(actions)
        rewards["efficiency"] = self.weights["efficiency"] * max(0, 5 - action_count) / 5

        # Tool accuracy (successful tool calls / total tool calls)
        tool_calls = [a for a in actions if a.action_type == "tool_call"]
        if tool_calls:
            success_rate = sum(1 for a in tool_calls if a.success) / len(tool_calls)
            rewards["tool_accuracy"] = self.weights["tool_accuracy"] * success_rate
        else:
            rewards["tool_accuracy"] = self.weights["tool_accuracy"]  # No tools needed

        # Safety penalty
        rewards["safety"] = self.weights["safety"] * max(0, 1 - safety_violations * 0.5)

        # User satisfaction (if available)
        if user_rating is not None:
            rewards["user_satisfaction"] = (
                self.weights["user_satisfaction"] * (user_rating - 1) / 4  # Normalise 1-5 to 0-1
            )
        else:
            rewards["user_satisfaction"] = 0

        rewards["total"] = sum(rewards.values())

        return rewards


# Example usage
if __name__ == "__main__":
    calculator = RewardCalculator()

    # Good interaction: task completed efficiently
    good_actions = [
        AgentAction("tool_call", "search_database", "database", True),
        AgentAction("response", "Here is the information you requested...")
    ]
    good_reward = calculator.calculate_reward(
        good_actions, task_completed=True, user_rating=5
    )
    print(f"Good interaction reward: {good_reward['total']:.2f}")

    # Poor interaction: multiple failed attempts
    poor_actions = [
        AgentAction("tool_call", "wrong_query", "database", False),
        AgentAction("tool_call", "another_wrong", "database", False),
        AgentAction("tool_call", "finally_right", "database", True),
        AgentAction("clarification", "Can you be more specific?"),
        AgentAction("response", "Sorry, I couldn't find that exactly...")
    ]
    poor_reward = calculator.calculate_reward(
        poor_actions, task_completed=False, user_rating=2
    )
    print(f"Poor interaction reward: {poor_reward['total']:.2f}")

In this stage, you have learned:

1. When fine tuning is worth it, and how to evaluate it honestly without confusing a tuning run with real improvement.

2. How to design enterprise architectures that isolate tenants, tools, and risk, with realistic expectations about adoption and failure.

3. How to ship an agent with monitoring, rollback, and incident readiness.

4. How to stay current without chasing hype or burning out, and where to get formal credentials.