Memory and context

Explore the different types of agent memory through this interactive tool. Test your understanding of when to use short-term, long-term, and external memory for different queries.

As conversations grow, they eventually exceed the context window. You need strategies to handle this.

"""
Conversation Memory Management
==============================
Strategies for maintaining conversation context.
"""

from dataclasses import dataclass, field
from typing import List, Optional
from datetime import datetime


@dataclass
class Message:
    """A single message in conversation history."""
    role: str  # "user", "assistant", or "system"
    content: str
    timestamp: datetime = field(default_factory=datetime.now)
    token_count: int = 0


class ConversationMemory:
    """
    Manages conversation history with context window awareness.
    """
    
    def __init__(self, max_tokens: int = 4000, preserve_system: bool = True):
        """
        Initialise conversation memory.
        
        Args:
            max_tokens: Maximum tokens to keep in history
            preserve_system: Always keep the system message
        """
        self.max_tokens = max_tokens
        self.preserve_system = preserve_system
        self.messages: List[Message] = []
        self.system_message: Optional[Message] = None
    
    def add_message(self, role: str, content: str) -> None:
        """Add a message to history."""
        # Estimate token count (roughly 4 chars per token)
        token_count = len(content) // 4
        
        message = Message(
            role=role,
            content=content,
            token_count=token_count
        )
        
        if role == "system":
            self.system_message = message
        else:
            self.messages.append(message)
        
        # Prune if needed
        self._prune_to_fit()
    
    def _prune_to_fit(self) -> None:
        """Remove old messages to fit within token limit."""
        current_tokens = sum(m.token_count for m in self.messages)
        
        if self.system_message:
            current_tokens += self.system_message.token_count
        
        # Remove oldest messages until we fit
        while current_tokens > self.max_tokens and len(self.messages) > 2:
            removed = self.messages.pop(0)
            current_tokens -= removed.token_count
    
    def get_messages(self) -> List[dict]:
        """Get messages in format suitable for LLM."""
        result = []
        
        if self.system_message:
            result.append({
                "role": "system",
                "content": self.system_message.content
            })
        
        for msg in self.messages:
            result.append({
                "role": msg.role,
                "content": msg.content
            })
        
        return result
    
    def summarise_and_compress(self, summariser_fn) -> None:
        """
        Summarise older messages to save tokens.
        
        Args:
            summariser_fn: Function that takes messages and returns summary
        """
        if len(self.messages) < 10:
            return
        
        # Take oldest 70% of messages
        split_point = int(len(self.messages) * 0.7)
        old_messages = self.messages[:split_point]
        recent_messages = self.messages[split_point:]
        
        # Summarise old messages
        old_content = "\n".join(
            f"{m.role}: {m.content}" 
            for m in old_messages
        )
        summary = summariser_fn(old_content)
        
        # Replace with summary
        summary_message = Message(
            role="system",
            content=f"[Summary of earlier conversation: {summary}]",
            token_count=len(summary) // 4
        )
        
        self.messages = [summary_message] + recent_messages

When you need to remember things across sessions or search through large amounts of information, vector databases are essential.

Context engineering is the most important shift in how practitioners think about working with AI. It replaces the narrower idea of "prompt engineering" with something much broader and more useful.

Andrej Karpathy put it well in June 2025 when he described context engineering as the art and science of filling the context window with just the right information for the next step. He proposed thinking of an LLM like a CPU and the context window as RAM. Your job as a builder is like an operating system. You load that working memory with the right code and data for the task at hand.

Four context strategies

Effective context engineering uses four complementary strategies.

Write. Save context outside the context window so it persists across turns. This includes scratchpads, memory files, persistent logs, and queryable storage. The key insight is that not everything needs to live in the active context. Write important information out, then bring it back when it is needed.

Select. Pull relevant context into the window when the agent needs it. This is what retrieval augmented generation does. Instead of hoping the model memorised the right facts, you search for relevant documents and inject them at the right moment.

Compress. Retain only the tokens required for the current task. Conversation histories grow quickly. Summarising older exchanges, discarding irrelevant tool outputs, and setting token budgets keeps the context window focused.

Isolate. Split context across multiple agents, each with a narrow sub-task focus. A research agent does not need access to the email sending tool. A code review agent does not need the deployment credentials. Isolation is both a security and a performance strategy.

Before reaching for fine-tuning or complex architectures, I follow a practical hierarchy that saves time and money.