Enterprise Architectures

A prototype agent running on a developer's laptop has different requirements from one processing 100,000 customer queries per day across four time zones. Enterprise scale introduces concerns that do not exist at prototype scale: queue depth, tail latency under load, cost per query at volume, audit trail completeness, model version control across environments, and the organisational complexity of multiple teams sharing the same AI infrastructure.

The most important architectural shift is moving from synchronous to asynchronous processing. In a synchronous architecture, every caller waits for their agent to complete. When traffic spikes, latency increases linearly. When the LLM API is slow, every user waits. An async queue architecture decouples request submission from processing entirely: callers submit a task and receive a job ID, workers process from the queue independently, and results are retrieved when ready.

The async queue pattern uses three components. An API gateway receives incoming requests, applies rate limiting and authentication, and places tasks onto a message queue. A pool of stateless worker processes reads from the queue, one task per worker at a time. Workers call the LLM API, execute tools, and write results to a storage layer. The queue is typically backed by Redis Streams, AWS SQS (Simple Queue Service), or RabbitMQ.

Workers are stateless by design: they hold no in-memory state between tasks. This makes horizontal scaling trivial. When queue depth grows, you start more workers. When it shrinks, you stop them. In Kubernetes (k8s), the container orchestration platform used by most cloud deployments, a Horizontal Pod Autoscaler (HPA) can trigger automatically based on queue depth metrics, scaling from 2 workers to 20 within seconds.

Queue-based architectures also provide natural backpressure. If the LLM API is throttled, workers slow down and the queue grows, but callers are not directly affected. If a worker crashes mid-task, the message remains unacknowledged and is re-delivered to another worker. This is fault tolerance that synchronous architectures cannot provide without complex retry logic scattered across the calling code.

Agent systems are distributed systems. A single user request may trigger multiple LLM calls, several tool executions, and database queries across different services. When something goes wrong, you need to reconstruct the exact sequence of events, their timing, and their inputs and outputs. Distributed tracing provides this.

OpenTelemetry is the open standard for distributed tracing, supported by every major observability platform (Datadog, Honeycomb, Jaeger, Grafana Tempo). You instrument your agent code by creating spans around each logical unit of work: the overall agent run, each LLM call, and each tool execution. Spans are linked by a trace ID that follows the request through every service. When you look up a failing task ID, you see every step, its duration, and its output in a single view.

Alongside tracing, collect structured metrics using Prometheus or a compatible metrics system. The four metrics that matter most for agents are: total requests (with success/failure labels), response latency as a histogram (to compute p50, p95, and p99 percentiles), total tokens consumed per model, and current queue depth. Alert on error rate above 1% and p99 latency above your SLA (Service Level Agreement) threshold.

At 200,000 queries per day with an average of 1,500 tokens per query routed to a frontier model at $15 per million input tokens, the monthly LLM cost is approximately $135,000. This is before output tokens, which can add another 30-50%. Cost management is not optional at enterprise scale.

Multi-model routing addresses this by sending each task to the cheapest model that can handle it reliably. Simple classification tasks (is this email spam or not spam?) do not require a frontier model. A compact, fast model at $0.25 per million tokens handles them with equivalent accuracy. Complex multi-step reasoning, code generation, or tasks requiring high accuracy use a more capable model. Routing logic based on task complexity, expected token count, and required accuracy can reduce monthly LLM costs by 40-70%.

Alongside routing, implement token budget enforcement. Define a daily token ceiling per agent or per customer, with alerts at 80% consumption and hard stops at 100%. Without enforcement, a single runaway agent or prompt injection attack can exhaust a day's budget in minutes. Token budgets are an operational control, not an optional optimisation.

Guardrails are programmatic constraints applied to agent inputs and outputs. They enforce safety, quality, and compliance requirements independently of the LLM. This independence is critical: a system prompt instruction that says "never include payment card numbers in your response" can be overridden by a sufficiently crafted prompt injection. An application-layer guardrail that runs a regex check on every response cannot be overridden regardless of what is in the context window.

Design guardrails in three categories. Blocking guardrails halt execution and return an error when triggered, for example detecting a potential credit card number in an output, or a response length that exceeds the contract. Warning guardrails log the issue and flag for human review but allow the response through, for example a response that mentions a competitor. Logging guardrails record the event for audit without affecting the response. Match the guardrail type to the severity of the constraint.

Banking, healthcare, insurance, and legal sectors operate under regulatory frameworks that impose specific technical requirements on AI systems. Data residency rules require that data processed or stored for certain customers must remain within a specific jurisdiction. Self-hosted models or regional cloud endpoints satisfy this; routing all data through a single US-based API does not.

Audit trail completeness means every AI-assisted decision must be reconstructable: who requested it, what input was provided, which model version was used, what output was generated, and when. This requires immutable, timestamped logs stored for the regulatory retention period. In financial services this is often seven years. In healthcare, longer.

Model version pinning is critical in regulated contexts. LLM providers update models continuously. An update to the underlying model can change outputs for identical prompts. In regulated industries, treat model upgrades as software releases: pin to a specific model version, conduct regression testing against a labelled test set before switching, and document the change in your change management system. EU AI Act Article 13 requires that high-risk AI systems provide users with information about the AI system's capabilities and limitations, including the model version.

For AI decisions that affect individuals (loan approval, insurance risk scoring, hiring recommendations), implement a human review queue. Decisions above a risk threshold or confidence threshold below a minimum should route to a human reviewer before the decision is finalised. This satisfies both regulatory expectations and the practical need for error correction in high-stakes contexts.