The gap nobody puts in the tutorial

Most agent tutorials end at the same place: a single LLM call wrapped in a loop, a couple of tools attached, and an “it works!” screenshot from a notebook. That’s a fine first step, but it’s also where most tutorials stop — right before the real engineering begins.

An agent that answers correctly once, in a notebook, with you watching, is a prototype. One that answers correctly the 10,000th time, unattended, at 3 a.m., with a throttled API — that’s a production system. The distance isn’t a bigger model or a cleverer prompt; it’s a set of architectural layers.

In my previous post, I walked through instrumenting an agent with Arize Phoenix — tracing, spans, the mechanics of seeing what an agent does. This post zooms out: the mental model I use to design an agent system end to end, before writing a line of orchestration code.

I’ll walk through four of them:

1. Triggering & execution — how an agent gets invoked, and what “reliable” means for something non-deterministic and expensive to retry
2. Routing — why not every request needs your most expensive model
3. The trust layer — how an agent earns the right to run unattended
4. Observability — the nervous system tying the other three together

Every concrete example here is AWS, purely for consistency — the four-layer model itself isn’t AWS-specific, and GCP and Azure have direct equivalents for nearly every piece (API Gateway ↔ Cloud Endpoints / Azure API Management, Lambda ↔ Cloud Functions / Azure Functions, Bedrock AgentCore ↔ Vertex AI Agent Builder / Azure AI Foundry Agent Service, and so on).

None of this is new — mostly known distributed-systems applied to a non-deterministic component.

Layer 1: Triggering & Execution — agents don’t live in chat boxes

Production agent architectures need at least four things in place:

• Edge/security tier — who’s allowed in, and how fast can they knock
• Application tier — the thin router, and the stateful runtime behind it
• Data/integration tier — what the agent is actually allowed to read and call
• An explicit resilience path — what happens when a step fails

The rest of this layer walks through each one.

If you’ve built agents through a chat interface, it’s easy to assume that’s how they get invoked in production. In practice, most are triggered by events, not by a person typing:

• A document lands in storage and needs summarizing and indexing
• A support ticket is created and needs triage and routing
• A webhook fires from a third-party system needing a response
• A scheduled job kicks off nightly batch analysis

This reframes the problem: you’re no longer designing a chatbot, you’re designing an event-driven system that happens to call an LLM as one step. The question isn’t “how do I write a good prompt” — it’s “how do I build something someone else can keep running when I’m not around.”

Two different jobs hide under the word “compute” and mixing them up is where a lot of “just put the agent on Lambda” designs get into trouble.

• The front door / request handler. This is the thin layer that authenticates the incoming request, validates it, extracts the payload, and routes it onward. It’s stateless and short-lived — a perfect match for Lambda: bursty traffic, pay-per-invocation, scales to zero when nothing’s happening.
• The agent runtime. This is where the actual agent loop lives — reasoning, tool calls, conversation state across turns. A single-shot task (summarize this document) is fine as a stateless invocation. But for anything conversational, a bare Lambda’s cold-start-per-request model works against you — reloading session state and re-establishing context on every turn, paying startup latency each time.

Purpose-built agent runtimes solve this. AWS’s reference pattern keeps the request-handling Lambda separate from a dedicated runtime (Bedrock AgentCore Runtime) that stays warm across turns — reducing Lambda’s job to four steps: extract payload, invoke agent, parse response, return it. Off Bedrock, that’s a long-running ECS/Fargate service behind the same thin layer. Either way: don’t make your stateless handler also your stateful agent host.

Auth has two jobs, too. OAuth (Cognito) authenticates the client to API Gateway; IAM then secures the internal hops — Gateway to Lambda, Lambda to runtime. The client’s token isn’t the token infrastructure uses internally. WAF sits in front of it all, catching abusive traffic before it reaches code that costs money.

Revised mental model: API Gateway (with WAF in front of it) decides if a request gets in. A thin Lambda decides where it goes. The agent runtime is where it actually happens.

Idempotency and dead-letter handling round out reliability. “Just retry on failure” is fine for a normal API — not for an agent, where a retried run might re-send an email or burn tokens on a call that already succeeded but timed out before writing its result. Every invocation needs an idempotency key, checked before handing off to the runtime again, with intermediate state written durably as you go so a retry resumes instead of restarting. For runs that still fail — a malformed input, a downed tool, a refused request — a dead-letter queue routes them somewhere instead of losing them silently, giving you a human-reviewable backlog that feeds into the trust layer later on.

The data/integration tier is what the agent is actually allowed to touch. Knowledge bases, vector stores, external APIs, the fallback search from Layer 3 — all of it sits behind the runtime, not in front of it. Scoping this tightly, with least-privilege, per-tool credentials, matters as much as the auth layer at the front door.

Put together, the shape of this layer looks roughly like:

This layer isn’t where the “AI” part lives; it’s the plumbing that makes it safe to run unwatched, precise about which piece is disposable (the router) and which piece needs to remember who it’s talking to (the runtime).

Layer 2: Cost-Aware Routing — not every request needs your best model

The default habit early on is sending every request to the same model — usually the most capable one, because that’s the one that made your demo look good. Fine until the bill arrives, or you notice you’re paying frontier-model prices to answer “what are your business hours.”

The fix is simple: classify the request, route to the cheapest model that can handle it. A one-line FAQ lookup doesn’t need the same model as a multi-step reasoning task. The interesting question isn’t whether to do this — it’s where the decision lives and how.

Two ways to build the router. Roll your own — a lightweight classifier or heuristic (prompt length, keywords, task type) picking a target model before the real call. Or lean on a managed version: Amazon Bedrock’s intelligent prompt routing predicts how well each model in a family handles a request and routes to the cheapest one clearing the quality bar — AWS reports cost savings ranging from 16% to 56% depending on model family, with RAG-specific workloads seeing up to 63.6% — all while holding response quality steady.

Where Bedrock fits. People bundle two things under “Bedrock”: a managed inference layer (one API, multiple providers) and a place to host agent runtimes (AgentCore, from Layer 1). For model access, it’s a three-way tradeoff:

None of these is universally “right” — a fast-shipping startup and a regulated enterprise land in different rows for the same feature. A cost-aware router means you don’t have to pick just one: route the easy 80% to a cheap managed endpoint, and reserve a self-hosted or fine-tuned model for the slice of requests where it earns its keep.

Keep the router swappable. Put it behind a single interface instead of hardcoding a provider call at every site that needs a model — a ModelRouter base class with route(request) -> ModelChoice, and implementations like RuleBasedRouter or ManagedPromptRouter. Ordinary OOP, but it’s the difference between “Bedrock’s router didn’t fit” being a one-file change versus a rewrite.

Where the decision lives. A cheap, rule-based decision lives in the thin request-handling layer, before the runtime is invoked. A decision that itself requires a model call belongs inside the runtime, covered by the same session and tracing as everything downstream. Either way — if the router costs nearly as much as the expensive model, it isn’t earning its place.

Layer 3: The Trust Layer — how an agent earns the right to run unattended

The first two layers make an agent reliable to invoke and affordable to run — neither makes it trustworthy. Trust comes down to one question: does the agent know when it doesn’t know, and does it have somewhere to go when it doesn’t?

Structured output is the foundation everything else sits on. Free-text output buries confidence and sourcing in prose something downstream has to regex out. A schema — a Pydantic model — turns those signals into first-class fields the system can act on:

class Citation(BaseModel):
    source_id: str
    snippet: str
    relevance_score: float
class AgentResponse(BaseModel):
    answer: str
    citations: list[Citation]
    overall_confidence: float
    needs_human_review: bool

Once those are real fields instead of implied vibes, everything downstream — routing to a human, falling back to search, logging for observability — gets simpler.

RAG with citations: show your work, not just the answer. Every claim traces back to a retrieved source — the user can verify it, and the system gets a check on its own grounding. Attach a relevance score to each citation; below some threshold, the agent hedges explicitly rather than presenting a shaky answer with full confidence. Hallucination prevention, here, is an architectural decision to make uncertainty visible, not a prompt trick.

Fall back to search when the knowledge base comes up empty. A RAG index is fixed; the real world isn’t. When retrieval returns nothing relevant, don’t let the model answer from memory — that’s where hallucination risk spikes. Fall back to a broader search instead, and say the answer now comes from outside the primary knowledge base.

Human-in-the-loop: let uncertainty trigger escalation, not a hardcoded category. A weaker pattern hardcodes rules like “always ask a human for refunds over $500.” A better one lets the agent’s own confidence set the same needs_human_review flag regardless of why — low retrieval confidence, a failed self-check, or a business rule funnel into one path. Pausing well matters as much as pausing at all: state needs to be durable — the same idempotency groundwork from Layer 1 — so a reviewer isn’t starting from scratch.

Self-reflective agents: let the agent grade its own work first. Execute → evaluate via a second LLM call acting as judge → return, retry, or escalate. Same idea used in offline eval frameworks like RAGAS or DeepEval, moved inline into the request path. The judge doesn’t need the generator’s capability — just enough to check whether the answer cites a source, is responsive, and doesn’t contradict the retrieved context.

Put together:

This layer turns “an agent that answers” into “an agent you can trust to run without reading every output” — and it generates most of the signals the next layer watches at scale.

Layer 4: Observability — the nervous system that ties it together

Every other layer generates signals worth watching. This layer watches them.

I covered the hands-on mechanics — tracing, spans, seeing what an agent is doing — in my previous post on instrumenting agents with Arize Phoenix. This is the shorter, conceptual version of why each piece matters.

Latency and cost dashboards. For a typical API, cost-per-request is an afterthought. For an agent, it’s a first-class metric — one request can fan out into several model calls, tool invocations, and retries, and that fan-out is where costs spiral. Track both per-request and aggregate: a latency spike and a cost spike often share a root cause.

Alerting on loops and failures. Agents fail in ways ordinary software mostly doesn’t — a tool-call loop that never terminates, a retry storm hitting a downstream service, a self-eval judge (Layer 3) that keeps rejecting its own output. These need their own alerts, distinct from generic error-rate monitoring, since a looping agent can look “healthy” on an uptime check while quietly burning budget.

Canary testing and rollback. Prompt and model changes deserve the same rigor as code deploys: route a small slice of traffic to the new version, compare quality and cost against baseline, and have a rollback path ready before shipping to everyone. This matters more for agents, since “quality regression” often shows up statistically, across many requests.

Evaluation as the quality half of observability. Dashboards and alerts cover the operational side — is it up, fast, affordable. Frameworks like RAGAS and DeepEval cover the other half — is it right: is retrieval relevant, is the answer faithful to its sources, does a reranker actually improve what gets retrieved. Layer 3’s inline self-eval is the same idea per-request; these are the batch version, run offline to catch regressions before production.

Put simply: observability is what turns the first three layers from “I built this once” into “I know this is still working.”

Putting the four layers together

None of these exist in isolation — observability in particular isn’t a step tacked onto the end, it’s a band that wraps around the other three, watching what they do:

This is the mental model I use to design these systems — not a checklist where every project needs all four layers at full strength, but a map of the weak points. A weekend ticket-triaging project might only need a thin Layer 1 and a basic Layer 3 hedge; a customer-facing refunds agent needs all four. Knowing where the weak points are — and which a project can afford to leave thin — is what separates “I built an agent” from “I can design a system that won’t give the on-call team headaches.”

If you want to see the observability side in practice, Layer 4 links to the write-up on instrumenting an agent with Arize Phoenix. Next up: something more hands-on on the routing layer, building a ModelRouter against Bedrock’s managed routing.

If you’ve built something similar, or think one of these weak points sits somewhere else, I’d like to hear about it.

References

• We Need to Talk About AI Agent Architectures
• AWS Serverless Multi-Tier Architectures with Amazon API Gateway and AWS Lambda – AWS whitepaper