The architecture of an in-app agent, for people who approve it

This post exists for the meeting where someone has to say yes. Not the implementation meeting. The one where a staff engineer, a security reviewer and someone accountable for money look at a diagram and decide whether the thing is safe to put in front of customers.

So it is a diagram and an explanation of every box, followed by the two rules that make the whole design defensible, and a short history of why this was not possible five years ago.

The three bands#

The top band is yours and mostly already exists. Your app, the SDK you embed in it, and your API. The one new thing is a set of tool definitions over that API: names, argument shapes, return shapes and error text written so a model can select and recover.

The middle band is the runtime. It holds the conversation, decides which tool to call, keeps connections to your services warm, stops before any write, builds the description of what should appear on screen, and turns speech into text and back. This is the band you either build or adopt, and the nine workstreams behind an in-house build is the decision in full.

The bottom band is vendors. A model provider, a speech vendor and whatever services the runtime is pointed at. Each is a dependency with its own availability, its own deprecation schedule and its own bad afternoons.

Identity: the part to settle first#

The agent carries the end user’s own credential on every request, rather than a service account or an elevated key of its own. That single decision means the agent’s blast radius is exactly the user’s blast radius, and your API’s existing authorisation checks keep working without modification.

The corollary is that the token has to be read fresh on each request rather than cached at session start, because tokens rotate and a stale one produces failures that look like network errors. Getting this wrong is the most common integration defect we see.

The two rules that make it approvable#

Rule one: a write leaves the middle band only through a user approval. The turn that selects a write operation stops short of running it and returns a description of what it would do. A second, separate request executes it after the user agrees. Approval is tiered by risk, so a low-value action is a tap and a money-moving one is a biometric prompt.

Rule two: anything drawn on screen leaves the middle band only by naming a shape from a closed list. The runtime cannot emit layout, markup or arbitrary HTML. It names a widget kind and supplies its data, and the client draws it with components that shipped in the binary.

Those two rules together close the attack that worries security reviewers most. The Model Context Protocol specification states the underlying risk directly: “Tools represent arbitrary code execution and must be treated with appropriate caution”, and tool behaviour descriptions “should be considered untrusted, unless obtained from a trusted server”. Text that reaches the model, including rows your own backend returned, can contain instructions. If a write requires a human tap and a screen can only be built from a fixed vocabulary, an injected instruction cannot reach either outcome on its own.

What changed, and when#

The pre-2023 architecture classified an utterance into one of a fixed list of intents, pulled slots out of it by rule, and called a handler written for that intent. Every new phrasing was a code change. Every new capability was a new screen. The failure mode was quiet and bad: an utterance outside the grammar matched the nearest intent and the system acted on it.

What replaced it is a loop. Yao and colleagues described the shape in ReAct (October 2022), interleaving reasoning with acting rather than treating them as separate problems, and reported that it “outperforms imitation and reinforcement learning methods by an absolute success rate of 34% and 10% respectively” on ALFWorld and WebShop, “while being prompted with only one or two in-context examples”. One or two examples is the part that changed the economics. A capability became a tool definition rather than a training set, and why the previous generation of assistants failed reads differently once you see that.

Anthropic’s Building effective agents (19 December 2024) draws the line that matters for an architecture review: “Workflows are systems where LLMs and tools are orchestrated through predefined code paths. Agents are systems where LLMs dynamically direct their own processes and tool usage.” If you are approving an agent, you are approving a system whose control flow is decided at runtime by a model. Everything in the middle band exists to bound what that can do.

The tool seam#

The runtime reaches your services through a tool interface, and the Model Context Protocol has become the usual way to express one. Its current revision, dated 2026-07-28, is worth knowing in outline because it differs from the one most explainers describe.

The base protocol is now “stateless, self-contained requests” with “per-request capability negotiation”. Servers offer resources, prompts and tools. Clients offer exactly one feature back, elicitation, which is a server-initiated request for more information from the user. And there is an extensions layer, opt-in and negotiated at initialisation, carrying things like asynchronous long-running tasks.

For an architecture review the practical consequences are that a tool call is self-describing rather than dependent on a long-lived session, and that a server can ask your user a question mid-turn, which is a surface worth knowing about before you enable it. Wiring the agent into the endpoints you already expose covers that half of the job, and what MCP is, revision by revision covers the protocol itself.

Latency, and where it comes from#

Human conversation has a narrow rhythm. Stivers and colleagues measured turn-taking across ten languages on five continents and found a mean response offset of +208 milliseconds, with every language’s distribution peaking between 0 and 200 milliseconds. That is the standard a spoken interface is unconsciously measured against, and no current agent stack comes close to it.

Our own warm turns land around 2.5 to 3 seconds end to end, and a cold one runs from 2.5 to 30 depending on how the upstream service is behaving. Almost none of that is compute we control. It is the round trip to the tool server plus a prompt cache that has not been primed. The mitigations are architectural rather than algorithmic: hold connections open, prime the cache at app launch, and keep the tool schema small, since it is re-sent on every turn.

An architecture review should ask what happens during those seconds, because the answer is a product decision. The SDK exposes a turn as five states, idle, listening, transcribing, thinking and speaking, and each is a chance to show the user where the time is going. One spinner covering all five is a different experience from a listening indicator, then a transcript, then a skeleton widget.

Failure modes worth designing for#

Four failures happen often enough to deserve a designed response rather than an exception handler.

The model provider is slow or unavailable. The turn has to end in something a user understands, and the useful answer is an apology plus the same information through a link into your existing UI, rather than a spinner that resolves into nothing.

A tool times out halfway through. The agent has already told the user it is looking something up, so silence reads as a hang. Decide what the assistant says at five seconds and what it says at twenty.

The agent picks the wrong operation. This is not rare, and the design question is how cheaply a user can correct it. One follow-up sentence is good; starting the conversation over is not.

A session token expires mid-conversation. The client should refresh and retry once without the user seeing anything, and fall back to a visible sign-in prompt only if that fails.

What to ask before approving#

Which operations can the agent call, and who wrote that list. What is the largest thing a single approved action can do. What happens when the model picks the wrong tool, and how would you know. Where does audio go, how long is it kept, and by whom. What is the fallback when the model provider is down. And how is the whole thing evaluated on a fixed question set before each release.

If those six have answers, the architecture is reviewable. The stack-level guide covers what sits in each layer, and holding a write until a person accepts it covers the gate in detail.

Sources#

• Model Context Protocol, Specification, revision 2026-07-28. Accessed 12 September 2026.
• Shunyu Yao and others, ReAct: Synergizing Reasoning and Acting in Language Models, 6 October 2022. Accessed 12 September 2026.
• Anthropic, Building effective agents, 19 December 2024. Accessed 12 September 2026.
• Tanya Stivers and others, Universals and cultural variation in turn-taking in conversation, PNAS, 2009. Accessed 12 September 2026.