How to add an AI agent to a mobile app: the complete guide

Most teams estimate this as a feature, the way a search bar is a feature. What they are adding is five separate systems that happen to arrive together, four of which have nothing to do with their product.

This guide covers what those five pieces are, which one is genuinely yours, what a first working turn takes, and what the second week costs.

What an in-app agent actually is#

An in-app agent is a loop that reads a user’s request, decides which of your operations to call, calls them, and then answers with both speech and screen. Anthropic draws the line this way in Building effective agents (19 December 2024): “Workflows are systems where LLMs and tools are orchestrated through predefined code paths. Agents are systems where LLMs dynamically direct their own processes and tool usage.”

That second sentence is the whole design problem. If the model chooses the path, you cannot enumerate the paths in advance, so you cannot write a screen for each one. Either the agent gets to draw its own answers, or it is limited to whatever sentences your existing UI can already produce.

A chatbot answers questions about your product. An agent operates it, which is a different piece of software with a different failure mode.

The five pieces, and which one is yours#

The runtime#

The runtime holds the conversation, picks tools, and handles the case where a tool returns something unexpected. It is model-shaped work: prompt structure, caching, retries, streaming, and a plan for the day a model you depend on is retired. Almost none of it is specific to your product, which is why teams who own it for a quarter tend to stop wanting to.

The tool layer#

This is yours, in every version of this project. A tool is one of your operations described so that a model can select it: a name, an argument shape, a return shape, and error text that tells the model what to do next rather than what went wrong internally. Anthropic’s own note on the subject is blunt about the ratio of effort involved. “We actually spent more time optimizing our tools than the overall prompt.”

Their later guidance on tool design (11 September 2025) adds the part teams get wrong first: “More tools don’t always lead to better outcomes.” A few well-named operations that map to real user goals beat a generated wrapper around every endpoint you have. If you are pointing an agent at a service you already run, the Model Context Protocol is the usual seam, and the job is smaller than it sounds.

The interface layer#

A spoken sentence is a poor container for twelve transactions, a balance breakdown, or a date picker. So the runtime returns a small structured description of what should appear, and the client draws it with its own native components. That pattern has a decade of mobile practice behind it. Thoughtworks placed server-driven UI at Trial in Volume 34 of the Technology Radar (April 2026), noting that separating rendering into a generic container lets “mobile teams bypass lengthy app store review cycles for every iteration”, while warning against letting the schema grow into a god-protocol.

The version of this that an agent needs is narrower than the general case, because the set of shapes an answer can take is small and closed. Drawing native UI from a spec covers what that closed set buys you.

The voice path#

Speech in, speech out, and the ability to be interrupted halfway through a sentence. Transcription is the easy part. The work is in knowing when the user has finished speaking, and in cancelling everything in flight the moment they start again.

The confirm gate#

Everything above is a reading feature until the agent can write. The moment it can move money, change an address, or cancel a booking, a transcript stops being consent. The gate is a protocol rather than a dialog box: the turn that selects a write operation stops short of running it and returns a description of what it would do, and a second, separate call runs it after the user agrees. Tiering that by risk is its own design problem, and it is the one legal will ask about first.

Build the other four, or adopt them#

The useful question here is who is on call when a speech vendor deprecates an endpoint at 2am. Cost comparisons in this category tend to price the build and ignore the decade of maintenance behind it.

Building is the right call in a narrow set of cases. If the agent is the product you sell, this stack is your moat. If a requirement rules every vendor out, such as on-device inference inside a regulated environment, the decision is already made. Outside those cases, the engineers you would assign to it are the ones who could be working on the thing only you can build. The comparison of what the current runtimes actually give you is the next thing to read if you are weighing it seriously.

One turn, end to end#

Here is the shape of a single request, including the part where the user has to agree before anything moves.

The SDK models a turn as five states: idle, listening, transcribing, thinking, speaking. Each needs something different on screen, and the boundary between the second and the third is where a user who pauses mid-sentence gets cut off.

The user’s own credentials travel with the request, so the agent can never reach anything that user could not reach by tapping through the app by hand. Design that in on day one; retrofitting it later is a rewrite.

What day one gets you, and what week two costs#

A first working turn is genuinely quick. Add a package, implement a delegate, point it at one read operation, and ask it a question out loud. On a good afternoon that is the whole of it.

Treat that milestone for what it measures. Nordic APIs, writing about time to first call (9 November 2021), puts it plainly: “Making a first call is very different from someone becoming an active user.”

Week two is where the work is. Naming and shaping the tools, writing error text a model can act on, designing what the confirm card shows, deciding which writes need a biometric, and then running a fixed set of questions repeatedly to see how often the agent picks the right operation. None of that is SDK work. All of it decides whether the feature survives contact with real users.

On speed: our own warm turns against a live merchant tenant land at roughly 2.5 to 3 seconds, and a cold one runs anywhere from 2.5 to 30. Almost none of that is our code. It is the round trip to the tenant’s own tool server plus an unprimed prompt cache. Sub-second claims here usually measure first audio out rather than a completed action.

What a hard language costs you#

Adoption is not the constraint people assume. Stack Overflow’s 2025 developer survey found 84% of the 33,662 who answered using or planning to use AI tools. On a separate question, answered by 33,244, only about a third said they trust the accuracy of what those tools produce, and of the 31,476 asked what frustrates them, 66% named “AI solutions that are almost right, but not quite”. Users arrive already knowing what an assistant is and already expecting it to be wrong sometimes. The design job is recovery, not novelty.

Language is a real constraint, and it is where most stacks quietly stop working. We run the Paymob merchant assistant in English and in Arabic, answering in Modern Standard Arabic with the whole interface flipped right to left. Getting there needed per-variety evaluation rather than a vendor’s language list, because a model that scores well on a standard variety can fall apart on how people actually speak. The method generalises to any language with meaningful regional variation, and it starts with measuring each variety separately rather than trusting an aggregate.

The rest of this cluster#

This guide is the stack. Fourteen posts underneath it take one decision each, and they divide into the case for building, the numbers to judge it by, and the verticals where the shape is already known.

Deciding whether to build. What an agent really costs to own is the nine workstreams behind the afternoon prototype. The business case and its missing number is how to model the return when the one input that decides the answer has no public benchmark. What typing on a phone costs your app is the friction the feature is meant to remove, with the measurement that supports it and the ones that do not. What a checkout screen costs takes the most-quoted abandonment figures apart and says which reasons a redesign can actually move.

Judging whether it worked. The seven metrics worth tracking names what each one hides, starting with the one that looks best. Why deflection is the wrong number is the same argument aimed at support, where a user who gave up scores as a success. The first week of real traffic is what live users teach that no test plan contains. Getting a user to first value covers the activation end, including four kinds of friction an agent genuinely cannot touch.

Where this is already happening. Which markets are adopting fastest matters if you are shipping outside your home country. The commerce statistics that survive checking is what was left after we opened every source behind the numbers people quote. A healthcare app has the hardest version of the escalation question, and the answer belongs in code rather than in a prompt. A logistics app is the case where hands-free is a legal requirement rather than a convenience. A travel app is where rebooking under disruption turns out to be the only task worth the build.

Questions people ask before they start#

What does the SDK own, and what do I still write?#

The SDK owns microphone handling, streaming, the conversation surface, the widget set, and the interruption behaviour. You write the tool layer over your own API, the theme, and the copy. You also decide which operations are readable, which are writable, and which need a biometric.

Do I need voice at all?#

No. The same agent answers typed questions and draws the same UI. Voice is one input mode, and it earns its place when the user’s hands or eyes are busy, or when the alternative is a form.

How do I know it works before I ship it?#

Write a fixed set of questions with known correct tool calls, and run the whole set repeatedly rather than once. An agent that picks the right operation eight times out of ten is not 80% working; it is a feature that fails for one user in five.

Can it act on behalf of a user safely?#

Only if writes are gated. Send the user’s own credentials with every turn, keep the write path as a separate call that runs after explicit approval, and never let the agent describe the approval method in what it says out loud.

How long does a real integration take?#

The first spoken turn takes an afternoon on any of the four platforms. What a day actually gets you is the honest schedule for everything after that, and it links the per-platform walkthroughs, which each come down to a handful of calls. A version you would put in front of paying users takes as long as your tool surface and confirm design take, which depends on how many operations you are exposing and how many of them move money.

Sources#

• Anthropic, Building effective agents, 19 December 2024. Accessed 12 September 2026.
• Anthropic, Writing tools for agents, 11 September 2025. Accessed 12 September 2026.
• Thoughtworks, Technology Radar: server-driven UI, Volume 34, April 2026. Accessed 12 September 2026.
• Stack Overflow, 2025 Developer Survey: AI. Accessed 12 September 2026.
• Nordic APIs, Why time to first call is a vital API metric, 9 November 2021. Accessed 12 September 2026.
• Model Context Protocol, Specification, revision 2026-07-28. Accessed 12 September 2026.