An agent in a healthcare app: booking, refills, and the stop line

Almost every published example of AI in healthcare is a phone system. It answers the clinic’s line, books an appointment, and hands off to a human when the caller says something it was not built for. That product exists because clinics drown in calls, and it has nothing to do with the app the patient already has on their phone.

This post is about the other one: an agent inside the patient-facing app, where the patient is already authenticated, already looking at their own record, and already holding the device that can approve an action. That changes what is worth building, and it changes what the agent should be allowed to see.

It assumes you have already read the five pieces you are signing up to own, because the scoping decisions below only make sense once you know which of those pieces is yours. Everything here is about the tool layer, which in this vertical is where the regulatory weight sits.

Three tasks, and no more to start with#

Finding and moving appointments is the task with the clearest shape. A patient knows roughly when they want to come in and does not know what is free. The app’s existing booking screen makes them do the search. An agent takes “something after work next week” and comes back with three slots, which is a query your backend can already answer and a sentence your UI cannot currently accept.

Rescheduling is the same tool with a write on the end, which means it goes through a confirm step. The patient sees the old time, the new time and the clinician, and taps. Nothing moves on the strength of a transcript.

Refill eligibility is the third, and the interesting design decision is what comes back. A tool that answers “is this refillable” can return a verdict, a date and a route, without returning the prescription. The agent never learns the drug, and a conversation that never contained the drug cannot leak it.

What is not on this list: results, notes, diagnoses, and anything that would make the agent a clinical instrument. Those are genuinely useful and they raise the cost of every other decision in this post by an order of magnitude.

Reminders sit slightly apart, because a reminder is something your system already sends. What an agent adds is a reply channel. A patient who receives “you have an appointment on Thursday at 2” can currently do nothing with it except open the app and navigate. If the reminder can be answered, “can we make it Friday” becomes one turn instead of a phone call to a receptionist who is already on another line.

What the tools should return#

The design that keeps this feature cheap is in the return shapes, and it is worth deciding before anyone writes a prompt.

A slot search returns times and clinician names. It does not return the appointment type, because the type is usually the reason for the visit, and a list of times is enough to have the conversation.

A refill check returns one of a small set of outcomes: refillable now, refillable after a date, needs the prescriber. Each outcome maps to a sentence the agent can say and, where relevant, an action it can propose. None of them requires the drug name to enter the conversation.

An appointment lookup returns the patient’s own upcoming appointments and nothing historical. History is where the sensitive content lives, and a patient who wants it has a screen for it already.

Three tools, each returning the smallest useful answer. That constraint does more for the privacy posture than any amount of prompt engineering, and it also makes the agent noticeably more reliable, because a model choosing between three well-named operations makes fewer mistakes than one choosing between thirty.

The stop line#

Triage is where these products go wrong, and the failure is not that the model is bad at medicine. The failure is that a system built to be helpful is built to answer, and the correct response to a subset of questions is to stop answering.

Draw the line by category, in code, before the model sees the turn. A question classified as symptom-related does not get routed to a careful prompt about being cautious. It gets routed to a response the model does not author.

This is the same argument OWASP makes about excessive agency in a different vocabulary: the fix for an action you do not want taken is to remove the capability, not to ask nicely. The wider threat model for an agent that can act covers where else that reasoning applies, and the approval step written as a branch in code covers the mechanics of intercepting a call before it runs.

The compliance question, answered plainly#

Under 45 CFR 160.103, a business associate is a person or organisation that “creates, receives, maintains, or transmits” protected health information on behalf of a covered entity, and the definition reaches subcontractors too. A model provider, a speech recognition vendor and an agent runtime that sees a patient’s record are all in that chain, and each link needs an agreement.

Our own position, stated rather than implied: Voqal publishes commitments around GDPR, CCPA and PDPL, and a SOC 2 audit in progress. HIPAA is not on that list today. If your app handles protected health information, the design above is the one that keeps working anyway, because it keeps the record out of the agent’s hands: eligibility verdicts instead of prescriptions, slot times instead of visit reasons, and identifiers resolved on your side of the boundary.

That is the same minimisation you would want with every agreement signed, because the alternative is a conversation log that contains a diagnosis.

Consent is a set of moments, not a screen#

The moments that matter are the ones where a specific thing is about to happen and the patient can stop it. A visible state telling them the microphone is open. A confirm card naming the appointment being moved. A way to interrupt mid-answer. A history they can read afterwards.

Our SDK runs five states for the first of those: idle, listening, transcribing, thinking and speaking, with transcribing separated out so the screen can show that words were captured before any answer exists. In a clinical context that separation earns its keep, because “it heard me” and “it is deciding” are different anxieties.

Discoverability decides whether any of this is used#

The hardest problem in a patient app has nothing to do with the model. Nobody knows the agent is there, and nobody guesses what it will accept.

A blank prompt gets “hello” and then nothing. A short list of the three tasks above gets used, and the list doubles as a statement of scope: a patient who reads “check a refill, find an appointment, move an appointment” has been told, without a disclaimer, that this is not where symptoms go.

Who benefits most, and what it costs them to fail#

The accessibility argument for speech in healthcare is real and it is usually overstated. Masina and colleagues ran a mixed-methods study of voice assistants with impaired users in the Journal of Medical Internet Research (2020), with 16 participants performing voice-controlled tasks. Average task accuracy came out around 58.5%, and performance was predicted by cognitive measures including MMSE score and the ability to repeat sentences.

Read that as encouragement and as a warning. Speech opens the app to people for whom a form is hard, and the users who benefit most are also the users for whom a failed turn is most costly. Error recovery, a visible transcript and an obvious route to a human are not polish in this vertical.

Where the value actually is#

The strongest case for this feature sits in the class of task where a patient’s intention already exists and the interface is what stops them acting on it, which is a different question from how many visits the app drives.

Medication is the clearest example. CDC’s Grand Rounds on medication adherence, published in MMWR in 2017, reports that approximately one in five new prescriptions are never filled, and that of those filled, roughly 50% are taken incorrectly with respect to timing, dosage, frequency or duration. It puts direct health care costs associated with non-adherence at approximately $100 to $300 billion annually in the United States.

An agent does not fix that. What it can remove is one specific friction: the patient who meant to request a refill, opened the app, could not find the screen, and closed it. Whether that friction is worth a project is a question your own funnel answers, and the deflection numbers that support teams quote are the wrong instrument for measuring it.

If you are still deciding whether this category fits your product at all, start with the difference between an agent and a help screen. If you have decided, how the confirm gate is configured covers the confirm interception and the device binding described above.

Sources#

• Legal Information Institute, Cornell Law School, “45 CFR 160.103, Definitions”. Accessed 12 September 2026.
• Centers for Disease Control and Prevention, “CDC Grand Rounds: Improving Medication Adherence for Chronic Disease Management”, MMWR 66(45), 2017.
• Masina, F. et al., “Investigating the Accessibility of Voice Assistants With Impaired Users: Mixed Methods Study”, Journal of Medical Internet Research, 2020.
• OWASP, “LLM06:2025 Excessive Agency”. Accessed 12 September 2026.