When Consumer AI Routes Patients to Care

A person has felt pressure in the centre of their chest for two hours. They ask a health assistant what to do. The assistant recognises the risk and recommends urgent care. Most medical AI evaluations would stop there.

The person still has to reach care. Which hospital appears next? Is it in network? Can the person afford it and get there? Does the hospital have the right service? A medically correct answer can still lead to no care.

Consumer health AI is moving into this gap. Systems now connect health conversations with records, provider discovery, booking, pharmacies, payment, and administrative work.¹ Once an assistant constructs the visible choices, ranks them, and acts on the patient’s selection, we are no longer evaluating only an answer. We are evaluating a pathway and the operator that shaped it.

Before the system can route anyone, it has to work out what problem it is dealing with. Wachter calls the clinician’s compressed account a problem representation. A medical AI should be able to turn the janky stream of consciousness most patients type into ChatGPT or Claude into a useful working problem representation. The patient should not have to learn to prompt like a doctor. Models will keep improving in that direction, but they cannot reliably do this today.²

The clinical summary should be linked to patient-confirmed goals, constraints, and communication preferences. A non-AI emergency-department study of the Humanistic Charting Tool placed patient narratives in the EHR. Experience ratings improved relative to previous encounters, but improvement did not differ significantly by clinician use of the summary.³ The design cannot establish a causal benefit. Routing evaluation should distinguish elicitation, faithful transfer, use in care, and patient-reported listening and involvement. Patients need opportunities to correct consequential details and control sharing.

Clinicians and patients should also share responsibility for defining evaluation outcomes. Clinicians can assess clinical decisions; patients can assess whether the system understood their concerns and proposed a workable route. Both should help construct cases and prespecify how disagreements will be handled, building on established patient and public involvement methods in healthcare AI.⁴ Trained standardized patients and virtual patients can test repeatable predeployment encounters, followed by studies with patients measuring experience, burden, safety, access, and outcomes.

Wachter writes that an AI-first system will require “a trustworthy triage protocol” and the harder task of “sorting through these scenarios, parsing what should be done by AI and what should be done by humans.”⁵

Consumer AI, once it begins acting inside healthcare, becomes a rather overloaded router. Wachter is describing mode routing: deciding “whether” the AI should continue or safely hand off the patient to a human doctor. Ant’s Afu makes this a live design question; ChatGPT Health may do so as it grows more connected to care.

Destination routing describes “how that handoff occurs”: which clinicians appear in the choice set, in what order, and whether they fit clinical need, patient preferences, location, insurance coverage, likely cost, language, and availability. The same system may increasingly control both decisions. We need to evaluate both.

Some patients may prefer to start with AI. For stigmatized concerns, including sexuality or thoughts of self-harm, ChatGPT or Claude may initially feel easier and less judgmental than another person. Severe worldwide shortages of mental-health professionals increase the possible access benefit. Current LLMs still mishandle some crises, so easier disclosure makes safe human escalation more important, not less.⁶

Routing also makes these systems ranking and recommendation engines. When an assistant shows three doctors, those three become the practical choice set. A recent synthetic preprint found that ratings, fees, position, demographic signals, and patient personas changed LLM recommendations, while explanations omitted some drivers.⁷ Audits must examine the candidate pool, ranking, and executed route.

Care cascades already measure dropout, implementation science covers adoption and abandonment, and PACT follows continuity, escalation, and completion.⁸ Access researchers have long distinguished available care from care that people can actually use. Agentic consumer AI can place these functions under one operator, which may frame the problem, build the choice set, rank it, act, observe the result, and earn money along the route.

The figure shows the change in the evaluation object. Model measures still include clinical accuracy, calibration, uncertainty, and emergency sensitivity. Pathway measures include clinical appropriateness, access, time, cost, completion, utilization, and outcome. Operator measures include candidate generation, ranking, affiliation, transaction, data use, human fallback, and exit. We should not squeeze all three into one score.

Human fallback must remain real. Modern flights rely heavily on autopilot, yet pilots return to simulators to practise rare failures and preserve core skills. Medicine may need the same balance. Evidence of AI-related de-skilling is early and mixed.⁹ If a catastrophic worldwide AI outage shut down OpenAI, Anthropic, and similar services, doctors should still provide safe care with zero AI assistance. Training programmes and health systems should test that capability.

Should regulators and qualified independent groups be able to audit ranking and recommendation? We strongly believe so. They need protected access to reconstruct what the system considered, excluded, ranked, and executed, including relevant commercial relationships. Hospitals must monitor system integrity, clinical performance, pathway outcomes, and ranking conduct, with an owner and response for every measure.¹⁰

Audit measures also need validation. In a single-center hospital analytics preprint, an agent produced executable queries but misinterpreted some cohort and procedure definitions. A procedure-count error persisted through five feedback rounds and was reproduced when its definition was reused.¹¹ Clinical and operational experts should independently validate the records used to establish eligibility and completed care. Agreement across runs cannot substitute for correctness.

Comparable functions need comparable duties across general assistants, health apps, insurer portals, hospital applications, and marketplaces. A company should not escape scrutiny by calling a routing system a general assistant.

Advertising makes the argument uncomfortable. We should not ban every separately displayed healthcare advertisement. A labelled notice can tell a patient that a clinic exists, speaks a needed language, or has appointments. But providers, insurers, pharmacies, and integrated health companies must not buy a higher place in a list presented as the best clinical route. Platforms must keep paid placements out of emergency routing, crisis response, treatment selection, and clinically indicated referrals.

Deeper integration creates the central tradeoff. It can remove handoffs, reduce administrative burden, and help more patients complete appropriate care. The same integration can concentrate ranking power, propagate an error across several steps, and make it harder for patients to compare or leave. We should measure both sides rather than assume that integration is either coordination or capture.

India offers a useful institutional test. The National Health Authority designed the Unified Health Interface as an open network for discovering, booking, paying for, and completing health services across different applications.¹² Researchers should compare open and closed systems on public-provider exposure, price, language, completion, portability, and exit. Open architecture is a hypothesis, not proof of fair care.

An answer can be medically right and still send a patient down a path that may not survive contact with reality. Maybe the care costs too much. Maybe it is too far away. Maybe family or cultural circumstances make it unworkable. Maybe it does not fit what the patient wants or is willing to do. We tend to file all of this under “non-clinical factors,” but whether we like it or not, these factors govern how care is negotiated and delivered in the real world.

What must a health agent log? Who should inspect it? Which commercial relationships must companies disclose? When should regulators prohibit paid placement? Who gets to stop the system when the pathway becomes unsafe?

The medical AI community should answer these questions before product defaults answer them for us.

Not by vibes.

I don’t have those answers. Maybe none of us do yet. But I think consumer AI is already starting to reshape the infrastructure through which people find and receive care. We need to work through these questions together, as a community, while we still have some say in how this new world of agentic medicine is built. I may not know what all the rules should be, but I know what they need to protect: patients, their ability to choose, and medicine’s obligation to do no harm.