For the Balkinization Symposium on the Global Political Economy of Artificial Intelligence.
Claudia E. Haupt
Professional
AI’s Dual Trust Problem
Each day,
more than forty million people ask ChatGPT health
questions. When OpenAI and Anthropic launched dedicated health AI tools that
let users upload their medical records and receive personalized guidance, the
obvious question was: “Should you trust them?” The question has only grown more pressing: since early 2026, five major
technology companies (OpenAI, Anthropic, Microsoft, Amazon, and Perplexity)
have released or expanded dedicated consumer-facing AI health applications,
each allowing users to connect medical records, lab results, and wearable data
to receive personalized guidance.
That question, it turns out, has
more than one layer. Instinctively, we might assume the concern is about output
accuracy: will the AI give bad professional advice? But a second, perhaps less
obvious, problem arises that reaches beyond any individual bad outcome.
Untrustworthy AI undermines the entire system of trust that makes human
professional advice work in the first place. It’s a dual trust problem.
I examine professional AI’s dual
trust problem in a forthcoming essay, Misplaced Trust in Artificial
Professional Advice,
which makes the argument in two steps.
The
First Problem: AI’s Professional Advice Isn’t Trustworthy
The professional relationship
with a doctor, lawyer, financial advisor, accountant, pharmacist, therapist, or
another advice-giving professional is a specific social interaction. The
professional possesses knowledge the client lacks; this results in a knowledge
asymmetry that creates vulnerability. The law responds with a set of
safeguards: licensing requirements, fiduciary duties, malpractice liability,
informed consent. These mechanisms protect the conditions under which a client
or patient can reasonably place confidence in a professional’s expertise.
Public-facing AI eliminates the
human professional. What remains looks like professional advice:
conversational, personalized, authoritative in tone. But the legal and ethical
framework that ensures professional advice is trustworthy is absent.
The accuracy problem is real and
documented: as of 2024, no commercially available AI app met professional
standards for skin cancer detection. Earlier studies on general health queries
found frequent errors, and more recent work reinforced those findings. A study
published in Nature Medicine found that participants using AI
chatbots to navigate common medical scenarios performed no better than a
control group relying on ordinary home resources such as internet searches—and
users describing the same symptoms sometimes received conflicting advice
depending on how they phrased their questions. A separate Mount Sinai study found that ChatGPT Health under-triaged
more than half of medical emergencies in structured clinical testing,
potentially directing patients with serious conditions toward routine follow-up
rather than urgent care.
But accuracy is not even the core
issue. Trust is an attitude; trustworthiness is a property. And as Ignacio
Cofone argues in a companion piece to this symposium, as well as in more detail
in a forthcoming article,[1]
trustworthiness is a property of institutions, not of AI systems. The
professional relationship, not the chatbot, carries that institutional
trustworthiness.
The
Second Problem: Untrustworthy AI Undermines Trust in Human Professionals
When a patient consults a
public-facing AI and then sees a physician whose advice diverges, the patient
faces a question they are not equipped to answer: who is right? And behind that
question lurks a more unsettling one: where does expertise actually live?
The proliferation of AI that
mimics professional judgment creates epistemic uncertainty about institutional
expertise itself. Beyond harming individual users, the AI systematically
undermines confidence in the professionals it displaces or contradicts. The
problem is structural, rooted in the same inequities of access that drive
people to seek AI as a substitute for healthcare in the first place.
Many people turn to AI health
tools precisely because they lack access to affordable human care. A March 2026
KFF tracking poll found that about one in five
adults who use AI for health advice cite inability to afford a provider as a
major reason, a figure that rises to nearly three in ten among users ages 18 to
29. Uninsured adults are more than twice as likely as insured adults to rely on
AI for mental health guidance. And the pattern tracks race: Black and Hispanic
adults turn to AI for mental health advice at substantially higher rates than
White adults. Viewed this way, AI health tools are an attempted patch for a
broken system.
The trust being displaced was
already fragile, and unevenly distributed across race, income, and geography.
Worse, the AI health tools with the most personalized features—those enabling
direct integration with medical records—are increasingly behind paywalls,
potentially placing them out of reach for those who are already struggling to
afford care. What consumer-facing health AI offers is not a substitute for the
human professional relationship.
As the law and political economy
literature would recognize, the roots of this problem predate AI. First
Amendment doctrine, as I have argued elsewhere, has long assumed the
availability of professional advice without reckoning with its unequal
distribution.[2] This assumption places a
heavier burden on those who can least afford expert counsel and who are most
dependent on publicly available information (however unreliable) as a
substitute. Consumer-facing health AI does not solve this problem; it exploits
it, offering a widely available facsimile of expert advice. And the data
suggests it’s relied on by users for whom the absence of access to professional
advice was already most consequential.
Trust in Institutions, Not AI
The stakes extend beyond
individual harm to institutional erosion. As Woodrow Hartzog and Jessica Silbey
argue, AI has the capacity to destroy the civic and professional institutions
on which public life depends.[3] It
may do so by steadily undermining the trust that sustains them. The professions
are no exception. Professional expertise generates trust because it is grounded
in training, accountable to standards, and answerable to the people it serves.
Deploying AI that mimics expertise without embodying any of those properties
creates bad individual outcomes and casts doubt on where expertise lives.
The question, then, is not simply whether we should trust AI. It is whether deploying untrustworthy AI erodes the very institutions whose trustworthiness we depend on, and what regulatory frameworks built around human professional relationships can do about it.
Claudia E. Haupt is Professor of Law and Political Science, Northeastern University. You can reach her by e-mail at c.haupt@northeastern.edu.
[1] Ignacio Cofone, Institutional
Accountability and Legitimate Inference in Algorithmic Adjudication: Beyond
Trustworthy AI, Cambridge Forum on AI Law and Governance (forthcoming
2026), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6516459
[2] Claudia E. Haupt, Assuming
Access to Professional Advice, 49 J.
Law, Med. & Ethics 531 (2021).
[3] Woodrow Hartzog & Jessica
Silbey, How AI Destroys Institutions, UC L. REV. (forthcoming 2026)
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5870623.