Professional AI’s Dual Trust Problem

Guest Blogger

For the Balkinization Symposium on the Global Political Economy of Artificial Intelligence.

Claudia E. Haupt

Professional AI’s Dual Trust Problem

Each day, more than forty million people ask ChatGPT health questions. When OpenAI and Anthropic launched dedicated health AI tools that let users upload their medical records and receive personalized guidance, the obvious question was: “Should you trust them?” The question has only grown more pressing: since early 2026, five major technology companies (OpenAI, Anthropic, Microsoft, Amazon, and Perplexity) have released or expanded dedicated consumer-facing AI health applications, each allowing users to connect medical records, lab results, and wearable data to receive personalized guidance.

That question, it turns out, has more than one layer. Instinctively, we might assume the concern is about output accuracy: will the AI give bad professional advice? But a second, perhaps less obvious, problem arises that reaches beyond any individual bad outcome. Untrustworthy AI undermines the entire system of trust that makes human professional advice work in the first place. It’s a dual trust problem.

I examine professional AI’s dual trust problem in a forthcoming essay, Misplaced Trust in Artificial Professional Advice, which makes the argument in two steps.

The First Problem: AI’s Professional Advice Isn’t Trustworthy

The professional relationship with a doctor, lawyer, financial advisor, accountant, pharmacist, therapist, or another advice-giving professional is a specific social interaction. The professional possesses knowledge the client lacks; this results in a knowledge asymmetry that creates vulnerability. The law responds with a set of safeguards: licensing requirements, fiduciary duties, malpractice liability, informed consent. These mechanisms protect the conditions under which a client or patient can reasonably place confidence in a professional’s expertise.

Public-facing AI eliminates the human professional. What remains looks like professional advice: conversational, personalized, authoritative in tone. But the legal and ethical framework that ensures professional advice is trustworthy is absent.

The accuracy problem is real and documented: as of 2024, no commercially available AI app met professional standards for skin cancer detection. Earlier studies on general health queries found frequent errors, and more recent work reinforced those findings. A study published in Nature Medicine found that participants using AI chatbots to navigate common medical scenarios performed no better than a control group relying on ordinary home resources such as internet searches—and users describing the same symptoms sometimes received conflicting advice depending on how they phrased their questions. A separate Mount Sinai study found that ChatGPT Health under-triaged more than half of medical emergencies in structured clinical testing, potentially directing patients with serious conditions toward routine follow-up rather than urgent care.

But accuracy is not even the core issue. Trust is an attitude; trustworthiness is a property. And as Ignacio Cofone argues in a companion piece to this symposium, as well as in more detail in a forthcoming article,[1] trustworthiness is a property of institutions, not of AI systems. The professional relationship, not the chatbot, carries that institutional trustworthiness.

The Second Problem: Untrustworthy AI Undermines Trust in Human Professionals

When a patient consults a public-facing AI and then sees a physician whose advice diverges, the patient faces a question they are not equipped to answer: who is right? And behind that question lurks a more unsettling one: where does expertise actually live?

The proliferation of AI that mimics professional judgment creates epistemic uncertainty about institutional expertise itself. Beyond harming individual users, the AI systematically undermines confidence in the professionals it displaces or contradicts. The problem is structural, rooted in the same inequities of access that drive people to seek AI as a substitute for healthcare in the first place.

Many people turn to AI health tools precisely because they lack access to affordable human care. A March 2026 KFF tracking poll found that about one in five adults who use AI for health advice cite inability to afford a provider as a major reason, a figure that rises to nearly three in ten among users ages 18 to 29. Uninsured adults are more than twice as likely as insured adults to rely on AI for mental health guidance. And the pattern tracks race: Black and Hispanic adults turn to AI for mental health advice at substantially higher rates than White adults. Viewed this way, AI health tools are an attempted patch for a broken system.

The trust being displaced was already fragile, and unevenly distributed across race, income, and geography. Worse, the AI health tools with the most personalized features—those enabling direct integration with medical records—are increasingly behind paywalls, potentially placing them out of reach for those who are already struggling to afford care. What consumer-facing health AI offers is not a substitute for the human professional relationship.

As the law and political economy literature would recognize, the roots of this problem predate AI. First Amendment doctrine, as I have argued elsewhere, has long assumed the availability of professional advice without reckoning with its unequal distribution.[2] This assumption places a heavier burden on those who can least afford expert counsel and who are most dependent on publicly available information (however unreliable) as a substitute. Consumer-facing health AI does not solve this problem; it exploits it, offering a widely available facsimile of expert advice. And the data suggests it’s relied on by users for whom the absence of access to professional advice was already most consequential.

Trust in Institutions, Not AI

The stakes extend beyond individual harm to institutional erosion. As Woodrow Hartzog and Jessica Silbey argue, AI has the capacity to destroy the civic and professional institutions on which public life depends.[3] It may do so by steadily undermining the trust that sustains them. The professions are no exception. Professional expertise generates trust because it is grounded in training, accountable to standards, and answerable to the people it serves. Deploying AI that mimics expertise without embodying any of those properties creates bad individual outcomes and casts doubt on where expertise lives.

The question, then, is not simply whether we should trust AI. It is whether deploying untrustworthy AI erodes the very institutions whose trustworthiness we depend on, and what regulatory frameworks built around human professional relationships can do about it.

Claudia E. Haupt is Professor of Law and Political Science, Northeastern University. You can reach her by e-mail at c.haupt@northeastern.edu.