Who is Your Therapy App Working For?

Frank Pasquale

For the Symposium on The Law and Policy of AI, Robotics, and Telemedicine in Health Care.

Myriad programs in Apple’s App Store claim to address mental health concerns. A larger assortment of less-vetted apps crowd the virtual shelves of the Google Play Store. Cheap mental health apps have been a godsend for health systems pressed by austerity to cut costs, like Britain’s National Health Service (NHS). Via an “NHS Apps Library,” UK authorities have recommended at least fourteen apps for those suffering from depression and anxiety. Unfortunately, according to a study in the journal Evidence Based Mental Health, "the true clinical value of over 85% of NHS accredited mental health apps is at present impossible to determine." Only 2 of the apps studied applied validated metrics. Nor is it clear how app stores arrange their wares, elevating some software and occluding others. Nor are the politics and ideology of app makers apparent on first glance.

This opacity is important, because it is by no means clear that digital substitutes for (or even complements to) extant mental health professionals will live up to the type of fiduciary and other standards that are expected of human providers. Already articulated in the realm of digital assistants, these concerns will only be more pronounced in health care.

The long history of therapy programs offers some perspective on the current, crowded market for counseling and therapy in app form. The MIT computer scientist Joseph Weizenbaum developed one of the most notable programs, ELIZA (named after the Eliza Doolittle of Pygmalion, since the program copied speech patterns of users). The program mimicked a Rogerian psychotherapist. As Weizenbaum explained it, the user's statement was “terminated by a double carriage return which serve[d] to turn control over to ELIZA. ELIZA then analyze[d] the user's statement and generate[d] some response ... Control is then again with the user.” For example, in a 1966 paper, Weizenbaum offered the following as an example of the type of exchange that might occur:

User: Men are all alike.

ELIZA: IN WHAT WAY

U: They're always bugging us about something or other.

E: CAN YOU THINK OF A SPECIFIC EXAMPLE

U: Well, my boyfriend made me come here.

E: YOUR BOYFRIEND MADE YOU COME HERE

U: He says i'm depressed much of the time.

E: I AM SORRY TO HEAR YOU ARE DEPRESSED.

At first, Weizenbaum was proud of his creation. He showed it off to both faculty and staff at MIT. While he had worried the text interface would prove too much of a barrier to open and frank communication, that concern quickly dissipated. Users readily “confided” in the software--so readily, in fact, that Weizenbaum soon terminated his work on it, worried that it could usurp other, richer forms of therapeutic encounter. Given contemporary health technocrats' obsession with cutting costs, that concern was prophetic. In an era of "value-based purchasing," a $2.99 app may be an irresistible "replacement" for a psychiatrist.

The Rogerian method itself, focused so intently on the therapy subject’s own words, strikes critics as a narcissistic endeavor. Simply prompting a person to consider and reconsider their own assumptions, and characterizations of situations, recalls contemporary anxieties about selfies and filter bubbles. The self-reinforcing dynamic of personalization--where past clicks inform the choices on offer in the future--was prefigured in ELIZA’s repurposing of past user data. We might also worry about gender and other biases in programming. As I mentioned in a recent lecture in Sydney, if a dating app’s algorithmic scoring of “desirability” is programmed by a team that is 95% male, that is likely unrepresentative of the heterosexual population using the app. (Unless, of course, fake bot profiles are rife.) Similar considerations should inform the implicit model of "optimal response" embedded in mental health apps. In the case of the exchange above, for example, there may well be gender dynamics that a more feminist Rogerian approach would surface.

Another line of technology criticism focuses on users’ vulnerability to undue manipulation by unknown forces. Consider, for instance, a therapy app “treating” a worker who complains a great deal about their time on the job. The worker feels underpaid and undervalued, and expresses those concerns to the app. There are diverse potential responses to such a problem. For example, the app might counsel assertiveness, pushing the worker to ask for a raise. At the other extreme, the app might prescribe a contemplative resignation to one’s fate, urging appreciation of all one already has. Or it might maintain a studied neutrality, digging ever deeper into the reasons for the worker’s unease. Guess which response an employer may want to see in wellness apps provided to its workers?

The great promise of predictive analytics in health care is the ability to find optimal ways of delivering care. But in the case of run-of-the-mill mental health concerns, there may be multiple ways to define the problem. Different commercial models can encourage different ways of defining mental illness, or its treatment. A free app with an advertising-based model may want to encourage users to return as often as possible. A subscription-based service would not necessarily optimize for “time on machine,” but might aim to use other forms of manipulation to promote itself. Self-reports of well-being may be an uncontroversial “ground truth” against which to measure the value of apps. But the concept of wellness is being colonized by corporations and governments, and tied intimately to more "objective" measures, like productivity.

Esteemed academics and activists have pointed out problematic biases among doctors and other providers. An algorithmic accountability movement in medicine will need to carry this work forward in critiquing, and fixing, the biases and other problems that will afflict computationally-inflected caregiving. Ultimately, the best structural safeguard is to assure that most apps are developed as intelligence augmentation (IA) for responsible professionals, rather than as AI replacing them. Many other aspects of health law and policy (such as licensure, reimbursement, and other rules) should also play a role in assuring humanistic (rather than behavioristic) mental health apps.
Posted 3:56 PM by Frank Pasquale [link]