Interview on the Black Box Society

Guest Blogger

Lawrence Joseph and Frank Pasquale

In the interview below, Lawrence Joseph interviews Balkinization blogger Frank Pasquale about his forthcoming book, The Black Box Society: The Secret Algorithms That Control Money and Information. Joseph is Tinnelly Professor of Law at St. John's School of Law, and a poet and author of literary prose and essays. Pasquale teaches at the University of Maryland and serves on the Council on Big Data, Ethics, and Society.

Lawrence Joseph: First of all, I want to say that I consider The Black Box Society a monumental and stunning achievement--its truly extraordinary range of critical analyses and research, its clear and masterfully written style. You mention that the book took ten years to write. Could you explain your project and how it evolved over a time of unprecedented changes in global technologies and political economies?

Frank Pasquale: Thank you, Larry. This project began with a contrarian hunch about search engines. As Google grew in the early 2000s, the primary policy question seemed to be: “how do we get law out of the way of this company so it can keep organizing the internet?” I shared that optimism at first--but I was also concerned about the downside.

I started compiling stories about people who felt Google was treating them unfairly--by, say, disappearing their site from results, or giving prominence to scurrilous or salacious material. I looked at firms with similar business models, ranging from data brokers to social networks. All used technologies of search to order incredibly diverse content.

Data-intensive technology kept spreading. Not only product choices, but personal reputations were increasingly determined by algorithms, too. The way we see the world was filtered through them. And as results got more personalized, reputation and search became intertwined, mutually reinforcing: what a search engine knows about me helps it anticipate what I want, even on the basis of a few letters typed into a search bar. But the actual programming behind such critical determinations was hidden behind layers of trade secrets and nondisclosure agreements.

Since this was such a new area for law, there were hundreds of ideas one could write up in response to it. As I tried to prioritize, my guiding principle was: where are search and reputation technologies most important? Briefly, my answer was: force and finance. Two cataclysmic Septembers (2001 and 2008) seared them in my mind.

The force sector—from local police to DOD to DHS and their many private contractors—garnered extraordinary new powers after 9/11. The finance sector is our arbiter of opportunity, making money cheap for some people and expensive for others. As it broke down in 2008, it became clear that a government ever more minimalist and lethargic on the economic front could suddenly transform into a Hamiltonian Leviathan when elite firms faltered. Both the force and finance sector were using new technology to accomplish incredibly fine-grained surveillance, and were being transformed by computerization. That transformation, I worried, was simply entrenching current inequalities and inequities, rather than challenging them.



L.J.: You bring to what has become a vast and sophisticated techno-politico-economic and cultural literature the sensibility, skills, and imagination of a lawyer and legal scholar. This makes you virtually unique…

F.P.: I think our academic culture is very good at analysis, but oft-adrift when it comes to synthesis. Specialization obscures the big picture. And law can succumb to this as easily any other field. For example, in the case of internet companies, cyberlawyers too often confine themselves to saying: “Google and Facebook should win key copyright cases, and subsequent trademark cases, and antitrust cases, and get certain First Amendment immunities, and not be classified as a ‘consumer reporting agency’ under relevant privacy laws,” etc. They may well be correct in every particular case. But what happens when a critical mass of close cases combines with network effects to give a few firms incredible power over our information about (and even interpretation of) events?

Similarly, old banking laws may fit poorly with the new globalized financial landscape. Finance lawyers churn out position papers dismantling the logic of Dodd-Frank, Basel, Sarbanes-Oxley, etc. But if too-big-to-fail firms keep growing bigger, assured of state support, while everything else the government does is deemed contingent: what kind of social contract is that?

The lawyers of the Progressive Era and the New Deal dealt with similar challenges: massive firms that warped the fabric of economic, political, and even cultural life to their own advantage. They consulted the best of social science to recommend regulation—but they didn’t let some narrow field (like neoclassical economics) act as a straitjacket (as, say, antitrust lawyers of today are all too prone to do).

The great promise of legal scholarship has been a body of professionals drawing from many disciplines to offer integrated, considered judgments about how to resolve disputes. Legal practice both feeds into those judgments, and improves their implementation. Chapter 5 of my book offers practical approaches to improving the law governing reputation and search, and reining in the power of the force and finance sectors.

L.J.: The epigraphs that you’ve chosen—quotations from Heraclitus and Gerard Manley Hopkins—could you explain their significance to the book?

F.P.: The Heraclitus—“For the waking there is one world, and it is common—but sleepers turn aside, each into a world of his own”—had a few resonances for me. It reminded me of the work of people like Nicholas Carr and Sherry Turkle, who are such insightful analysts of technoculture. They anticipated how the integration of reputation and search via personalization could isolate us--to the extent that the “imagined communities” of nations, cities, churches, even families would disintegrate into hypnotic fascination with whatever images, sounds, and text that algorithms determined would maximize users’ “time-on-machine.” The film Her is an extraordinary evocation of such a future, too. My book is an effort to develop some common, wakeful awareness of whom the algorithms behind the screen are ultimately serving.

The Heraclitus quote is also the epigraph for Interpretive Social Science: A Reader--a book that’s influenced me since college. Interpretive social scientists try to explain events as a text to be clarified, debated, argued about. They do not aspire to model our understanding of people on our understanding of atoms or molecules. The human sciences are not natural sciences. Critical moral questions can’t be settled via quantification, however refined “cost benefit analysis” and other political calculi become. Sometimes the best interpretive social science leads not to consensus, but to ever sharper disagreement about the nature of the phenomena it describes and evaluates. That’s a feature, not a bug, of the method: rather than trying to bury normative differences in jargon, it surfaces them.

For me, Charles Taylor is the key philosopher of interpretive social science. Much of his early work was inspired by an engagement with the behaviorists and artificial intelligence (AI) researchers of his day. They aspired to analyze (and eventually mimic or manipulate) only the observable actions of persons, with very little or no attention paid to internal thought processes. Taylor showed why this is a recipe for bad social science, and an inhumane society. The reasons for action matter. If people are simply herded along toward maximally productive activity (via nudges or shoves), rarely if ever questioning the how or why of their own opportunities or aspirations, we may well maximize GDP, but to what end?

By the early 1970s, philosophers like Taylor helped change the climate of opinion around AI and behaviorism, deflating both their goals and methods. But both ways of thinking have reemerged today in the fetishization of “big data,” which is touted as a way to understand and control society without reference to the history (or patterns of thought) that gave rise to the data analyzed. A finance firm may say, for example, “we charge 15% interest to someone who had a past default, just because past patterns of data show that such people often default again,” in a process agnostic as to whether a defiant refusal to repay, or a family medical emergency, caused the prior default. The police may say, “we’re intensively policing this neighborhood because it had 10% more crime in the past.”

But what if defaults resulted from excessive interest rates in the past, caused by discriminatory lending practices? And what if the above-normal crime in the neighborhood simply reflected past patterns of intense policing that reflected racism? What if each decision makes future defaults, or excess crime rates, more likely? Then the “science of society” promised by big data morphs into a subjugation of certain parts of society. The algorithms behind such judgments become less “objective arbiters” of opportunity and punishment, than ways of laundering subjective, biased decisions into ostensibly objective, fair scores. Those affected lose a chance at individualized treatment and understanding, as technical systems treat people as a mere collection of data points.

In short, if we forget the human origins and purpose of algorithmic judgments, we lose our chance to develop a humane (and intelligible) society. That is where the Gerard Manley Hopkins epigraph (“quench [nature’s] bonniest, dearest to her, her clearest-selvèd spark / Man, how fast his firedint, his mark on mind, is gone!”) inspires my work. Like so much of Hopkins’ poetry, it is a vivid reminder of the dignity of persons, and how easy it is to treat them as mere objects when “all is seared with trade.”

L.J.: The book’s method of analysis--could you explain it?

F.P.: I try to read widely in the social sciences. This is a response to the work of Ian Shapiro (who has called for problem-driven research to complement method-driven work), Russ Roberts (a long-time critic of mathematical modeling in economics), and Thomas Piketty (who has integrated historical, economic, and literary approaches in Capital in the 21st Century).

The Black Box Society’s central subject--agnotology, the suppression or destruction of knowledge--is a particularly difficult topic to interpret methodically. But I’ve tried to highlight some very important disputes, show their broader relevance, and explain what laws would need to change for us to really understand the value of what data brokers, search engines, financiers, or homeland security contractors are doing. I justify those policy proposals with reference to emerging work in more normatively oriented branches of political economy and social science.

I also take some inspiration from Nancy Krieger, who challenged those who attributed events to a “web of causation” to think more deeply about what united the causes. “Where is the spider?”, Krieger would ask about such webs. In my case, events like the financial crisis still strike many as the result of incredibly complex and opaque transactions, bets, and promises. But who ultimately benefits? Which of their actions did they trumpet, and which did they hide? Who is trapped in the web, and who glides effortlessly on top of it?

L.J.: You mention the need for more legal scholarship in political economy—was the filling of this gap one of the book’s ambitions?

F.P.: Yes, absolutely. Political economy is a venerable discipline. While it has, of late, been dominated by “positive political economists” focused on the pathologies of governance, there is a venerable tradition of political economists studying the “ideal role of the state in the economic and social organization of a country” (as Piketty puts it). Lawyers are particularly well-suited to the task of studying political economy, because we are the ones drafting, interpreting, and applying the rules governing the interface between state actors and firms.

Integrating the long-divided fields of politics and economics, a renewal of modern political economy could unravel “wicked problems” neither states nor markets alone can address. But it’s actually more urgent than that, because the very terms “state” and “market” seem antiquated. For example, Medicare may be publicly funded, but it’s ultimately run by a panoply of private contractors. Banks may make tremendous profits from financial “markets,” but the main reason they have deposits and counterparties to deal with is governmental guarantees that take the sting out of credit risk—and, in turn, reward many of those administering such guarantees with lucrative jobs once they leave government.

So a purely economic approach to “markets” here, or a purely political approach to “states,” misses the critical interaction between the two. A political economic approach is vital—and that’s what has made social theory ranging from Smith and Mill, to Tocqueville and Durkheim, to Weber and Habermas, of such enduring interest. In law, we still read Robert Lee Hale and the legal realists for exactly the same reason. My concluding chapter tries to revive this political economic perspective, suggesting reforms beyond the purely legal concerns of the penultimate chapter.

L.J.: How would you summarize the book’s primary themes and factual claims?

F.P.: The book critiques Silicon Valley and Wall Street, offers a guide to what a better legal response to their abuses might be, and serves as an introduction to a political economy of information. Let me take each of these in turn.

Here’s one paradox I’m trying to resolve: even as economic change seems to accelerate, existing hierarchies of power and wealth seem ever more ossified, stable, secure. I trace this problem to two, mutually reinforcing trends: the financialization of data, and the data-fication of finance.

Data has always been important. But with new surveillance technologies, it’s possible to automate huge swathes of economic activity. Have a Baxter robot watch a worker long enough, and the robot can perform the worker’s tasks. Record enough traffic and road data, and a driverless car becomes a reality. Phone trees replace call center workers. Track everything a person does online, and you’ll be able to predict ever higher percentages of what they’ll do next. Filmmakers imagine that, out of the billions of conversations captured by phone companies (or Google, DOD, or DARPA), an OS perfectly imitating a girlfriend will emerge.

The big questions now are: who will have the chance to analyze all that data? Who will raise the capital to build immense computing centers and hire the best workers? (Rumor has it that Google paid one employee $100 million to keep him from moving to Twitter, while colluding with other tech giants to tamp down rank-and-file engineer pay.) Government spending matters here, of course, but the real determinant is global financial markets: how will controllers of capital invest? To make their decisions, they need ever more access to data—or, to gain temporary advantages, they need to obscure what’s really going on. Control of data is vital for them, too.

As happened in prior gold rushes, plutechrats’ race to amass, control, and manipulate data inevitably hurts bystanders—ordinary persons and companies who need loans or investments, or a platform, an audience, apps, etc. Part of the story I tell is how masterful manipulation of the law has allowed tech and finance giants to grow incredibly fast, deflecting the types of regulation that would have created a more level economic playing field. Europe is beginning to catch up to them by applying extant laws. My book shows how the US could, perhaps, catch up to Europe, using tools ranging from consumer protection law to renewed financial regulation.

I argue for much more aggressive policing of the use of data by large firms. They should not be able to fire someone, or deny credit or insurance to her, simply because she is sick (or, more likely in a data-driven age, because some secret algorithm weighted with health-indicating factors has demonstrated her to be “likely to decline in productivity” or “of unstable means”). Nor should they be able to make decisions based on political views of workers or would-be customers. In an era of pervasive surveillance, it’s just too easy for some star chamber to foreclose opportunities.

I also acknowledge how extensions of current laws may not be up to the task of regulating a new, data-driven economy. So I build up a case for public alternatives in many critical areas: public search engines, banking, and credit scores are a start.

L.J.: The book’s analysis critically incorporates, with great acumen and skill, political, economic, and social theories. How, as a lawyer and a legal scholar, did you see yourself contributing to these?

F.P.: I was just at a conference called Rethinking Economics where a critic of the discipline, Philip Mirowski, pointed out that there is no one, standard, ordinary market. Legal rules deeply influence all of them. So rather than simply importing economic (or sociological, or psychological) theories into law, we need to assure that social scientists’ models of reality reflect the legal rules that so deeply shape social structures.

In The Black Box Society, I highlight the centrality of the acceleration of regulatory capture to our current predicament—a near-terminal diagnosis that, nevertheless, also illuminates a way forward. We need to stop speaking as if there is some set of stable corporate interests that “capture” government. Instead, there are warring segments within any given industry (and even within large firms) that do better than others at influencing key regulators—and then build on those regulatory victories to capture even more profits, which in turn underwrite further regulatory victories.

As Piketty showed empirically with respect to wealth, I hope to show historically and theoretically with power: there is no natural “braking mechanism” to stop its accumulation and concentration. That’s how we’ve arrived at the absurd situation where data brokers can invade nearly every aspect of our privacy, but then use trade secrecy law to refuse to tell where they got (or where they sold on) their data. I chronicle bizarre inconsistencies like that throughout the book.

That may sounds like a rather bleak model of our political economy. But it contains the seeds of a constructive response. Because if the most unaccountably powerful firms in our society aren’t simply surfing a historical tidal wave of “skills-biased technological change,” “globalization,” ad nauseam, but are instead critically indebted to law and policy for their power—those can be changed. As Langdon Winner put it in Autonomous Technology, societies “do not yield passively to the ‘thrust’ of modernization. Political and economic actors of the world’s nation-states make conscious decisions about what kinds of technological development to encourage and then carry out these decisions in investments, laws, sanctions, subsidies, and so on.” I think The Black Box Society shows in great detail how laws ranging from copyright to Sarbanes-Oxley helped get us into the fix we’re in—and how changes to them could help get us out.

L.J.: The final citation in the book is to Pope Francis. Any special significance to that?

F.P.: I’ve written on Catholic social thought before, and I still consider it an extraordinary source of person-centered social theory. Francis is a particularly eloquent advocate, critiquing “an impersonal economy lacking a truly human purpose.” We’ve had decades of policy arguing for more “flexible workers,” who can turn on a dime to meet any demand by employers (and who will now be monitored ever more closely to assure compliance). The Pope replies, simply, Why not create economic structures better suited for human flourishing?

So what are the stakes of such an orientation? I recall that Catholic legal scholar Michael Perry’s excellent book, Morality, Politics, and Law, had what seemed at first to be an odd epigraph for the first chapter, by Noam Chomsky:

A vision of future social order is ... based on a concept of human nature. If in fact man is an indefinitely malleable, completely plastic being, with no innate structures of mind and no intrinsic needs of a cultural or social character, then he is a fit subject for the "shaping behavior" by the state authority, the corporate manager, the technocrat, or the central committee. Those with some confidence in the human species . . . will try to determine the intrinsic human characteristics that provide the framework for intellectual development, the growth of moral consciousness, cultural achievement, and participation in a free community.

That may have sounded science fiction-y to many when it was written. But think about the pervasiveness of hidden algorithms, and how much of our experience they shape. One has to wonder if, at some point, those at the top of society will see fit to engineer out of daily experience all manner of inconvenient cultural and social practices. The least we can hope for is some clear understanding of how the computational strategies they deploy affect how we see the world, how we are seen, and how capital is deployed.

The alternative is frightening. Nick Land envisions machinic processes accelerating alongside the human, largely indifferent to the welfare of, say, 99% of us. Your own work describes “technocapital” operating according to its own, perverse internal logic. More prosaically, even the establishment literature on finance bristles with warnings about potentially disastrous systemic risk and short-termism.

I’m sure some will say that the first half of my book paints such a bleak picture of current power structures that the optimistic proposals in its conclusion seem forced. But even an institution with some of the most conservative decision rules imaginable—where past leaders choose the set of persons who choose their successors—has managed to reform and renew itself. (Perhaps not as much as some of us would like—but I try to temper my own sense of disappointment with a keen awareness of my own many shortcomings.)

In Men in Dark Times, Hannah Arendt quotes a “Roman chambermaid” on the pontificate of Pope John XXIII: "Madam," she said to Arendt, “this Pope was a real Christian. How could that be? And how could it happen that a true Christian would sit on St. Peter's chair? Didn't he first have to be appointed Bishop, and Archbishop, and Cardinal, until he finally was elected to be Pope? Had nobody been aware of who he was?" Alienated Catholics have had the same sense of wonder at Francis…perhaps as the downtrodden of the 1930s wondered at Franklin Roosevelt, or in the 1960s, at the leadership of Martin Luther King. “Once in a lifetime / the longed for tidal wave / of justice can rise up,” as Seamus Heaney puts it. It may take a miracle to reform an age ever more regimented by secret technologies and finance—but we can hope.
Posted 8:22 AM by Guest Blogger [link]