Balkinization: An FDA For Algorithms

An FDA For Algorithms

Guest Blogger

Andrew Tutt

For the Unlocking the Black Box conference -- April 2, 2016 at Yale Law School

The views expressed in this essay are the author’s only and do not necessarily reflect the views of the Department of Justice or the Office of Legal Counsel. This post is adapted from a working paper on the same subject.

On February 14, 2016, a Google self-driving car struck a bus on a California street, apparently the first time one of Google’s self-driving vehicles has caused an accident. The bus failed to yield, the car failed to stop, there was a collision. Google’s engineers were unsure how much responsibility to put at the autonomous vehicle’s feet. The head of Google’s self-driving car project, Chris Urmson, said that although Google’s car bore some responsibility, fault for the accident was “not black and white.” In a written statement, Google called the collision “a classic example of the negotiation that’s a normal part of driving—we’re all trying to predict each other’s movements.” Google also wrote that “[w]e’ve now reviewed this incident (and thousands of variations on it) in our simulator in detail and made refinements to our software. From now on, our cars will more deeply understand that buses and other large vehicles are less likely to yield to us than other types of vehicles, and we hope to handle situations like this more gracefully in the future.”

Google may have tweaked its algorithm, and run thousands of simulations (and variants on the simulation)—but how can we be sure that Google’s vehicles are safe? How many thousands of miles, or tens of thousands of hours, should an autonomous vehicle algorithm log before it’s road-ready? How do we decide that we are confident enough that, when an autonomous vehicle algorithm does fail, it won’t fail catastrophically? The answers to those questions are all still being worked out.

My contribution to the Unlocking the Black Box conference is to suggest that the rise of increasingly complex semi-autonomous algorithms—like those that power Google’s self-driving cars—calls for developing a new specialist regulatory agency to regulate algorithmic safety. An FDA for algorithms.

That might sound strange at first, but hear me out. The algorithms of the future will be similar to pharmaceutical drugs: The precise mechanisms by which they produce their benefits and harms will not be well understood, easy to predict, or easy to explain. They may work wonders, but exactly how they do it will likely remain opaque. To understand why will require a dive into the future of algorithms.

The future of algorithms is algorithms that learn. Such algorithms go by many names, but the most common are “Machine Learning,” “Predictive Analytics,” and “Artificial Intelligence.” Basic machine learning algorithms are already ubiquitous. How does Google guess whether a search query has been misspelled? Machine learning. How do Amazon and Netflix choose which new products or videos a customer might want to buy or watch? Machine learning. How does Pandora pick songs? Machine learning. How do Twitter and Facebook curate their feeds? Machine learning. How did Obama win reelection in 2012? Machine learning. Even online dating is guided by machine learning. The list goes on and on.

Machine learning algorithms are becoming increasingly sophisticated, and at an accelerating pace. The outputs of machine learning algorithms are sometimes almost indistinguishable from magic. Only a few years ago it was thought that problems like accurate speech recognition, image recognition, machine translation, and self-driving cars were many years from satisfactory algorithmic solutions. But it is now apparent that learning algorithms can apply extraordinary processing power to immense datasets to achieve results that come close to human-level performance. And as “remarkable” as the growth in machine learning algorithms is, “it’s only a foretaste of what’s to come.” When “algorithms in the lab make it to the front lines, Bill Gates’s remark that a breakthrough in machine learning would be worth ten Microsofts will seem conservative.”

A team of researchers was able to use deep reinforcement learning to create a single super-algorithm that could be taught to play more than a half-dozen Atari games using information “it learned from nothing but the video input, the reward and terminal signals, and the set of possible actions—just as a human player would.” The trained algorithm surpassed the performance of previous game-specific AIs on six of the seven games and exceeded human expert performance on three of them. Video of the expert algorithm playing the Atari games is astonishing. A graduate student was recently able to use machine learning to develop a chess AI, “Giraffe,” that after training for “72 hours on a machine with 2x10-core Intel Xeon E5-2660 v2 CPU,” became capable of playing chess “at least comparably to the best expert-designed counterparts in existence today, many of which have been fine-tuned over the course of decades.”

Importantly, because machine learning algorithms do not “think” like humans do, it will soon become surpassingly complicated to deduce how trained algorithms take what they have been taught and use it to produce the outcomes that they do. As a corollary, it will become hard, if not impossible, to know when algorithms will fail and what will cause them to do so.

Looking twenty to forty years ahead, a fear of many futurists is that we may develop an algorithm capable of recursive self-improvement, i.e. producing learning algorithms more efficient and effective than itself. That development is popularly known in the Artificial Intelligence community as the “singularity.” A learning algorithm capable of developing better learning algorithms could rapidly and exponentially improve itself beyond humanity’s power to comprehend through methods humans could never hope to understand. But that development is probably a long way off.

Looking to the more immediate future, the difficulties we confront, as learning algorithms become more sophisticated, are the problems of “predictability” and “explainability.” An algorithm’s predictability is a measure of how difficult its outputs are to predict; its explainability a measure of how difficult its outputs are to explain. Those problems are familiar to the robotics community, which has long sought to grapple with the concern that robots might misinterpret commands by taking them too literally (i.e. instructed to darken a room, the robot destroys the lightbulbs). Abstract learning algorithms run headlong into that difficulty. Even if we can fully describe what makes them work, the actual mechanisms by which they implement their solutions are likely to remain shrouded: difficult to predict and sometimes difficult to explain. And as they become more complex and more autonomous, that difficulty will increase.

What we know—and what can be known—about how an algorithm works will play crucial roles in determining whether it is dangerous or discriminatory. Algorithmic predictability and explainability are hard problems. And they are as much public policy and public safety problems as technical problems. At the moment, however, there is no centralized standards-setting body that decides how much testing should be done, or what other minimum standards machine learning algorithms should meet, before they are introduced into the broader world. Not only are the methods by which many algorithms operate non-transparent, many are trade secrets.

A federal consumer protection agency for algorithms could contribute to the safe development of advanced machine learning algorithms in multiple ways. First, it could help to develop performance standards, design standards, and liability standards for algorithms. Second, it could engage with diverse stakeholders to develop methods of ensuring the algorithms are transparent and accountable. Third, for especially complex algorithms involved in applications that may pose significant risks to human health and safety—for example when used in self-driving cars—such an agency could be empowered to prevent the introduction of algorithms onto the market until their safety and efficacy have been proven through evidence-based pre-market trials.

Not everyone will be enthusiastic about an algorithmic safety agency, at least not at first. Not everyone was enthusiastic about the FDA at first either. But the United States created the FDA and expanded its regulatory reach after several serious tragedies revealed its necessity. If we fail to begin thinking critically about how we are going to grapple with the future of algorithms now, we may see more than a minor fender-bender before we’re through.

Andrew Tutt is an Attorney-Adviser at the Office of Legal Counsel at U.S. Department of Justice, and was until recently a Visiting Fellow at the Yale Information Society Project.

Labels: Black Box

Posted 1:36 PM by Guest Blogger [link]