Andrew Tutt
For the Unlocking the Black Box conference -- April 2, 2016 at Yale Law School
The views expressed in this essay are the author’s only and
do not necessarily reflect the views of the Department of Justice or the Office
of Legal Counsel. This post is adapted from a working paper on the same
subject.
On February 14, 2016, a Google self-driving car struck a bus on a California
street, apparently the first time one of Google’s self-driving vehicles has
caused an accident. The bus failed to
yield, the car failed to stop, there was a collision. Google’s engineers were
unsure how much responsibility to put at the autonomous vehicle’s feet. The head of Google’s self-driving car project, Chris Urmson,
said that although Google’s car bore some responsibility, fault for the
accident was “not black and white.” In a written statement, Google called the collision
“a classic example of the negotiation that’s a normal part of driving—we’re all
trying to predict each other’s movements.” Google also wrote that “[w]e’ve now reviewed
this incident (and thousands of variations on it) in our simulator in detail
and made refinements to our software. From now on, our cars will more deeply
understand that buses and other large vehicles are less likely to yield to us
than other types of vehicles, and we hope to handle situations like this more
gracefully in the future.”
Google
may have tweaked its algorithm, and run thousands of simulations (and variants
on the simulation)—but how can we be sure that Google’s vehicles are safe? How
many thousands of miles, or tens of thousands of hours, should an autonomous
vehicle algorithm log before it’s road-ready? How do we decide that we are
confident enough that, when an autonomous vehicle algorithm does fail, it won’t
fail catastrophically? The answers to those questions are all still
being worked out.
My
contribution to the Unlocking the Black
Box conference is to suggest that the rise of increasingly complex semi-autonomous algorithms—like those that
power Google’s self-driving cars—calls for developing a new specialist
regulatory agency to regulate algorithmic safety. An FDA for algorithms.
That
might sound strange at first, but hear me out. The algorithms of the future
will be similar to pharmaceutical drugs: The precise mechanisms by which they
produce their benefits and harms will not be well understood, easy to predict,
or easy to explain. They may work wonders, but exactly how they do it will
likely remain opaque. To understand why will require a dive into the future of
algorithms.
The
future of algorithms is algorithms that learn. Such algorithms go by many
names, but the most common are “Machine Learning,” “Predictive Analytics,” and
“Artificial Intelligence.” Basic machine learning algorithms are already
ubiquitous. How does Google guess whether a search query has been misspelled?
Machine learning. How do Amazon and Netflix choose which new products or videos
a customer might want to buy or watch? Machine learning. How does Pandora pick
songs? Machine learning. How do Twitter and Facebook curate their feeds?
Machine learning. How did Obama win reelection in 2012? Machine learning. Even
online dating is guided by machine learning. The list goes on and on.
Machine
learning algorithms are becoming increasingly sophisticated, and at an
accelerating pace. The outputs of machine learning algorithms are sometimes almost indistinguishable from
magic.
Only a few years ago it was thought that problems like accurate speech
recognition, image recognition, machine translation, and self-driving cars were
many years from satisfactory algorithmic solutions. But it is now apparent that
learning algorithms can apply extraordinary processing power to immense
datasets to achieve results that come close to human-level performance. And as
“remarkable” as the growth in machine learning algorithms is, “it’s only a foretaste of what’s to come.” When “algorithms in the lab make it to the front
lines, Bill Gates’s remark that a breakthrough in machine learning would be worth
ten Microsofts will seem conservative.”
A
team of researchers was able to use deep reinforcement learning to create a
single super-algorithm that could be taught to play more than a half-dozen
Atari games using information “it learned from nothing but the video input, the reward and
terminal signals, and the set of possible actions—just as a human player
would.” The trained algorithm surpassed the
performance of previous game-specific AIs on six of the seven games and
exceeded human expert performance on three of them. Video of the expert algorithm playing the Atari games is
astonishing. A graduate student was recently able to use machine learning
to develop a chess AI, “Giraffe,” that after training for “72 hours on a machine with 2x10-core Intel Xeon E5-2660 v2
CPU,” became capable of playing chess “at least comparably to the best expert-designed
counterparts in existence today, many of which have been fine-tuned over the
course of decades.”
Importantly,
because machine learning algorithms do not “think” like humans do, it will soon
become surpassingly complicated to deduce how trained algorithms take what they
have been taught and use it to produce the outcomes that they do. As a
corollary, it will become hard, if not impossible, to know when algorithms will
fail and what will cause them to do so.
Looking
twenty to forty years ahead, a fear of many futurists is that we may
develop an algorithm capable of recursive self-improvement, i.e. producing
learning algorithms more efficient and effective than itself. That development is popularly known in the Artificial
Intelligence community as the “singularity.” A learning algorithm capable of developing
better learning algorithms could rapidly and exponentially improve itself
beyond humanity’s power to comprehend through methods humans could never hope
to understand. But that development is probably a long way off.
Looking
to the more immediate future, the difficulties we confront, as learning
algorithms become more sophisticated, are the problems of “predictability” and
“explainability.” An algorithm’s predictability is a measure of how difficult
its outputs are to predict; its explainability a measure of how difficult its
outputs are to explain. Those problems are familiar to the robotics community,
which has long sought to grapple with the concern that robots might
misinterpret commands by taking them too literally (i.e. instructed to darken a
room, the robot destroys the lightbulbs). Abstract learning algorithms run
headlong into that difficulty. Even if we can fully describe what makes them
work, the actual mechanisms by which they implement their solutions are likely
to remain shrouded: difficult to predict and sometimes difficult to explain.
And as they become more complex and more autonomous, that difficulty will
increase.
What
we know—and what can be known—about how an algorithm works will play crucial
roles in determining whether it is dangerous or discriminatory. Algorithmic
predictability and explainability are hard problems. And they are as much
public policy and public safety problems as technical problems. At the moment,
however, there is no centralized standards-setting body that decides how much
testing should be done, or what other minimum standards machine learning
algorithms should meet, before they are introduced into the broader world. Not
only are the methods by which many algorithms operate non-transparent, many are
trade secrets.
A
federal consumer protection agency for algorithms could contribute to the safe
development of advanced machine learning algorithms in multiple ways. First, it
could help to develop performance standards, design standards, and liability
standards for algorithms. Second, it could engage with diverse stakeholders to
develop methods of ensuring the algorithms are transparent and accountable.
Third, for especially complex algorithms involved in applications that may pose
significant risks to human health and safety—for example when used in
self-driving cars—such an agency could be empowered to prevent the introduction of algorithms onto the
market until their safety and efficacy have been proven through evidence-based
pre-market trials.
Not everyone will
be enthusiastic about an algorithmic safety agency, at least not at first. Not
everyone was enthusiastic about the FDA at first either. But the United States
created the FDA and expanded its regulatory reach after several serious
tragedies revealed its necessity. If we fail to begin thinking critically about
how we are going to grapple with the future of algorithms now, we may see more
than a minor fender-bender before we’re through.
Andrew Tutt is an Attorney-Adviser at the Office of Legal Counsel at U.S. Department of Justice, and was until recently a Visiting Fellow at the Yale Information Society Project.