Jerry Z. Muller on The Tyranny of Metrics

Today, organizations of all kinds are ruled by the belief that the path to success is quantifying human performance, publicizing the results, and dividing up the rewards based on the numbers. But in our zeal to instill the evaluation process with scientific rigor, we’ve gone from measuring performance to fixating on measuring itself. The result is a tyranny of metrics that threatens the quality of our lives and most important institutions. In this timely and powerful book, Jerry Muller uncovers the damage our obsession with metrics is causing—and shows how we can begin to fix the problem. Complete with a checklist of when and how to use metrics, The Tyranny of Metrics is an essential corrective to a rarely questioned trend that increasingly affects us all.

What’s the main idea?

We increasingly live in a culture of metric fixation: the belief in so many organizations that scientific management means replacing judgment based upon experience and talent with standardized measures of performance, and then rewarding or punishing individuals and organizations based upon those measures. The buzzwords of metric fixation are all around us: “metrics,” “accountability,” “assessment,” and “transparency.” Though often characterized as “best practice,” metric fixation is in fact often counterproductive, with costs to individual satisfaction with work, organizational effectiveness, and economic growth.

The Tyranny of Metrics treats metric fixation as the organizational equivalent of The Emperor’s New Clothes. It helps explain why metric fixation has become so popular, why it is so often counterproductive, and why some people have an interest in pushing it. It is a book that analyzes and critiques a dominant fashion in contemporary organizational culture, with an eye to making life in organizations more satisfying and productive.

Can you give a few examples of the “tyranny of metrics?”

Sure. In medicine, you have the phenomenon of “surgical report cards” that purport to show the success rates of surgeons who perform a particular procedure, such as cardiac operations. The scores are publicly reported. In an effort to raise their scores, surgeons were found to avoid operating on patients whose complicated circumstances made a successful operation less likely. So, the surgeons raised their scores. But some cardiac patients who might have benefited from an operation failed to get one—and died as a result. That’s what we call “creaming”—only dealing with cases most likely to be successful.

Then there is the phenomenon of goal diversion. A great deal of K-12 education has been distorted by the emphasis that teachers are forced to place on preparing students for standardized tests of English and math, where the results of the tests influence teacher retention or school closings. Teachers are instructed to focus class time on the elements of the subject that are tested (such as reading short prose passages), while ignoring those elements that are not (such as novels). Subjects that are not tested—including civics, art, and history—receive little attention.

Or, to take an example from the world of business. In 2011 the Wells Fargo bank set high quotas for its employees to sign up customers who were interested in one of its products (say, a deposit account) for additional services, such as overdraft coverage or credit cards. For the bank’s employees, failure to reach the quota meant working additional hours without pay and the threat of termination. The result: to reach their quotas, thousands of bankers resorted to low-level fraud, with disastrous effects for the bank. It was forced to pay a fortune in fines, and its stock price dropped.

Why is the book called The Tyranny of Metrics?

Because it helps explain and articulate the sense of frustration and oppression that people in a wide range of organizations feel at the diversion of their time and energy to performance measurement that is wasteful and counterproductive.

What sort of organizations does the book deal with?

There are chapters devoted to colleges and universities, K-12 education, medicine and health care, business and finance, non-profits and philanthropic organizations, policing, and the military. The goal is not to be definitive about any of these realms, but to explore instances in which metrics of measured performance have been functional or dysfunctional, and then to draw useful generalizations about the use and misuse of metrics.

What sort of a book is it? Does it belong to any particular discipline or political ideology?

It’s a work of synthesis, drawing on a wide range of studies and analyses from psychology, sociology, economics, political science, philosophy, organizational behavior, history, and other fields. But it’s written in jargon-free prose, that doesn’t require prior knowledge of any of these fields. Princeton University Press has it classified under “Business,” “Public Policy,” and “Current Affairs.” That’s accurate enough, but it only begins to suggest the ubiquity of the cultural pattern that the book depicts, analyzes, and critiques. The book makes use of conservative, liberal, Marxist, and anarchist authors—some of whom have surprising areas of analytic convergence.

What’s the geographic scope of the book?

In the first instance, the United States. There is also a lot of attention to Great Britain, which in many respects was at the leading edge of metric fixation in the government’s treatment of higher education (from the “Teaching Quality Assessment” through the “Research Excellence Framework”), health care (the NHS) and policing, under the rubric of “New Public Management.” From the US and Great Britain, metric fixation—often carried by consultants touting “best practice”—has spread to Continental Europe, the Anglosphere, Asia, and especially China (where the quest for measured performance and university rankings is having a particularly pernicious effect on science and higher education).

Is the book simply a manifesto against performance measurement?

By no means. Drawing on a wide range of case studies from education to medicine to the military, the book shows how measured performance can be developed and used in positive ways.

Who do you hope will read the book?

Everyone who works in an organization, manages an organization, or supervises an organization, whether in the for-profit, non-profit, or government sector. Or anyone who wants to understand this dominant organizational culture and its intrinsic weaknesses.

Jerry Z. Muller is the author of many books, including Adam Smith in His Time and Ours and Capitalism and the Jews. His writing has appeared in the New York Times, the Wall Street Journal, the Times Literary Supplement, and Foreign Affairs, among other publications. He is professor of history at the Catholic University of America in Washington, D.C., and lives in Silver Spring, Maryland.

Matthew J. Salganik on Bit by Bit: Social Research in the Digital Age

In just the past several years, we have witnessed the birth and rapid spread of social media, mobile phones, and numerous other digital marvels. In addition to changing how we live, these tools enable us to collect and process data about human behavior on a scale never before imaginable, offering entirely new approaches to core questions about social behavior. Bit by Bit is the key to unlocking these powerful methods—a landmark book that will fundamentally change how the next generation of social scientists and data scientists explores the world around us. Matthew Salganik has provided an invaluable resource for social scientists who want to harness the research potential of big data and a must-read for data scientists interested in applying the lessons of social science to tomorrow’s technologies. Read on to learn more about the ideas in Bit by Bit.

Your book begins with a story about something that happened to you in graduate school. Can you talk a bit about that? How did that lead to the book?

That’s right. My dissertation research was about fads, something that social scientists have been studying for about as long as there have been social scientists. But because I happened to be in the right place at the right time, I had access to an incredibly powerful tool that my predecessors didn’t: the Internet. For my dissertation, rather than doing an experiment in a laboratory on campus—as many of my predecessors might have—we built a website where people could listen to and download new music. This website allowed us to run an experiment that just wasn’t possible in the past. In my book, I talk more about the scientific findings from that experiment, but while it was happening there was a specific moment that changed me and that directly led to this book. One morning, when I came into my basement office, I discovered that overnight about 100 people from Brazil had participated in my experiment. To me, this was completely shocking. At that time, I had friends running traditional lab experiments, and I knew how hard they had to work to have even 10 people participate. However, with my online experiment, 100 people participated while I was sleeping. Doing your research while you are sleeping might sound too good to be true, but it isn’t. Changes in technology—specifically the transition from the analog age to the digital age—mean that we can now collect and analyze social data in new ways. Bit by Bit is about doing social research in these new ways.

Who is this book for?

This book is for social scientists who want to do more data science, data scientists who want to do more social science, and anyone interested in the hybrid of these two fields. I spend time with both social scientists and data scientists, and this book is my attempt to bring the ideas from the communities together in a way that avoids the jargon of either community.  

In your talks, I’ve heard that you compare data science to a urinal.  What’s that about?

Well, I compare data science to a very specific, very special urinal: Fountain by the great French artist Marcel Duchamp. To create Fountain, Duchamp had a flash of creativity where he took something that was created for one purpose—going to the bathroom—and turned it a piece of art. But most artists don’t work that way. For example, Michelangelo, didn’t repurpose. When he wanted to create a statue of David, he didn’t look for a piece of marble that kind of looked like David: he spent three years laboring to create his masterpiece. David is not a readymade; it is a custommade.

These two styles—readymades and custommades—roughly map onto styles that can be employed for social research in the digital age. My book has examples of data scientists cleverly repurposing big data sources that were originally created by companies and governments. In other examples, however, social scientists start with a specific question and then used the tools of the digital age to create the data needed to answer that question. When done well, both of these styles can be incredibly powerful. Therefore, I expect that social research in the digital age will involve both readymades and custommades; it will involve both Duchamps and Michelangelos.

Bit by Bit devotes a lot attention to ethics.  Why?

The book provides many of examples of how researchers can use the capabilities of the digital age to conduct exciting and important research. But, in my experience, researchers who wish to take advantage of these new opportunities will confront difficult ethical decisions. In the digital age, researchers—often in collaboration with companies and governments—have increasing power over the lives of participants. By power, I mean the ability to do things to people without their consent or even awareness. For example, researchers can now observe the behavior of millions of people, and researchers can also enroll millions of people in massive experiments. As the power of researchers is increasing, there has not been an equivalent increase in clarity about how that power should be used. In fact, researchers must decide how to exercise their power based on inconsistent and overlapping rules, laws, and norms. This combination of powerful capabilities and vague guidelines can force even well-meaning researchers to grapple with difficult decisions. In the book, I try to provide principles that can help researchers—whether they are in universities, governments, or companies—balance these issues and move forward in a responsible way.

Your book went through an unusual Open Review process in addition to peer review. Tell me about that.

That’s right. This book is about social research in the digital age, so I also wanted to publish it in a digital age way. As soon as I submitted the book manuscript for peer review, I also posted it online for an Open Review during which anyone in the world could read it and annotate it. During this Open Review process dozens of people left hundreds of annotations, and I combined these annotations with the feedback from peer review to produce a final manuscript. I was really happy with the annotations that I received, and they really helped me improve the book.

The Open Review process also allowed us to collect valuable data. Just as the New York Times is tracking which stories get read and for how long, we could see which parts of the book were being read, how people arrived to the book, and which parts of the book were causing people to stop reading.

Finally, the Open Review process helped us get the ideas in the book in front of the largest possible audience. During Open Review, we had readers from all over the world, and we even had a few course adoptions. Also, in addition to posting the manuscript in English, we machine translated it into more than 100 languages, and we saw that these other languages increased our traffic by about 20%.

Was putting your book through Open Review scary?

No, it was exhilarating. Our back-end analytics allowed me see that people from around the world were reading it, and I loved the feedback that I received. Of course, I didn’t agree with all the annotations, but they were offered in a helpful spirit, and, as I said, many of them really improved the book.

Actually, the thing that is really scary to me is putting out a physical book that can’t be changed anymore. I wanted to get as much feedback as possible before the really scary thing happened.

And now you’ve made it easy for other authors to put their manuscripts through Open Review?

Absolutely. With a grant from the Sloan Foundation, we’ve released the Open Review Toolkit. It is open source software that enables authors and publishers to convert book manuscripts into a website that can be used for Open Review. And, as I said, during Open Review, you can receive valuable feedback to help improve your manuscript, feedback that is very complimentary to the feedback from peer review. During Open Review, you can also collect valuable data to help launch your book. Furthermore, all of these good things are happening at the same time that you are increasing access to scientific research, which is a core value of many authors and academic publishers.

SalganikMatthew J. Salganik is professor of sociology at Princeton University, where he is also affiliated with the Center for Information Technology Policy and the Center for Statistics and Machine Learning. His research has been funded by Microsoft, Facebook, and Google, and has been featured on NPR and in such publications as the New Yorker, the New York Times, and the Wall Street Journal.