Against metrics: how measuring performance by numbers backfires

by Jerry Muller

More and more companies, government agencies, educational institutions and philanthropic organisations are today in the grip of a new phenomenon. I’ve termed it ‘metric fixation’. The key components of metric fixation are the belief that it is possible – and desirable – to replace professional judgment (acquired through personal experience and talent) with numerical indicators of comparative performance based upon standardised data (metrics); and that the best way to motivate people within these organisations is by attaching rewards and penalties to their measured performance. 

The rewards can be monetary, in the form of pay for performance, say, or reputational, in the form of college rankings, hospital ratings, surgical report cards and so on. But the most dramatic negative effect of metric fixation is its propensity to incentivise gaming: that is, encouraging professionals to maximise the metrics in ways that are at odds with the larger purpose of the organisation. If the rate of major crimes in a district becomes the metric according to which police officers are promoted, then some officers will respond by simply not recording crimes or downgrading them from major offences to misdemeanours. Or take the case of surgeons. When the metrics of success and failure are made public – affecting their reputation and income – some surgeons will improve their metric scores by refusing to operate on patients with more complex problems, whose surgical outcomes are more likely to be negative. Who suffers? The patients who don’t get operated upon.

When reward is tied to measured performance, metric fixation invites just this sort of gaming. But metric fixation also leads to a variety of more subtle unintended negative consequences. These include goal displacement, which comes in many varieties: when performance is judged by a few measures, and the stakes are high (keeping one’s job, getting a pay rise or raising the stock price at the time that stock options are vested), people focus on satisfying those measures – often at the expense of other, more important organisational goals that are not measured. The best-known example is ‘teaching to the test’, a widespread phenomenon that has distorted primary and secondary education in the United States since the adoption of the No Child Left Behind Act of 2001.

Short-termism is another negative. Measured performance encourages what the US sociologist Robert K Merton in 1936 called ‘the imperious immediacy of interests … where the actor’s paramount concern with the foreseen immediate consequences excludes consideration of further or other consequences’. In short, advancing short-term goals at the expense of long-range considerations. This problem is endemic to publicly traded corporations that sacrifice long-term research and development, and the development of their staff, to the perceived imperatives of the quarterly report.

To the debit side of the ledger must also be added the transactional costs of metrics: the expenditure of employee time by those tasked with compiling and processing the metrics in the first place – not to mention the time required to actually read them. As the heterodox management consultants Yves Morieux and Peter Tollman note in Six Simple Rules (2014), employees end up working longer and harder at activities that add little to the real productiveness of their organisation, while sapping their enthusiasm. In an attempt to staunch the flow of faulty metrics through gaming, cheating and goal diversion, organisations often institute a cascade of rules, even as complying with them further slows down the institution’s functioning and diminishes its efficiency.

Contrary to commonsense belief, attempts to measure productivity through performance metrics discourage initiative, innovation and risk-taking. The intelligence analysts who ultimately located Osama bin Laden worked on the problem for years. If measured at any point, the productivity of those analysts would have been zero. Month after month, their failure rate was 100 per cent, until they achieved success. From the perspective of the superiors, allowing the analysts to work on the project for years involved a high degree of risk: the investment in time might not pan out. Yet really great achievements often depend on such risks.

The source of the trouble is that when people are judged by performance metrics they are incentivised to do what the metrics measure, and what the metrics measure will be some established goal. But that impedes innovation, which means doing something not yet established, indeed that hasn’t even been tried out. Innovation involves experimentation. And experimentation includes the possibility, perhaps probability, of failure. At the same time, rewarding individuals for measured performance diminishes a sense of common purpose, as well as the social relationships that motivate co-operation and effectiveness. Instead, such rewards promote competition.

Compelling people in an organisation to focus their efforts on a narrow range of measurable features degrades the experience of work. Subject to performance metrics, people are forced to focus on limited goals, imposed by others who might not understand the work that they do. Mental stimulation is dulled when people don’t decide the problems to be solved or how to solve them, and there is no excitement of venturing into the unknown because the unknown is beyond the measureable. The entrepreneurial element of human nature is stifled by metric fixation.

Organisations in thrall to metrics end up motivating those members of staff with greater initiative to move out of the mainstream, where the culture of accountable performance prevails. Teachers move out of public schools to private and charter schools. Engineers move out of large corporations to boutique firms. Enterprising government employees become consultants. There is a healthy element to this, of course. But surely the large-scale organisations of our society are the poorer for driving out staff most likely to innovate and initiate. The more that work becomes a matter of filling in the boxes by which performance is to be measured and rewarded, the more it will repel those who think outside the box.

Economists such as Dale Jorgenson of Harvard University, who specialise in measuring economic productivity, report that in recent years the only increase in total-factor productivity in the US economy has been in the information technology-producing industries. The question that ought to be asked next, then, is to what extent the culture of metrics – with its costs in employee time, morale and initiative, and its promotion of short-termism – has itself contributed to economic stagnation?Aeon counter – do not remove

Jerry Z. Muller is the author of many books, including The Tyranny of Metrics. His writing has appeared in the New York Times, the Wall Street Journal, the Times Literary Supplement, and Foreign Affairs, among other publications. He is professor of history at the Catholic University of America in Washington, D.C., and lives in Silver Spring, Maryland.

 

This article was originally published at Aeon and has been republished under Creative Commons.

Jerry Z. Muller on The Tyranny of Metrics

Today, organizations of all kinds are ruled by the belief that the path to success is quantifying human performance, publicizing the results, and dividing up the rewards based on the numbers. But in our zeal to instill the evaluation process with scientific rigor, we’ve gone from measuring performance to fixating on measuring itself. The result is a tyranny of metrics that threatens the quality of our lives and most important institutions. In this timely and powerful book, Jerry Muller uncovers the damage our obsession with metrics is causing—and shows how we can begin to fix the problem. Complete with a checklist of when and how to use metrics, The Tyranny of Metrics is an essential corrective to a rarely questioned trend that increasingly affects us all.

What’s the main idea?

We increasingly live in a culture of metric fixation: the belief in so many organizations that scientific management means replacing judgment based upon experience and talent with standardized measures of performance, and then rewarding or punishing individuals and organizations based upon those measures. The buzzwords of metric fixation are all around us: “metrics,” “accountability,” “assessment,” and “transparency.” Though often characterized as “best practice,” metric fixation is in fact often counterproductive, with costs to individual satisfaction with work, organizational effectiveness, and economic growth.

The Tyranny of Metrics treats metric fixation as the organizational equivalent of The Emperor’s New Clothes. It helps explain why metric fixation has become so popular, why it is so often counterproductive, and why some people have an interest in pushing it. It is a book that analyzes and critiques a dominant fashion in contemporary organizational culture, with an eye to making life in organizations more satisfying and productive.

Can you give a few examples of the “tyranny of metrics?”

Sure. In medicine, you have the phenomenon of “surgical report cards” that purport to show the success rates of surgeons who perform a particular procedure, such as cardiac operations. The scores are publicly reported. In an effort to raise their scores, surgeons were found to avoid operating on patients whose complicated circumstances made a successful operation less likely. So, the surgeons raised their scores. But some cardiac patients who might have benefited from an operation failed to get one—and died as a result. That’s what we call “creaming”—only dealing with cases most likely to be successful.

Then there is the phenomenon of goal diversion. A great deal of K-12 education has been distorted by the emphasis that teachers are forced to place on preparing students for standardized tests of English and math, where the results of the tests influence teacher retention or school closings. Teachers are instructed to focus class time on the elements of the subject that are tested (such as reading short prose passages), while ignoring those elements that are not (such as novels). Subjects that are not tested—including civics, art, and history—receive little attention.

Or, to take an example from the world of business. In 2011 the Wells Fargo bank set high quotas for its employees to sign up customers who were interested in one of its products (say, a deposit account) for additional services, such as overdraft coverage or credit cards. For the bank’s employees, failure to reach the quota meant working additional hours without pay and the threat of termination. The result: to reach their quotas, thousands of bankers resorted to low-level fraud, with disastrous effects for the bank. It was forced to pay a fortune in fines, and its stock price dropped.

Why is the book called The Tyranny of Metrics?

Because it helps explain and articulate the sense of frustration and oppression that people in a wide range of organizations feel at the diversion of their time and energy to performance measurement that is wasteful and counterproductive.

What sort of organizations does the book deal with?

There are chapters devoted to colleges and universities, K-12 education, medicine and health care, business and finance, non-profits and philanthropic organizations, policing, and the military. The goal is not to be definitive about any of these realms, but to explore instances in which metrics of measured performance have been functional or dysfunctional, and then to draw useful generalizations about the use and misuse of metrics.

What sort of a book is it? Does it belong to any particular discipline or political ideology?

It’s a work of synthesis, drawing on a wide range of studies and analyses from psychology, sociology, economics, political science, philosophy, organizational behavior, history, and other fields. But it’s written in jargon-free prose, that doesn’t require prior knowledge of any of these fields. Princeton University Press has it classified under “Business,” “Public Policy,” and “Current Affairs.” That’s accurate enough, but it only begins to suggest the ubiquity of the cultural pattern that the book depicts, analyzes, and critiques. The book makes use of conservative, liberal, Marxist, and anarchist authors—some of whom have surprising areas of analytic convergence.

What’s the geographic scope of the book?

In the first instance, the United States. There is also a lot of attention to Great Britain, which in many respects was at the leading edge of metric fixation in the government’s treatment of higher education (from the “Teaching Quality Assessment” through the “Research Excellence Framework”), health care (the NHS) and policing, under the rubric of “New Public Management.” From the US and Great Britain, metric fixation—often carried by consultants touting “best practice”—has spread to Continental Europe, the Anglosphere, Asia, and especially China (where the quest for measured performance and university rankings is having a particularly pernicious effect on science and higher education).

Is the book simply a manifesto against performance measurement?

By no means. Drawing on a wide range of case studies from education to medicine to the military, the book shows how measured performance can be developed and used in positive ways.

Who do you hope will read the book?

Everyone who works in an organization, manages an organization, or supervises an organization, whether in the for-profit, non-profit, or government sector. Or anyone who wants to understand this dominant organizational culture and its intrinsic weaknesses.

Jerry Z. Muller is the author of many books, including Adam Smith in His Time and Ours and Capitalism and the Jews. His writing has appeared in the New York Times, the Wall Street Journal, the Times Literary Supplement, and Foreign Affairs, among other publications. He is professor of history at the Catholic University of America in Washington, D.C., and lives in Silver Spring, Maryland.