Andrew Lo on Adaptive Markets: Financial Evolution at the Speed of Thought

Half of all Americans have money in the stock market, yet economists can’t agree on whether investors and markets are rational and efficient, as modern financial theory assumes, or irrational and inefficient, as behavioral economists believe. In this groundbreaking book, Andrew Lo cuts through this debate with a new framework, the Adaptive Markets Hypothesis, in which rationality and irrationality coexist. Adaptive Markets shows that the theory of market efficiency isn’t wrong but merely incomplete. Lo’s new paradigm explains how financial evolution shapes behavior and markets at the speed of thought. An ambitious new answer to fundamental questions in economics, Adaptive Markets is essential reading for anyone who wants to know how markets really work. We asked him to explain the Adaptive Markets Hypothesis, the strengths and limitations on the current theories, and how this new thinking can be practically applied.

What led you to write this book?

AL: Ever since I was a graduate student in economics, I’ve been struggling with the uncomfortable observation that economic theory doesn’t seem to work in practice. As elegant as this theory is, there are so many examples where the data just don’t support the theory that, after a while, I started wondering just how useful our theories were. For example, stock market prices don’t follow random walks, market prices don’t always seem rational, and people often make poor decisions, especially when it comes to financial matters. But it takes a theory to beat a theory. Rather than just criticizing existing theories, I decided to develop an alternative—this book describes the personal journey I took to arrive at that alternative, which I call the Adaptive Markets Hypothesis.

What’s the Adaptive Markets Hypothesis?

AL: The Adaptive Markets Hypothesis is my solution to the longstanding debate in financial economics between two competing camps. One camp consists of the disciples of the Efficient Markets Hypothesis, who believe that investors are rational decision makers and market prices fully reflect all available information. The opposing camp consists of the psychologists and behavioral economists who believe that investors are irrational and market prices are driven by “animal spirits.” It turns out that both camps have correctly captured certain aspects of human behavior, but neither camp offers a complete picture of how investors and markets behave. The Adaptive Markets Hypothesis fills this gap.


AL: By drawing on recent research in psychology, neuroscience, evolutionary biology, and artificial intelligence, I show that human behavior is the result of several different components of the brain, some of which produce rational behavior while others produce more instinctive emotional behavior. These components often work together, but occasionally they compete with each other. And for obvious evolutionary reasons, rationality can be trumped by emotion and instinct when we’re confronted with extreme circumstances like physical threats—we “freak out.” The problem is that these hardwired responses to physical threats are also triggered by financial threats, and freaking out is generally not the best way to deal with such threats. Therefore, investors and markets have a split personality: sometimes they’re quite rational but every so often, they freak out.

Are you suggesting that the Efficient Markets Hypothesis, which dominates financial thinking today, is wrong?

AL: No! On the contrary, the Efficient Markets Hypothesis is one of the most useful, powerful, and beautiful pieces of economic reasoning that economists have ever proposed. Generations of investors and portfolio managers have been saved from bad investment decisions because of the Efficient Markets Hypothesis, which says that if something seems too good to be true, it probably is. The Efficient Markets Hypothesis is not wrong; it’s merely incomplete. Its focus is the behavior of investors and markets in normal business environments, where the “wisdom of crowds” rules the day. What’s missing is the “madness of mobs,” when investors are reacting emotionally and instinctively in response to extreme business environments—good or bad—leading either to irrational exuberance or panic selling. The Adaptive Markets Hypothesis provides a more complete framework in which both types of behaviors are possible. The combination of these behaviors yields a much richer set of implications for price dynamics, investment strategies, risk management, and financial regulation.

Who is the intended audience for this book?

AL: My intention was to write this book for the general reader, but only time will tell whether or not I’ve succeeded. In fact, I’m hoping that there’s something for everyone in this book. For example, readers wondering whether or not it’s possible to beat the stock market using mathematical models will want to read Chapter 2, “If You’re So Smart, Why Aren’t You Rich?” For readers already convinced that it’s possible and want to understand the neuroscientific basis of irrational behavior, they’ll want to read Chapter 3, “If You’re So Rich, Why Aren’t You Smart?” No book on finance would be complete without a discussion of how the recent financial crisis could have happened to us—a country with one of the most sophisticated financial systems in the world—and that’s Chapter 9, “Fear, Greed, and Financial Crisis.” And for readers interested in getting a glimpse of the future of the financial industry and the amazing things that can be accomplished with finance if used properly, there’s Chapter 12, “To Boldly Go Where No Financier Has Gone Before.” Although the book is based on my academic research, I’ve worked hard to translate “academic-speak” into plain English, using simple analogies and real-life examples to make the research come alive. In fact, there’s not a single equation or mathematical formula in the book, which is no easy feat for someone from MIT!

In Adaptive Markets you take an interdisciplinary view of financial markets, bringing in cognitive neuroscience, biology, computer science, and engineering. How did you come to bring all of these seemingly disparate fields together and why is that important?

AL: Although I do enjoy learning new things and have broad-ranging interests, when I started my academic career as a financial economist, I had no interest or intention in doing “interdisciplinary” research. I was perfectly happy spending my days and nights working on traditional neoclassical financial economics—portfolio theory, derivatives pricing models, asset pricing models, financial econometrics, and so on. But the more I tried to fit financial theories to data, the more frustrated I became that these theories performed so poorly. So I started trying to understand why the theories broke down and how they could be fixed. I began by studying behavioral economics and finance, which led me to psychology, which then to the cognitive neurosciences, and so on. I was dragged—sometimes kicking and screaming—from one field of study to the next in my quest to understand why financial markets don’t work the way we think (and want them to). This process ultimately led me to the Adaptive Markets Hypothesis, which is a very satisfying (for me, at least) integration of various disciplines that have something to say about human behavior. I’m especially pleased by the fact that Adaptive Markets reconciles the two competing schools of thought in financial economics, both of which are compelling in their own right even though they’re incomplete.

Why do we need to understand the evolution of finance?

AL: Many authors and academics will use evolution as a metaphor when referring to the impact of change. In Adaptive Markets, I use evolution quite literally because financial markets and institutions are nothing short of evolutionary adaptations that Homo sapiens has developed to improve our chances of survival. Therefore, if we really want to understand how the financial system works, how it changes over time and circumstances, and what we can do to improve it, we need to understand the evolution of finance. And unlike animal species, which evolve from one generation to the next, the financial system evolves at the speed of thought.

You argue that economics wishes it were more like the hard science of physics where 99% of all observable phenomena can be explained with three laws. Will we ever have a complete understanding of how financial markets function?

AL: It’s true that most economists—myself included—suffer from a psychological disorder called “physics envy.” We wish we could explain 99% of economic behavior with three laws like the physicists but this is a pipe dream. The great physicist Richard Feynman put it best when he said, “Imagine how much harder physics would be if electrons had feelings!” I tell all my students at the start of the semester that all economic theories are approximations to a much more complex reality, so the key question for investors and portfolio managers is not “is the theory correct?” but rather, “how good is the approximation?” The answer to this question lies largely in the environment, which plays a huge role in evolutionary theories. Whether we’ll ever be able to develop a truly complete theory of human behavior—and, therefore, how financial markets function—is hard to say. But I do believe that we can get much closer to that complete theory through the Adaptive Markets Hypothesis.

How can investors and portfolio managers incorporate the Adaptive Markets Hypothesis into their investment philosophies?

AL: The Adaptive Markets Hypothesis has a relatively straightforward but sweeping implication for all investment philosophies, and that has to do with change. During normal business environments, the principles of Efficient Markets are an excellent approximation to reality. For example, from the 1930s to the early 2000s, a period where the U.S. stock market had relatively consistent average returns and volatility, a long-only passive investment strategy of 60% stocks and 40% bonds produced pretty decent returns, particularly for those who were investing over a 10- or 20-year horizon. The problem is that this approach doesn’t always work. When market conditions change and we experience large macro shocks like the financial crisis of 2008, then simple heuristics like 60/40 no longer work as well because financial markets have changed in their dynamics. Today’s markets are now much more responsive to intervention by governments and their central banks and punctuated by the irregular cycle of fear and greed. So since 2007 and 2008, we’ve seen a very different market dynamic than over the previous six decades. The point of Adaptive Markets is not simply to be wedded to any static theory, but rather to understand how the nature of markets can change. And once it does change, we need to change with it. John Maynard Keynes put it best when, in responding to criticism that he flip-flopped on the gold standard, he said, “When the facts change, sir, I change my mind. What do you do?”

Can you give an example of how change might impact today’s investors?

AL: One important implication of Adaptive Markets for investors and portfolio managers is that passive investing is changing and we have to adapt. John Bogle—the founder of the Vanguard Group and the father of passive investing and index funds—had an incredibly important insight in the 1970s which he calls the “Cost Matters Hypothesis:” reducing trading costs can have a huge impact on wealth accumulation. Bogle has done more for the individual investor than anyone else I can think of; he democratized the investment process. Thanks to technological innovations like automated trading, electronic market-making, and big data analytics, we’re ready to take the next evolutionary step that builds on Bogle’s legacy. For example, like the trend in healthcare towards personalized medicine, we can now create personalized indexes that are passive portfolios designed to achieve specific goals for a given individual. You might be more risk tolerant than your neighbor so your portfolio will have more equities, but because you work in the financial industry and she works in big pharma, your personalized portfolio will have fewer financial stocks and hers will have fewer biopharma stocks. Also, personalized indexes can manage the risk more actively to suit an individual’s threshold of “pain.” Current financial wisdom criticizes investors who don’t invest for the long run, and I’ve always thought such criticism to be terribly unfair. After all, how easy is it for someone to stick with an investment that’s lost 50% of its value over just a few months? Well, that’s exactly what happened between the fourth quarter of 2008 and the first quarter of 2009. Traditional investment advice is a bit like trying to prevent teenage pregnancies by asking teenagers to abstain—it’s not bad advice, but it’s unrealistic. Why not manage the risk of an individual’s portfolio more actively so as to reduce the chances of freaking out?

Finance has developed a bad reputation in the popular press, particularly in the aftermath of the recent financial crisis. Does the Adaptive Markets Hypothesis have anything to say about this and how things can be improved?

AL: Absolutely. At the heart of all bad behavior, regardless of the industry or context, is human nature. Humans are the Curious George of the animal kingdom, but there’s no “man in the yellow hat” to bail us out when we get into trouble. Homo sapiens has evolved in some remarkable ways and we’re capable of extraordinary things, both good and bad. The same social and cultural forces that give rise to wonderful organizations like the Peace Corps, the Red Cross, and Doctors without Borders can sometimes lead to much darker and destructive organizations. The only way for us to deal more effectively with the negative aspects of society is to acknowledge this dual nature of human behavior. Chapter 11 of Adaptive Markets, titled “Fixing Finance,” is devoted entirely to this objective. We have to be careful not to throw out the baby with the bathwater—the financial system definitely can be improved, but we shouldn’t vilify this critically important industry because of a few bad actors.

What are some specific proposals for how to fix finance?

AL: Well, before we can fix finance, we need to understand where financial crises come from, and the Adaptive Markets Hypothesis has a clear answer: crises are the product of human behavior coupled with free enterprise. If you can eliminate one or both of these two components, you can eliminate financial crises. Otherwise, financial crises are an avoidable fact of modern life. Human misbehavior is a force of Nature, not unlike hurricanes, flash floods, or earthquakes, and it’s not possible to legislate away these natural disasters. But this doesn’t mean we can do anything about it—we may not be able to prevent hurricanes from occurring, but we can do a great deal to prepare for them and reduce the damage they do. We can do a lot to prepare for financial crises and reduce the damage they do to those individuals and institutions least able to withstand their devastating consequences. This perspective is important because it goes against the traditional narrative that financial crises are caused by a few greedy unscrupulous financiers and once we put them in jail, we’ve taken care of the problem. The Adaptive Markets perspective suggests something different: the problem is us. Specific proposals for dealing with crises include: using new technologies in data science to measure economic activity and construct early warning indicators of impending crises; studying crises systematically like the way the National Transportation Safety Board studies airplane crashes so we know how to make the financial system safer; creating adaptive regulations that change with the environment, becoming more restrictive during booms and less restrictive during busts; and systematically measuring individual behavior and corporate culture quantitatively so we can engage in “behavioral risk management.”

Now that you’ve written this book, where do you see your research going from here?

AL: Well, this is still early days for the Adaptive Markets Hypothesis. There’s so much left to be done in exploring the implications of the theory and testing the implications empirically and experimentally whenever possible. The Efficient Markets Hypothesis took decades and hundreds of academic studies to get established, and the same will be true of this one. One of my goals in writing this book is to motivate my academic and industry colleagues to start this vetting process. In the same way that Darwin’s theory of evolution had to be tested and challenged from many different perspectives, the Adaptive Markets Hypothesis has to go through the gauntlet of academic scrutiny. One important implication of the Adaptive Markets perspective is that we need to change the way we collect data and test theories in financial economics. For example, traditional tests of financial theories involve collecting stock market prices and analyzing the statistical properties of their risks and returns. Contrast this approach with how an ecologist would study a newly discovered tropical island in an effort to preserve it. He would begin by first cataloguing the flora and fauna, identifying the key species, and measuring their biomasses and behaviors. Next, he would determine the food chain, environmental threats, and predator/prey relationships, and then turn to population dynamics in the context of the changing environment. Ultimately, such a process would lead to a much deeper understanding of the entire ecosystem, allowing ecologists to determine the best way to ensure the long-term health and sustainability of that island. Imagine doing the same thing with the financial industry. We would begin by cataloguing the different types of financial institutions and investors, measuring their financial biomass, and identifying key species—banks, hedge funds, pension funds, retail investors, regulators, etc.—and their behaviors. Then we would determine the various types of business relationships and interdependencies among these species, which are critical for mapping the population dynamics of this financial ecosystem. This approach seems sensible enough, but it’s not yet being done today (except by my collaborators and me!).

How do you continue to evolve your own thinking? What do you do?

AL: Someone very wise once said that the beginning of wisdom is humility, and I’m convinced that this is how we make progress as a civilization. Once we’re convinced that we have all the answers, we stop asking new questions and learning. So I’m continually looking for new ways to understand financial market behavior, and constantly humbled by how little I know compared to how much we have yet to discover. In this respect, I guess I’m an intellectual opportunist—I don’t care where an idea comes from or what academic discipline it belongs to; if it gives me new insight into an existing problem, I’ll use it and build on it. I’m currently working on several applications of the Adaptive Markets Hypothesis to investments, risk management, and financial regulation, and also hoping to test the theory in the context of individual and institutional investment decisions. The initial results are quite promising and show that financial industry participants adapt much more quickly than we thought. These results point to several important unintended consequences that have clear implications for how we should regulate the industry so as to reduce the chances of another financial crisis.

Andrew W. Lo is the Charles E. and Susan T. Harris Professor at the MIT Sloan School of LoManagement and director of the MIT Laboratory for Financial Engineering. He is the author of Hedge Funds and Adaptive Markets: Financial Evolution at the Speed of Thought. He is also the founder of AlphaSimplex Group, a quantitative investment management company based in Cambridge, Massachusetts.

What are Wall Street’s smartest people reading? Lasse Pedersen’s EFFICIENTLY INEFFICIENT

Pedersen jacketLasse Pedersen’s new book, Efficiently Inefficient, a look at the key trading strategies used by hedge funds, just made two lists of top investment books. The Wall Street Journal included it in a list of “the books Wall Street’s smartest people think you should read this summer”, where it was recommended by Torsten Slok, ‎chief international economist at Deutsche Bank. also gave the book a shout out, naming it one of the “must read books for serious investors”.

Lasse Pedersen, a finance professor at Copenhagen Business School and New York University’s Stern School of Business, and a principal at AQR Capital Management, is determined to show how markets really work in a world where they are neither perfectly efficient nor completely inefficient. So what exactly does he mean by the contradiction in terms “efficiently inefficient”? From

Imperfectly Efficient

Regarding the book’s title, Pedersen explains: “Markets cannot be perfectly efficient and always reflect all information. If they were perfect, no one would have any incentive to collect information and trade on it, and then how could markets become efficient in the first place? Markets also cannot be so inefficient that making money is very easy because, in that case, hedge funds and other active investors would have an incentive to trade more and more.”

Efficiently Inefficient includes an array of interviews with leading hedge fund managers, including Lee Ainslie, Cliff Asness, Jim Chanos, Ken Griffin, David Harding, John Paulson, Myron Scholes, and George Soros. Free problem sets are available online on Pedersen’s website. The introduction is available for download here.