Philip E. Tetlock, author of Expert Political Judgment: How Good Is It? How Can We Know?, is heading into Year 3 of the Intelligence Advanced Research Projects Agency’s forecasting tournament. The competition began in fall 2011 to see what team could best predict a series of short-term foreign affairs issues. The competition is based on Tetlock’s book which evaluates expert opinion and ways to make better decisions.
Check out this feature on the New York Times to learn more about the tournament.
By DAVID BROOKS
In 2006, Philip E. Tetlock published a landmark book called “Expert Political Judgment.” While his findings obviously don’t apply to me, Tetlock demonstrated that pundits and experts are terrible at making predictions.
But Tetlock is also interested in how people can get better at making forecasts. His subsequent work helped prompt people at one of the government’s most creative agencies, the Intelligence Advanced Research Projects Agency, to hold a forecasting tournament to see if competition could spur better predictions.
In the fall of 2011, the agency asked a series of short-term questions about foreign affairs, such as whether certain countries will leave the euro, whether North Korea will re-enter arms talks, or whether Vladimir Putin and Dmitri Medvedev would switch jobs. They hired a consulting firm to run an experimental control group against which the competitors could be benchmarked.
Five teams entered the tournament, from places like M.I.T., Michigan and Maryland. Tetlock and his wife, the decision scientist Barbara Mellers, helped form a Penn/Berkeley team, which bested the competition and surpassed the benchmarks by 60 percent in Year 1.
How did they make such accurate predictions? In the first place, they identified better forecasters. It turns out you can give people tests that usefully measure how open-minded they are.
For example, if you spent $1.10 on a baseball glove and a ball, and the glove cost $1 more than the ball, how much did the ball cost? Most people want to say that the glove cost $1 and the ball 10 cents. But some people doubt their original answer and realize the ball actually costs 5 cents.
Tetlock and company gathered 3,000 participants. Some got put into teams with training, some got put into teams without. Some worked alone. Some worked in prediction markets. Some did probabilistic thinking and some did more narrative thinking. The teams with training that engaged in probabilistic thinking performed best. The training involved learning some of the lessons included in Daniel Kahneman’s great work, “Thinking, Fast and Slow.” For example, they were taught to alternate between taking the inside view and the outside view.