THE ART & SCIENCE OF FORECASTING
The Superforecasters How a college-based team of students, faculty and volunteers bested government intelligence community prognosticators and gave birth to Superforecasting BY ED MCKINLEY
A
husband-and-wife team of scientists became giants in the field of forecasting by creating and overseeing the Good Judgment Project, a massive and lengthy academic inquiry into how to train predictors and improve their predictions. In the process, they coined and popularized the term “Superforecasting” and spawned a company that provides forecasts for businesses and governments. “The story started in the summer of 2009,” recalls Barbara Mellers, one half of the research duo. “My husband, Phil Tetlock, and I decided to apply for a contract with IARPA.” That’s the government’s Intelligence Advanced Research Projects Activity, an entity she describes as “the little sister to DARPA,” the much larger Defense Advanced Research Projects Agency.
GETTING PREDICTIONS RIGHT The U.S. intelligence community failed to foresee the 9-11 attacks and erroneously predicted that Iraq possessed weapons of mass destruction. The time had come to find better ways of using the wisdom of the crowd for forecasting. That’s when intel officials hit upon the idea of staging a tournament to challenge academic minds to develop better ways to foretell the future. One of the competing teams—a University of Pennsylvania group called the Good Judgment Project—rose to the challenge and may have changed history. Under the leadership of Profs. Barbara Mellers and Phil Tetlock the Penn group discovered and then pursued Superforecasting.
22
Barbara Mellers
“We put this grant together as a chance to work together on something that interested both of us,” notes Mellers. She and her husband both teach psychology, and they’re both cross-appointed at the University of Pennsylvania to the School of Arts and Sciences and The Wharton School. They were seeking funds that IARPA awards to projects of interest to the intelligence community. Unlike DARPA, which tends to keep its research classified, IARPA shares some of its findings with the public, which fits with the mission of academia. But a slight misalignment arose when IARPA took three or four months to notify Mellers and Tetlock that their application for a project had been accepted. The delay made things a little “awkward” as Mellers remembers it. By then, She and Tetlock were leaving the University of California at Berkeley to assume their current
Phil Tetlock
positions at Penn. Some of the funding had to filter through UC Berkeley, so the project initially included five or 10 students and faculty members from that school under the direction of Prof. Don Moore. Mellers and Tetlock recruited 20 or 25 students and faculty from Penn, as she tells it. Contingents from Rice University, Hebrew University and other institutions later joined the project. Tournament of forecasting They were working on what IARPA calls an ACE tournament, which stands for Aggregative Contingent Estimation. Think of it as five university-led teams competing to come up with the best probabilistic geopolitical forecasts from a large, dispersed crowd of volunteers. The project got underway in 2011 and continued until 2015. After the second year, IARPA eliminated funding for all of the university
luckbox | december 2020 / january 2021
2012-topics-mellers.indd 22
11/24/20 10:49 AM
