submitting forecasts to the Fed’s Survey of Professional Forecasters, are underconfident in forecasting inflation rates — perhaps because inflation is so variable — and so exterior trimming of probabilities is necessary to hone the forecast. “You get rid of the extremes, which then might reflect better the view of the group, rather than let the view of the group be pulled in a direction by the guy who has the wildest view,” Lichtendahl says. The opposite problem occurs when forecasters take a shot at predicting GDP growth, for example. They tend to be overconfident, are too narrow in their forecasts and could miss some important event. “Those people in the middle are saying the same thing over and over,” says Lichtendahl. “Our trimming method will help you balance overconfidence and underconfidence,” says Grushka-Cockayne. They’ve extended this method into the machine learning environment. Machine learning technology has become faster in handling big data. Machines can spit out hundreds of model predictions — think of each as an expert’s opinion — also known as weak learners. “Each is weak because it’s narrow,” says Grushka-Cockayne. “But when we average across them … we improve by trimming the probability forecast.” How big is the improvement? The professors have tested their method in theory. They found that, for example, in the case of predicting diamond prices, they improved the forecast by 3.3 percent. In predicting GDP growth, the improvement came in at a whopping 18.5 percent. “This is an evolving thing,” says Lichtendahl. “We want to start using it in the real world to see how much economic gain there is. There are statistical improvements that we can demonstrate, but economic gains are going to come from in-use experience.” Grushka-Cockayne says that other researchers “have tried to find better ways to combine. It’s a known problem and one that is interesting to many people. What I like about trimming is it’s very straightforward to execute.” The two professors are currently writing a trimmed opinion pool algorithm for R — an opensource software program for statistical computation
THIS IS AN EVOLVING THING. WE WANT TO START USING IT IN THE REAL WORLD TO SEE HOW MUCH ECONOMIC GAIN THERE IS. THERE ARE STATISTICAL IMPROVEMENTS THAT WE CAN DEMONSTRATE, BUT ECONOMIC GAINS ARE GOING TO COME FROM IN-USE EXPERIENCE. and graphics — “so that somebody can take a data set into this environment and very quickly get out a trimmed probability forecast,” says Lichtendahl. The next step is to tell the business world, which is still loyal to point forecasting, about their method. “Now that we have results and have already coded the method completely in the software package, and we know how to optimize it — how to make it work beyond the theoretical mathematics — we’re basically broadcasting it,” says Grushka-Cockayne.”We’re trying to get more people aware of it.” “This is a long journey,” says Lichtendahl. “It’s going to take us many years.”
Yael Grushka-Cockayne, assistant professor of business administration at Darden, and Kenneth C. Lichtendahl Jr., associate professor of business administration at Darden, co-authored with Victor Richmond R. Jose the paper “Trimmed Opinion Pools and the Crowd’s Calibration Problem,” published in Management Science (Volume 60, Issue 2, February 2014).
Faculty Research from the University of Virginia Darden School of Business Spring 2014