Quantitative methods and judgment: Competitors or complements?

Quantitative methods and judgment Competitors or complements?

Allan Timmermann

January 12, 2018

Quantitative methods: Strengths

• Flexibility: – Machine learning methods can be used to analyze highly complex and high-dimensional problems • Thousands of predictors, nonlinear terms

– Good progress has been made on controlling “overfitting”

• Avoids subjective biases of human forecasting • Efficiency: Given an objective function, quantitative methods will provide the best fit

Quantitative methods: Limitations

• Quant methods extrapolate past data patterns into the future – Limited library of historical events (LTCM default, Global Financial Crisis, etc.)

• The next crisis will almost certainly be different from previous ones

• What if the future isn’t like the past? Breaks – Are US stocks overvalued now? CAPE suggests so, but interest rates are uniquely low

• How much data do we really have at the business cycle frequency?

Forecasting European inflation using information in the crosssection (Smith and Timmermann, 2017)

Quantitative methods: Limitations

• The “signal-to-noise” ratio is very low in many financial forecasting problems – far less than 1%, so 99% is left unexplained

• Model parameters have to be estimated – Parameter estimation – Model uncertainty

• Models are approximations – “All models are wrong. Some are useful.”

Quantitative methods: Limitations

• Best quant methods use ensemble techniques which combine forecasts from many weak signals and from different models – Many signals are weak – no golden needle in the haystack – Obviously strong signals are already reflected in market prices

Quantitative methods (Limitations)

• Are quantitative methods “too flexible” – Multiple hypothesis testing problem: If many models are tested, some will appear “by luck” to be amazing • Fooled by randomness

– This is not an issue if we can generate large new data sets • Test and fit models on different data sets • In asset allocation analysis we often don’t have the luxury of new data

• Arms race between (over-) fitting and controlling the overfit (familywise error rate)

Controlling for multiple models: White’s Reality Check

Judgmental methods

• Can in principle incorporate forward-looking information and heuristics – Predicting the effect of the Trump tax cut on US GDP growth: no precedent

• Well-known biases • No individual forecaster or judgmental method consistently beats a simple average of judgements • Combination methods work best – Wisdom of the crowd

Combining quantitative and judgmental methods

• For many macroeconomic variables, a simple equalweighted average of survey forecasts beats sophisticated econometric model forecasts – One exception is stock returns

• No method completely dominates the other – Combine the methods using state- and timedependent combination weights – Include judgmental forecasts as an extra signal in a quantitative model – Use quantitative methods to de-bias survey forecasts

Combining quantitative and judgmental methods (cont.)

• Bayesian methods offer a way to combine subjective and quantitative forecasts – Ideally we know their joint probability distribution – Prior on the relative merits of the methods