The Best of Both Worlds: Forecasting US Equity Market Returns Using a Hybrid Machine Learning–Time Series Approach

Abstract

Predicting long-term equity market returns is of great importance for investors to strategically allocate their assets. The authors explore machine learning (ML) methods to forecast 10-year-ahead US stock returns and compare the results with the traditional Shiller regression-based forecasts more commonly used in the asset-management industry. The authors find that ML techniques can only modestly improve the forecast accuracy of a traditional Shiller cyclically adjusted price-to-earnings ratio model, and they actually result in worse performance than the vector autoregressive model (VAR)–based two-step approach. The authors then implement this approach with ML techniques and allow for unspecified nonlinear relationships (a hybrid ML-VAR approach). They find about 50% improvement in real-time forecast accuracy for 10-year annualized US stock returns.

TOPICS: Security analysis and valuation, big data/machine learning, quantitative methods, statistical methods, performance measurement

Key Findings

• ▪ Applying machine learning (ML) techniques within a robust economic framework such as Davis et al.’s (2018) two-step approach is superior than applying such techniques in isolation (directly forecasting equity returns).

• ▪ Using the two-step approach, integrating ML with the vector autoregressive model (ML-VAR) to dynamically forecast earning yields reduces dramatically out-of-sample forecast errors, yielding an improvement of about 50% in forecast accuracy for long-horizon U.S. stock market returns.

• ▪ Among the ML algorithms tested, the ensemble method, which averages all other model forecasts, consistently provides improved predictive power.