Modular Machine Learning for Model Validation: An Application to the Fundamental Law of Active Management

Abstract

The author introduces a modular machine learning framework for model validation in which the output from one procedure serves as the input to another procedure within a single validation framework. A defining feature of the described methodology is the use of both traditional econometrics and data science. The author uses an econometric model in the first module to classify data in an economically intuitive way. Proceeding modules apply data science techniques to evaluate the predictive characteristics of the model components. The author applies his framework to the fundamental law of active management, a well-known formal characterization of portfolio managers’ alpha generation process. In contrast to standard applications of the law, in which it has been used to evaluate a manager’s existing active management process, the author recasts the law within his framework as a means to test investment signals for potential use, individually or collectively, in a manager’s investment process. To illustrate how this application works, the author provides an example using the well-known Fama–French factors as test signals.

TOPICS: Statistical methods, simulations, big data/machine learning

Key Findings

• • Implementing model validation through a set of interdependent modules that uses both traditional econometrics and data science techniques can produce robust assessments of the predictive effectiveness of investment signals in an economically intuitive manner.

• • The proposed methodology, modular machine learning, also answers several practical questions that arise when applying block time series cross-validation, such as the length of the folds and the block size to use between folds.

• • It is possible to reinterpret the fundamental law of active management into a model validation framework by expressing its fundamental concepts—information coefficient and breadth—using the formal language of data science.