# CV-TMLE and double machine learning

This post is part of our Q&A series.

Question:

@mark_vdlaan Is there an applied researcher’s guide to choosing between double machine learning and TMLE + cross-fitting? PS: Thanks for making these methods and resources so easily accessible!

Thanks for this interesting question. In the past several years, the interest in these machine learning-based estimators has become more widespread, since they allow for the statistical answer to a question to be framed in terms of scientifically meaningful parameters (e.g., defined through causal inference), incorporating machine learning in the estimation process, while providing formal statistical inference.

In order to provide a comprehensive answer, it is suitable to first define the different classes of asymptotically efficient estimators. To start with, to refer to a class of estimators as double machine learning (DML) estimators suggests that this class only applies to a particular subclass of estimation problems that allow for double robustness, or, involves machine learning of multiple nuisance functions. Here, we are reminded that this robustness refers to preservation of the consistency of the estimator under inconsistent estimation of one or more of its nuisance parameters. Different estimation problems have different structures, ranging from no form of robustness (i.e., consistency of the estimator under misspecification of a nuisance function), to double robustness, triple robustness, an array of configurations of robustness, to complete robustness. Therefore, (wrongly) terming an estimator in terms of a very specific form of robustness suggests that it is a very limited procedure fully relying on this very specific type of estimation problem. In addition, in many estimation problems, one only has to apply machine learning to a single functional parameter.

Efficient estimators are generally defined and will be double robust if the estimation problem falls in that category, and, in general, can be tailored to be, or is naturally, as robust as the estimation problem allows. So, it is more sensible to discuss different classes of efficient estimators – as we will see, the double machine learning approach falls in the category of estimating equation-based estimators. Efficient estimators rely on estimating nuisance parameters (functionals of the data distribution), and, if the statistical model for the data distribution is large, this would naturally require machine learning (also called data adaptive estimation). Thus, any of these efficient estimators could be called machine learning-based estimators, and, in fact, they have been applied using machine learning from very early on (circa 1980) in the literature.

We will only consider target parameters (which we will denote with $\psi$) that are so-called pathwise differentiable, making them potentially estimable at $\sqrt{n}$-rate. For simplicity, let’s focus on the case that we observe $n$ independent identically distributed copies of a random variable $O$ with true data distribution $P_0$. Note that all we really need is local asymptotic normality of the log-likelihood; therefore, this also applies to time series and other dependent data structures (e.g., see Efficient and Adaptive Estimation for Semiparametric Models by Bickel, Klaassen, Ritov, and Wellner). So, what does it mean to state that an estimator of a pathwise differentiable target parameter is asymptotically efficient at a given data distribution? An estimator is asymptotically efficient if it can be approximated in first order as an empirical mean of the so-called canonical gradient of the pathwise derivative of the target parameter. To determine the canonical gradient, one views the estimand as a mapping from all possible data distributions in the statistical model, say $\mathcal{M}$, to the parameter space (typically, the real line $\mathbb{R}$) and determine its pathwise derivative along a rich collection of paths through the data distribution – this pathwise derivative is uniquely characterized by its canonical gradient. Owing to its use in the definition of efficient estimators, the canonical gradient at a data distribution is also called the efficient influence curve ($\text{EIC}$ or $\text{EIC}(P)$ at a particular data distribution $P$). Recall that an estimator is asymptotically linear if it can be approximated by an empirical mean of a particular function of the unit data structure $O$; the influence curve is this same particular function. Importantly, an efficient estimator is asymptotically the best (i.e., most efficient) among the class of all regular asymptotically linear estimators – or, in fact, among all regular estimators. What’s more, an efficient estimator is asymptotically normally distributed with asymptotic variance equal to the variance of the efficient influence curve.

Given a statistical model $\mathcal{M}$ (i.e., a set of possible probability distributions of the unit-level data $O$) and that the target parameter $\psi$ is a mapping from the statistical model to the parameter space (e.g., the real line $\mathbb{R}$), the statistical estimation problem is now fully defined, and one can proceed to compute the canonical gradient of the pathwise derivative of the target parameter. Perhaps unsurprisingly, the construction of an efficient estimator necessarily involves using the canonical gradient. The three classes of such efficient estimators are composed of

1. the one-step estimator (OSE);
2. the estimating equation estimator (EEE); and
3. the targeted minimum loss estimator (TMLE).

The asymptotic efficiency of these estimators relies on a second order term to be negligible, thereby typically requiring highly adaptive estimators of the nuisance functions in the canonical gradient (those that will achieve the convergence rate of $n^{\frac{1}{4}}$). This second order term is defined in terms of the canonical gradient (i.e., $R_2(P, P_0) = \Psi(P) - \Psi(P_0) + \mathbb{E}_{P_0} \text{EIC}(P)$ in my work). Additionally, when using variants of these estimators that do not make use sample splitting, one also then relies on a Donsker class condition (from empirical process theory), while, if one uses the sample splitting variants, then this Donsker class condition is not needed. It is the form that this second order remainder $R_2(P,P_0)$ takes that determines the robustness structure of the estimation problem – for example, in double robust problems, the term typically involves products of differences between nuisance function under $P$ and true nuisance function (i.e., under $P_0$).

Recall that the classical way to obtain an efficient estimator is via maximum likelihood estimation. Unfortunately, when the statistical model is large, the the maximum likelihood estimator (MLE) is generally not defined. To work around this issue, one might consider regularizing the MLE by introducing a tuning parameter such as a sieve-based MLE, involving data adaptive selection of a submodel among a collection of submodels that approximate the actual statistical model. If this tuning parameter in the regularized MLE is optimized w.r.t. the density itself, the resulting regularized MLE (a plug-in estimator) of the target parameter will still fail to be asymptotically linear. Despite this, there remain interesting classes of regularized MLEs in which an undersmoothed choice of tuning parameter will result in the construction of an efficient estimator. In recent work, we established this result for the so-called Highly Adaptive Lasso (HAL) MLE (minimum loss estimation), in which one computes an MLE over a class of cadlag functions with a universal bound on the sectional variation norm, selecting the sectional variation norm with an undersmoothing-based selector. In any case, such regularized MLEs are not targeted towards a particular estimand and could therefore be considered less powerful than one of the efficient estimators that utilize the canonical gradient for a particular estimand. There is more to say about this, including great work in the literature on this topic by Shen and Newey, among others, but that lies beyond the scope of this comment.

We mention the HAL-MLE above since it is very relevant for efficient estimation. Just recently, Bibaut and vdL (2019) showed that the HAL-MLE converges to the true functional parameter at a rate as fast as $n^{-\frac{1}{3}}$ up to a $\log(n)$ factor. Thus, using this HAL-MLE to estimate the nuisance functions in the OSE, EEE, or TMLE would guarantee that these estimators are indeed asymptotically efficient. In addition, the HAL-MLE automatically satisfies the Donsker class condition as well, implying that the non-sample split variants of the OSE, EEE, and TMLE using will be asymptotically efficient when constructed via the HAL-MLE; the only assumption embedded in this approach is that the true nuisance functions are cadlag and have finite sectional variation norm (with the $\text{EIC}$ being a bounded function).

With this background, we are now well-positioned to proceed with discussing the three classes of efficient estimators (OSE, EEE, and TMLE). Along the way, we’ll provide some historical context and discuss how DML fits into this landscape.

Historically, the first efficient estimator was the so called one-step estimator using sample splitting (the work of Levin, Pfanzagl, and Klaassen circa 1970-1980, see references in the book by Bickel, Klaassen, Ritove, and Wellner). For example, Klaassen (1986) showed that the one-step estimator using sample splitting is efficient under minimal conditions. The one-step estimator (and its non-sample splitting version) is the efficient estimator presented in the comprehensive book on efficient estimation in semiparametric models (Efficient and Adaptive Estimation for Semiparametric Models) by Bickel, Klaassen, Ritov, and Wellner (1997). This one-step estimator is defined as a plug-in initial estimator of the target parameter, plus the empirical mean of the EIC at this same initial estimator. The sample splitting variant of this estimator is defined as the average across a number of sample splits in training and validation sample (say, V-fold), of the initial plug-in estimator based on a training sample plus the empirical mean over the validation sample of the EIC at this same initial plug-in estimator (fitted on the training sample). A little later (circa 1980 and onward), as empirical process theory emerged as a field, the sample splitting based one-step estimator was often replaced by the “regular” (non-sample slitting) one-step estimator that relied on the Donsker class condition (i.e., the sample splitting will come at a small price if the Donsker class condition nicely holds, but one should make sure that the initial estimator is not overfitted).

In the 1990s, the corresponding approach of estimating equation-based estimators (EEE) was rigorously developed by Robins and collaborators, in which one constructs efficient estimators as solutions of the EIC estimating equation, – i.e., one sets empirical mean of $\text{EIC}(\psi, \eta)$ (for some nuisance parameter vector $\eta$) equal to zero – by estimating the nuisance function $\eta$ in the EIC with an initial estimator and solving for the target parameter $\psi$. This relies on the EIC admitting an estimating function representation, i.e., having $\text{EIC}(P)$ depend on the data distribution $P$ through the target parameter $\Psi(P)$ and a nuisance parameter $\eta$. If such a representation of $\text{EIC}(P) = \text{EIC}(\psi, \eta)$ exists, the one-step estimator is equivalent to the first step of the Newton-Raphson algorithm for solving the estimating equation. Fundamental work on EEE was done by Robins and Rontitzky (circa 1992 and onward), and collaborators, in the context of censored data and causal inference models, involving clever representations and derivations of the EIC (e.g., the augmented IPCW representation of the EIC). A comprehensive review and treatment of the general efficient estimating equation methodology – going beyond application to censored data and causal inference models, including its theory – is presented in the book Unified Methods for Censored Longitudinal Data and Causality by van der Laan and Robins (2001).

In contrast with EEE, the OSE is always well-defined while the EEE requires

1. the estimating function representation;
2. that a solution of the EIC estimating equation exists; and
3. if multiple solutions exists that we know how to select among them.

On the other hand, asymptotics of the OSE rely on the initial estimator, making it potentially less robust than EEE. For example, if the estimation problem is doubly robust (DR), but the initial estimator is not DR, then the OSE is generally not DR either, even though the EEE will be DR in such a case. Completely analogously to the OSE, one can use a sample splitting analogue of the EEE by defining the estimating equation as an average across the sample splits of the empirical mean over the validation sample of the EIC, in which nuisance parameters are estimated based on the training sample. In this manner, the first step of the Newton-Raphson algorithm for solving this sample splitting-based estimating equation is equivalent to the one-step estimator based on sample splitting. In the book by van der Laan and Robins (2001), focus was placed on the non-sample splitting-based EEE; thus, our theorems throughout that book (as well as the more general theorems in chapter 2) rely on a Donsker class condition. Specifically, we showed that if one were to use machine learning-based estimators whose realizations are cadlag functions with finite sectional variation norm, then this Donsker class condition holds, allowing for highly flexible machine learning (including the HAL-MLE) to be used.

Here, it is important to note that any gradient of the pathwise derivative of the target parameter can be used to construct an asymptotically linear estimator with influence curve equal to that gradient; moreover, by using the unique canonical gradient, the constructed estimator will be asymptotically efficient. Indeed, in Robins and Rotnitzky’s work, they always refer to the class of all estimating functions orthogonal to the nuisance tangent space, which is equivalent with the class of all gradients of the pathwise derivative of the target parameter. Robins et al. show that the class of gradients can be computed by determining the class of all functions of the unit-level data that are orthogonal (in terms of the covariance operator) to all nuisance scores (i.e., scores of paths for which the pathwise derivative of the target parameter equals zero). In problems in which the EIC is difficult to compute, one might decide to settle for an inefficient OSE or EEE implied by an easier-to-compute gradient (see the book by van der Laan and Robins, 2001). In an estimation problem with the double robustness structure, any of such inefficient OSE or EEE based on an inefficient gradient will still be double robust (i.e., the robustness implied by the second order remainder $R_2(P, P_0)$ applies to any gradient, since what matters is that it is orthogonal to the nuisance scores, not that it has minimal variance).

A problem with both EEE and OSE (regardless of whether sample splitting is used) is that these estimators suffer from the fact that they are not plug-in estimators (i.e., one applies the target parameter mapping to an estimated data distribution in the model); thus, they might not satisfy crucial global constraints in the statistical model. For example, if the canonical gradient is highly variable or if the sample size is small, it can easily happen that the OSE and EEE end up estimating a probability with a number larger than one or smaller than zero, thus not respecting the bounds on the parameter (i.e., that probabilities lie between zero and one, inclusive). Moreover, by potentially failing to respect these important global statistical constraints in the model, the OSE and EEE lack robustness (e.g., just one observation could throw off the estimator badly), in stark contrast to an MLE, which always necessarily fully respects global constraints of the model. Of course, global constraints are asymptotically irrelevant, yet are very important in finite samples.

By contrast, TMLE produces an efficient plug-in estimator, performing updates in the model space so as to avoid any risk of violating bounds on the density of the data and on the parameter space. Specifically, the targeted maximum likelihood estimator takes an initial estimator of the density of the data, constructs a parametric model through this initial estimator with score spanning the canonical gradient at the initial estimator, and fits the unknown parameter(s) in this so-called least favorable parametric model (LFM) via MLE. The resultant targeted density estimator is then plugged in to the target parameter mapping to obtain the TMLE of the target parameter. The LFM through the initial estimator can often be chosen to be a standard parametric model treating as offset the initial estimator. For example, the targeting step in the TMLE of the ATE can be carried out with a logistic regression using a clever covariate, using as offset the initial estimator. In more recent work (vdL and Gruber, 2015), we propose a so-called universal least favorable submodel, which guarantees both that the targeting step is maximally robust and that the TMLE exactly solves the EIC estimating equation in a single step. By contrast, the originally proposed local LFM will solve the EIC equation up to an approximation error that is asymptotically negligible if the initial estimator achieves the $n^{-\frac{1}{4}}$ rate of convergence; however, in this case, iteration of the TMLE can be used to solve the EIC equation to an arbitrary level of precision.

In general, one does not need to always estimate the whole data density – instead, one specifies the required functional of the data density the target estimand relies upon, determines a loss function and LFM through the initial estimator of this functional (i.e., since the generalized score spans the canonical gradient), computes the MLE of the unknown parameters of the least favorable parametric model, and finally plugs the targeted estimator of the functional into the target parameter mapping. This plug-in estimator is now called the targeted minimum loss estimator (TMLE), since it does not require the loss to be the log-likelihood loss and allows one to focus only on the relevant functional of the data distribution.

In many problems, the target parameter depends on a collection of functionals (i.e., nuisance functions) of the data distribution. In that case, one can separately determine a least favorable submodel for each of these functionals by requiring its score (w.r.t. a specified loss) to span the corresponding component of the efficient influence curve. In this manner, one can target each nuisance function separately or sequentially (e.g., in such a way that a single-step TMLE already exactly solves the EIC equation). Such a TMLE solves the empirical mean of each component of the EIC, thus solving the EIC equation.

As with any of the three efficient estimation methods, the TMLE can also be carried out with sample splitting, a variant which we termed cross-validated TMLE (CV-TMLE) in Zheng and vdL (2011). The CV-TMLE uses an initial estimator fit on the training sample, carries out the TMLE updating step on the validation sample, and defines the CV-TMLE as the average across all of the sample splits of the resultant plug-in TMLE. In fact, the TMLE update step can also be pooled across the validation samples. Just as the sample-splitting variants of OSE and EEE, the CV-TMLE is asymptotically efficient without a need to assume the Donsker class condition, thereby allowing for overfitted initial estimators of the nuisance functions, so long as long as the second order remainder $R_2(\hat{P}, P_0)$ is asymptotically negligible.

Importantly, the TMLE is a plug-in estimator and thereby fully respects and incorporates the global constraints of both the statistical model and target parameter mapping, just like an MLE. A nice example of how TMLE fully utilizes constraints is given by the rare outcome TMLE (Balzer et al., 2016), in which both the initial estimator of the prediction function and the targeted version respect that the predictions are known to be small, greatly enhancing its performance in finite samples. In contrast to EEE, TMLE does not rely on the EIC being an estimating function; thus, it is always well defined – that is, there are no concerns about the existence of multiple solutions or the nonexistence of a solution. Since one need only compute a simple MLE, even if that MLE were to allow multiple solutions of its score equation, the empirical risk provides a criterion for choosing amongst such solutions. We also note that the TMLE is the only estimator that actually generalizes the MLE – if the MLE is well-defined and used as initial estimator, then the TMLE is exactly equivalent to the MLE (i.e., the targeting step will select zero fluctuation).

In addition, the TMLE procedure is able to augment the least favorable parametric fluctuation model with additional fluctuation parameters that generate additional estimating equations for the TMLE to then solve (that is, beyond the EIC equation). This is simply not possible within the OSE or EEE frameworks, since these have to commit to one particular gradient (e.g., see vdL, 2014 and Benkeser et al., 2017 demonstrating this nicely). These additional fluctuation parameters can then be chosen so that the TMLE obtains additional statistical properties beyond asymptotic efficiency. In this manner, we have proposed a very general TMLE procedure that

1. is guaranteed to be more efficient than a user-supplied estimator, even if one of the nuisance parameters is misspecified;
2. is not only double robust w.r.t. consistency but also preserves asymptotic linearity under misspecification of one or more of the nuisance parameters, thereby provideing double robust inference;
3. and may be used to construct higher-order TMLEs that reduce the second order remainder by incorporating the so-called higher-order efficient influence curve.

What’s more, there are many additional nuances to the TMLE framework that again make TMLE unique relative to OSE or EEE, including (to name but a few) collaborative TMLE (C-TMLE) for the targeted estimation of orthogonal nuisance functions (e.g., treatment and censoring mechanisms) in the least favorable submodel and targeted fluctuations based on universal least favorable submodels that target multidimensional or even infinite-dimensional target parameters (e.g., treatment-specific survival curves).

It should also be noted that from the very beginning, any of our applications of both EEE and TMLE have always accommodated machine learning; see, e.g., many of the articles, circa 1990 and onward, on EEE cited in the book by vdL and Robins (2001) as well as those on TMLE in the books by vdL and Rose (Targeted Learning (2011) and Targeted Learning in Data Science (2018)). By allowing for the use of machine learning, these approaches directly allow for double robust machine learning-based estimators, when they are applied to double robust estimation problems.

P.S., remember to write in to this blog at vanderlaan (DOT) blog [AT] berkeley (DOT) edu or @-mention mark_vdlaan on Twitter. Interesting questions will be answered on this blog!