*This post is part of our Q&A series.*

A question from Twitter on choosing between double machine learning and TMLE with cross-validation: https://twitter.com/emaadmanzoor/status/1208924841316880385

**Question:**

@mark_vdlaan Is there an applied researcher’s guide to choosing between double machine learning and TMLE + cross-fitting? PS: Thanks for making these methods and resources so easily accessible!

**Answer:**

Thanks for this interesting question. In the past several years, the interest in these machine learning-based estimators has become more widespread, since they allow for the statistical answer to a question to be framed in terms of scientifically meaningful parameters (e.g., defined through causal inference), incorporating machine learning in the estimation process, while providing formal statistical inference.

In order to provide a comprehensive answer, it is suitable to first define the
different classes of asymptotically efficient estimators. To start with, to
refer to a class of estimators as double machine learning (DML) estimators
suggests that this class only applies to a particular subclass of estimation
problems that allow for double *robustness*, or, involves machine learning of
multiple nuisance functions. Here, we are reminded that this robustness refers
to preservation of the consistency of the estimator under inconsistent
estimation of one or more of its nuisance parameters. Different estimation
problems have different structures, ranging from no form of robustness (i.e.,
consistency of the estimator under misspecification of a nuisance function), to
double robustness, triple robustness, an array of configurations of robustness,
to complete robustness. Therefore, (wrongly) terming an estimator in terms of a
very specific form of robustness suggests that it is a very limited procedure
fully relying on this very specific type of estimation problem. In addition, in
many estimation problems, one only has to apply machine learning to a single
functional parameter.

Efficient estimators are generally defined and will be double robust if the estimation problem falls in that category, and, in general, can be tailored to be, or is naturally, as robust as the estimation problem allows. So, it is more sensible to discuss different classes of efficient estimators – as we will see, the double machine learning approach falls in the category of estimating equation-based estimators. Efficient estimators rely on estimating nuisance parameters (functionals of the data distribution), and, if the statistical model for the data distribution is large, this would naturally require machine learning (also called data adaptive estimation). Thus, any of these efficient estimators could be called machine learning-based estimators, and, in fact, they have been applied using machine learning from very early on (circa 1980) in the literature.

We will only consider target parameters (which we will denote with `$\psi$`

)
that are so-called *pathwise differentiable*, making them potentially estimable
at `$\sqrt{n}$`

-rate. For simplicity, let’s focus on the case that we observe
`$n$`

independent identically distributed copies of a random variable `$O$`

with
true data distribution `$P_0$`

. Note that all we really need is local asymptotic
normality of the log-likelihood; therefore, this also applies to time series and
other dependent data structures (e.g., see *Efficient and Adaptive Estimation
for Semiparametric
Models*
by Bickel, Klaassen, Ritov, and Wellner). So, what does it mean to state that an
estimator of a pathwise differentiable target parameter is *asymptotically
efficient* at a given data distribution? An estimator is asymptotically
efficient if it can be approximated in first order as an empirical mean of the
so-called canonical gradient of the pathwise derivative of the target parameter.
To determine the canonical gradient, one views the estimand as a mapping from
all possible data distributions in the statistical model, say `$\mathcal{M}$`

,
to the parameter space (typically, the real line `$\mathbb{R}$`

) and determine
its pathwise derivative along a rich collection of paths through the data
distribution – this pathwise derivative is uniquely characterized by its
canonical gradient. Owing to its use in the definition of efficient estimators,
the canonical gradient at a data distribution is also called the *efficient
influence curve* (`$\text{EIC}$`

or `$\text{EIC}(P)$`

at a particular data
distribution `$P$`

). Recall that an estimator is *asymptotically linear* if it
can be approximated by an empirical mean of a particular function of the unit
data structure $O$; the influence curve is this same particular function.
Importantly, an efficient estimator is asymptotically the best (i.e., most
efficient) among the class of all regular asymptotically linear estimators –
or, in fact, among all regular estimators. What’s more, an efficient estimator
is asymptotically normally distributed with asymptotic variance equal to the
variance of the efficient influence curve.

Given a statistical model `$\mathcal{M}$`

(i.e., a set of possible probability
distributions of the unit-level data `$O$`

) and that the target parameter
`$\psi$`

is a mapping from the statistical model to the parameter space (e.g.,
the real line `$\mathbb{R}$`

), the statistical estimation problem is now fully
defined, and one can proceed to compute the canonical gradient of the pathwise
derivative of the target parameter. Perhaps unsurprisingly, the construction of
an efficient estimator necessarily involves using the canonical gradient. The
three classes of such efficient estimators are composed of

- the one-step estimator (OSE);
- the estimating equation estimator (EEE); and
- the targeted minimum loss estimator (TMLE).

The asymptotic efficiency of these estimators relies on a second order term to
be negligible, thereby typically requiring highly adaptive estimators of the
nuisance functions in the canonical gradient (those that will achieve the
convergence rate of `$n^{\frac{1}{4}}$`

). This second order term is defined in
terms of the canonical gradient (i.e., ```
$R_2(P, P_0) = \Psi(P) - \Psi(P_0) +
\mathbb{E}_{P_0} \text{EIC}(P)$
```

in my work). Additionally, when using variants
of these estimators that do not make use sample splitting, one also then relies
on a Donsker class condition (from empirical process theory), while, if one uses
the sample splitting variants, then this Donsker class condition is not needed.
It is the form that this second order remainder `$R_2(P,P_0)$`

takes that
determines the robustness structure of the estimation problem – for example, in
double robust problems, the term typically involves products of differences
between nuisance function under `$P$`

and true nuisance function (i.e., under
`$P_0$`

).

Recall that the classical way to obtain an efficient estimator is via maximum likelihood estimation. Unfortunately, when the statistical model is large, the the maximum likelihood estimator (MLE) is generally not defined. To work around this issue, one might consider regularizing the MLE by introducing a tuning parameter such as a sieve-based MLE, involving data adaptive selection of a submodel among a collection of submodels that approximate the actual statistical model. If this tuning parameter in the regularized MLE is optimized w.r.t. the density itself, the resulting regularized MLE (a plug-in estimator) of the target parameter will still fail to be asymptotically linear. Despite this, there remain interesting classes of regularized MLEs in which an undersmoothed choice of tuning parameter will result in the construction of an efficient estimator. In recent work, we established this result for the so-called Highly Adaptive Lasso (HAL) MLE (minimum loss estimation), in which one computes an MLE over a class of cadlag functions with a universal bound on the sectional variation norm, selecting the sectional variation norm with an undersmoothing-based selector. In any case, such regularized MLEs are not targeted towards a particular estimand and could therefore be considered less powerful than one of the efficient estimators that utilize the canonical gradient for a particular estimand. There is more to say about this, including great work in the literature on this topic by Shen and Newey, among others, but that lies beyond the scope of this comment.

We mention the HAL-MLE above since it is very relevant for efficient estimation.
Just recently, Bibaut and vdL (2019) showed
that the HAL-MLE converges to the true functional parameter at a rate as fast as
`$n^{-\frac{1}{3}}$`

up to a `$\log(n)$`

factor. Thus, using this HAL-MLE to
estimate the nuisance functions in the OSE, EEE, or TMLE would guarantee that
these estimators are indeed asymptotically efficient. In addition, the HAL-MLE
automatically satisfies the Donsker class condition as well, implying that the
non-sample split variants of the OSE, EEE, and TMLE using will be asymptotically
efficient when constructed via the HAL-MLE; the only assumption embedded in this
approach is that the true nuisance functions are cadlag and have finite
sectional variation norm (with the `$\text{EIC}$`

being a bounded function).

With this background, we are now well-positioned to proceed with discussing the three classes of efficient estimators (OSE, EEE, and TMLE). Along the way, we’ll provide some historical context and discuss how DML fits into this landscape.

Historically, the first efficient estimator was the so called one-step estimator
using sample splitting (the work of Levin, Pfanzagl, and Klaassen circa
1970-1980, see references in the book by Bickel, Klaassen, Ritove, and
Wellner). For example, Klaassen (1986) showed that the one-step estimator using
sample splitting is efficient under minimal conditions. The one-step estimator
(and its non-sample splitting version) is the efficient estimator presented in
the comprehensive book on efficient estimation in semiparametric models
(*Efficient and Adaptive Estimation for Semiparametric
Models*)
by Bickel, Klaassen, Ritov, and Wellner (1997). This one-step estimator is
defined as a plug-in *initial* estimator of the target parameter, plus the
empirical mean of the EIC at this same initial estimator. The sample splitting
variant of this estimator is defined as the average across a number of sample
splits in training and validation sample (say, V-fold), of the initial plug-in
estimator based on a training sample plus the empirical mean over the validation
sample of the EIC at this same initial plug-in estimator (fitted on the training
sample). A little later (circa 1980 and onward), as empirical process theory
emerged as a field, the sample splitting based one-step estimator was often
replaced by the “regular” (non-sample slitting) one-step estimator that relied
on the Donsker class condition (i.e., the sample splitting will come at a small
price if the Donsker class condition nicely holds, but one should make sure that
the initial estimator is not overfitted).

In the 1990s, the corresponding approach of estimating equation-based estimators
(EEE) was rigorously developed by Robins and collaborators, in which one
constructs efficient estimators as solutions of the EIC estimating equation,
– i.e., one sets empirical mean of `$\text{EIC}(\psi, \eta)$`

(for some nuisance
parameter vector `$\eta$`

) equal to zero – by estimating the nuisance function
`$\eta$`

in the EIC with an initial estimator and solving for the target
parameter `$\psi$`

. This relies on the EIC admitting an estimating function
representation, i.e., having `$\text{EIC}(P)$`

depend on the data distribution
`$P$`

through the target parameter `$\Psi(P)$`

and a nuisance parameter
`$\eta$`

. If such a representation of `$\text{EIC}(P) = \text{EIC}(\psi, \eta)$`

exists, the one-step estimator is equivalent to the first step of the
Newton-Raphson algorithm for solving the estimating equation. Fundamental work
on EEE was done by Robins and Rontitzky (circa 1992 and onward), and
collaborators, in the context of censored data and causal inference models,
involving clever representations and derivations of the EIC (e.g., the
*augmented IPCW* representation of the EIC). A comprehensive review and
treatment of the general efficient estimating equation methodology – going
beyond application to censored data and causal inference models, including its
theory – is presented in the book *Unified Methods for Censored Longitudinal
Data and Causality*
by van der Laan and Robins (2001).

In contrast with EEE, the OSE is always well-defined while the EEE requires

- the estimating function representation;
- that a solution of the EIC estimating equation exists; and
- if multiple solutions exists that we know how to select among them.

On the other hand, asymptotics of the OSE rely on the initial estimator, making it potentially less robust than EEE. For example, if the estimation problem is doubly robust (DR), but the initial estimator is not DR, then the OSE is generally not DR either, even though the EEE will be DR in such a case. Completely analogously to the OSE, one can use a sample splitting analogue of the EEE by defining the estimating equation as an average across the sample splits of the empirical mean over the validation sample of the EIC, in which nuisance parameters are estimated based on the training sample. In this manner, the first step of the Newton-Raphson algorithm for solving this sample splitting-based estimating equation is equivalent to the one-step estimator based on sample splitting. In the book by van der Laan and Robins (2001), focus was placed on the non-sample splitting-based EEE; thus, our theorems throughout that book (as well as the more general theorems in chapter 2) rely on a Donsker class condition. Specifically, we showed that if one were to use machine learning-based estimators whose realizations are cadlag functions with finite sectional variation norm, then this Donsker class condition holds, allowing for highly flexible machine learning (including the HAL-MLE) to be used.

Here, it is important to note that any gradient of the pathwise derivative of
the target parameter can be used to construct an asymptotically linear estimator
with influence curve equal to that gradient; moreover, by using the unique
canonical gradient, the constructed estimator will be asymptotically efficient.
Indeed, in Robins and Rotnitzky’s work, they always refer to the class of all
estimating functions orthogonal to the nuisance tangent space, which is
equivalent with the class of all gradients of the pathwise derivative of the
target parameter. Robins et al. show that the class of gradients can be computed
by determining the class of all functions of the unit-level data that are
orthogonal (in terms of the covariance operator) to all nuisance scores (i.e.,
scores of paths for which the pathwise derivative of the target parameter equals
zero). In problems in which the EIC is difficult to compute, one might decide to
settle for an inefficient OSE or EEE implied by an easier-to-compute gradient
(see the book by van der Laan and Robins, 2001). In an estimation problem with
the double robustness structure, any of such inefficient OSE or EEE based on an
inefficient gradient will still be double robust (i.e., the robustness implied
by the second order remainder `$R_2(P, P_0)$`

applies to any gradient, since
what matters is that it is orthogonal to the nuisance scores, not that it has
minimal variance).

A problem with both EEE and OSE (regardless of whether sample splitting is used) is that these estimators suffer from the fact that they are not plug-in estimators (i.e., one applies the target parameter mapping to an estimated data distribution in the model); thus, they might not satisfy crucial global constraints in the statistical model. For example, if the canonical gradient is highly variable or if the sample size is small, it can easily happen that the OSE and EEE end up estimating a probability with a number larger than one or smaller than zero, thus not respecting the bounds on the parameter (i.e., that probabilities lie between zero and one, inclusive). Moreover, by potentially failing to respect these important global statistical constraints in the model, the OSE and EEE lack robustness (e.g., just one observation could throw off the estimator badly), in stark contrast to an MLE, which always necessarily fully respects global constraints of the model. Of course, global constraints are asymptotically irrelevant, yet are very important in finite samples.

By contrast, TMLE produces an efficient plug-in estimator, performing updates
in the model space so as to avoid any risk of violating bounds on the density of
the data and on the parameter space. Specifically, the targeted maximum
likelihood estimator takes an initial estimator of the density of the data,
constructs a parametric model through this initial estimator with score spanning
the canonical gradient at the initial estimator, and fits the unknown
parameter(s) in this so-called least favorable parametric model (LFM) via MLE.
The resultant targeted density estimator is then plugged in to the target
parameter mapping to obtain the TMLE of the target parameter. The LFM through
the initial estimator can often be chosen to be a standard parametric model
treating as offset the initial estimator. For example, the targeting step in the
TMLE of the ATE can be carried out with a logistic regression using a clever
covariate, using as offset the initial estimator. In more recent work (vdL
and Gruber, 2015), we
propose a so-called *universal* least favorable submodel, which guarantees both
that the targeting step is maximally robust and that the TMLE exactly solves the
EIC estimating equation in a single step. By contrast, the originally proposed
*local* LFM will solve the EIC equation up to an approximation error that is
asymptotically negligible if the initial estimator achieves the
`$n^{-\frac{1}{4}}$`

rate of convergence; however, in this case, iteration of
the TMLE can be used to solve the EIC equation to an arbitrary level of
precision.

In general, one does not need to always estimate the whole data density –
instead, one specifies the required functional of the data density the target
estimand relies upon, determines a loss function and LFM through the initial
estimator of this functional (i.e., since the generalized score spans the
canonical gradient), computes the MLE of the unknown parameters of the least
favorable parametric model, and finally plugs the targeted estimator of the
functional into the target parameter mapping. This plug-in estimator is now
called the *targeted minimum loss estimator* (TMLE), since it does not require
the loss to be the log-likelihood loss and allows one to focus only on the
relevant functional of the data distribution.

In many problems, the target parameter depends on a collection of functionals (i.e., nuisance functions) of the data distribution. In that case, one can separately determine a least favorable submodel for each of these functionals by requiring its score (w.r.t. a specified loss) to span the corresponding component of the efficient influence curve. In this manner, one can target each nuisance function separately or sequentially (e.g., in such a way that a single-step TMLE already exactly solves the EIC equation). Such a TMLE solves the empirical mean of each component of the EIC, thus solving the EIC equation.

As with any of the three efficient estimation methods, the TMLE can also be
carried out with sample splitting, a variant which we termed *cross-validated
TMLE* (CV-TMLE) in Zheng and vdL
(2011). The
CV-TMLE uses an initial estimator fit on the training sample, carries out the
TMLE updating step on the validation sample, and defines the CV-TMLE as the
average across all of the sample splits of the resultant plug-in TMLE. In fact,
the TMLE update step can also be pooled across the validation samples. Just as
the sample-splitting variants of OSE and EEE, the CV-TMLE is asymptotically
efficient without a need to assume the Donsker class condition, thereby allowing
for overfitted initial estimators of the nuisance functions, so long as
long as the second order remainder `$R_2(\hat{P}, P_0)$`

is asymptotically
negligible.

Importantly, the TMLE is a plug-in estimator and thereby fully respects and incorporates the global constraints of both the statistical model and target parameter mapping, just like an MLE. A nice example of how TMLE fully utilizes constraints is given by the rare outcome TMLE (Balzer et al., 2016), in which both the initial estimator of the prediction function and the targeted version respect that the predictions are known to be small, greatly enhancing its performance in finite samples. In contrast to EEE, TMLE does not rely on the EIC being an estimating function; thus, it is always well defined – that is, there are no concerns about the existence of multiple solutions or the nonexistence of a solution. Since one need only compute a simple MLE, even if that MLE were to allow multiple solutions of its score equation, the empirical risk provides a criterion for choosing amongst such solutions. We also note that the TMLE is the only estimator that actually generalizes the MLE – if the MLE is well-defined and used as initial estimator, then the TMLE is exactly equivalent to the MLE (i.e., the targeting step will select zero fluctuation).

In addition, the TMLE procedure is able to augment the least favorable parametric fluctuation model with additional fluctuation parameters that generate additional estimating equations for the TMLE to then solve (that is, beyond the EIC equation). This is simply not possible within the OSE or EEE frameworks, since these have to commit to one particular gradient (e.g., see vdL, 2014 and Benkeser et al., 2017 demonstrating this nicely). These additional fluctuation parameters can then be chosen so that the TMLE obtains additional statistical properties beyond asymptotic efficiency. In this manner, we have proposed a very general TMLE procedure that

- is guaranteed to be more efficient than a user-supplied estimator, even if one of the nuisance parameters is misspecified;
- is not only double robust w.r.t. consistency but also preserves asymptotic linearity under misspecification of one or more of the nuisance parameters, thereby provideing double robust inference;
- and may be used to construct
*higher-order TMLEs*that reduce the second order remainder by incorporating the so-called higher-order efficient influence curve.

What’s more, there are many additional nuances to the TMLE framework that again make TMLE unique relative to OSE or EEE, including (to name but a few) collaborative TMLE (C-TMLE) for the targeted estimation of orthogonal nuisance functions (e.g., treatment and censoring mechanisms) in the least favorable submodel and targeted fluctuations based on universal least favorable submodels that target multidimensional or even infinite-dimensional target parameters (e.g., treatment-specific survival curves).

It should also be noted that from the very beginning, any of our applications of
both EEE and TMLE have always accommodated machine learning; see, e.g., many of
the articles, circa 1990 and onward, on EEE cited in the book by vdL and Robins
(2001) as well as those on TMLE in the books by vdL and Rose (*Targeted
Learning* (2011)
and *Targeted Learning in Data Science*
(2018)). By allowing
for the use of machine learning, these approaches directly allow for double
robust machine learning-based estimators, when they are applied to double robust
estimation problems.

Finally, we can discuss how the recent DML methodology fits into this rich
previous literature on OSE, EEE, and TMLE. DML falls in the category of the
estimating equation-based estimators, which construct estimators as solutions of
a gradient-based estimating equation, resulting in the EIC estimating equation
if one selects the canonical gradient of the pathwise derivative. DML represents
a subclass of the EEE approach but uses a different terminology than that of
Robins and Rotnitzky (and, thus, different too from that of vdL and Robins,
2001): it enforces sample-splitting and establishes various interesting and
important theoretical results for a variety of estimation problems, going beyond
practical implementations. A first example of the different terminology is the
name *cross-fitting*, which had been called sample-splitting before (or simply
cross-validation in CV-TMLE). Despite the change in name, cross-fitting
precisely represents the sample-splitting used early on in OSE and in CV-TMLE.
Another example of different terminology is the usage of the *Neyman
orthogonality* of estimating functions; in Robins and Rotnitzky’s work, one
defines the class of estimating functions in terms of the orthogonal complement
of the nuisance tangent space, thereby guaranteeing that the estimating
functions at their true parameter values are orthogonal to any nuisance score
(e.g., see chapter 1 of the book by vdL and Robins). Equivalently, the class of
estimating functions is simply provided by the class of gradients of the target
parameter, where any gradient is automatically orthogonal to the nuisance
tangent space. Thus, in the DML line of work, the term “orthogonal to the
nuisance tangent space” (from the work of Robins) is replaced by Neyman
orthogonality. This orthogonality of the estimating function to the nuisance
tangent space guarantees that the estimation of the nuisance parameter only
results in second order contributions to the EEE, often resulting in so-called
robustness (as in unbiasedness) of the estimating function under
misspecification of the nuisance parameter and, thus, also of the corresponding
EEE (again, see chapter 1 of the book by vdL and Robins for formal results).
Lastly, it is my understanding that, contrary to determining the class of
gradients and specifically the canonical gradient of the target parameter (as is
done in EEE), in DML, one often starts out with a particular estimating function
that has a nuisance parameter and then subtract off its projection on the
nuisance tangent space. This is precisely how Robins and Rotnitzky obtained the
class of augmented inverse probability of censoring weighted (AIPCW) estimating
functions, i.e., by subtracting off from an IPCW estimating function its
projection on the tangent space of the censoring mechanism, often only requiring
the assumption of coarsening at random to obtain maximal efficiency.
Nonetheless, orthogonalizing a given estimating function w.r.t. a nuisance
parameter maps the estimating function into a particular element of the
orthogonal complement in the nuisance tangent space for the statistical model
under consideration (i.e., one of the gradients of the pathwise derivative of
the target parameter, up to a normalizing matrix/constant which does not matter
for an estimating equation). Depending on the starting estimating function, the
orthogonalized version ends up being a gradient or canonical gradient, up to a
normalizing constant/matrix. Therefore, the DML approach is precisely a subset
of the general EEE approach originally developed by Robins and Rotnitzky – in
particular, in coarsening at random censored data models, the general AIPCW
representation theorem of Robins and Rotnitzky for the class of all estimating
functions (i.e., those in the orthogonal complement of the nuisance tangent
space) also characterizes the initial IPCW estimating function whose
orthognalized variant is the canonical gradient and, thus, the EIC (see chapter
1, section 1.5, in the book by vdL and Robins).

There is much more that could be elaborated upon here, but we have covered some of the points most relevant to distinguishing between the TMLE, EEE, OSE, and the DML frameworks. A former PhD student of mine, Iván Díaz (now faculty at Weill Cornell Biostatistics), has recently written a review article discussing the two approaches of TMLE and DML in some detail as well.

Best Wishes,

Mark and Nima

**P.S.**, remember to write in to this blog at ```
vanderlaan (DOT) blog [AT]
berkeley (DOT) edu
```

or @-mention
`mark_vdlaan`

on Twitter. Interesting
questions will be answered on this blog!