The performance measures described in the prior chapter are all defined in terms of an infinite number of replications of the data-generating process. Of course, simulations only involve a finite set of replications, based on which we estimate the measures. If we re-ran the simulation (with a different seed), our estimates would change. We use the term Monte Carlo error to describe the variability of performance measure estimates that is due to using a random, finite set of replications. To understand how good our estimates of performance are, we need to assess how large the Monte Carlo error is likely to be.
To account for Monte Carlo error, we can think of simulation results as a sample from a population. Each replication is an independent draw from the population of the sampling distribution. Once we frame the problem in these terms, we can apply standard statistical techniques to calculate standard errors for our performance measures, which we will call Monte Carlo Standard Errors, or MCSEs. For most performance measures, closed-form expressions for MCSEs are available. If no closed-form expressions are available, we can still estimate MCSEs using techniques such as the jackknife or bootstrap.
10.1 MCSEs for conventional measures for point estimators
For the performances measures that we have described for evaluating point estimators, Monte Carlo standard errors can be calculated using conventional formulas.1 When we run a simulation, we end up with \(R\) replicates of a point estimator \(T\) of a target parameter \(\theta\). A performance measure of \(T\) (e.g., bias, RMSE, or SE) is a summary across these \(R\) replicates, approximating what we would get if we did \(R=\infty\) replications.
For example, the true bias of \(T\) is \(\E(T - \theta)\), which is the expected value of \(T\) minus the true parameter value, and we estimate the bias as \(\bar{T} - \theta\). Each replicate of \(T - \theta\) is an estimate of the expected value of \(T - \theta\), and, in this case, we have \(R\) independent draws of \(T - \theta\) and can use simple sampling theory to get the estimated standard error for the population mean of them: \[
MCSE\left(\widehat{\Bias}(T)\right) = \sqrt{\frac{S_T^2}{R}},
\tag{10.1}\] where \(S_T\) is the sample standard deviation of \(T\) across the \(R\) replications of the simulation process (noting that \(Var(T-\theta) = Var(T)\)). Equation 10.1 is simply the standard error for the mean of \(R\) independent samples from a population.
This approach is readily extended to estimators of other performance measures. If we have a performance measure \(M\) for each replicate in a simulation, we can estimate average performance as the average of the sample of \(R\) performances, \(\bar{M}\). The standard error for average performance then follows along the same lines as Equation 10.1. For example, the Mean Square Error (the square of RMSE) is simply the average of \(M = (T-\theta)^2\) and the estimated MCSE of the MSE is then \[
MCSE( \widehat{MSE} ) = \sqrt{\frac{\Var(M)}{R}} = \sqrt{\frac{\Var\left((T-\theta)^2\right)}{R}}.
\tag{10.2}\] where \(\Var\left((T-\theta)^2\right)\) is the sampling variance of the \(R\)\(M=(T-\theta)^2\) values. For quantifying the accuracy of an estimator, we recommend using RMSE, or the square root of the MSE, as shown in Equation 9.5. To get the MCSE for the RMSE, we can use a delta method approximation.2: \[
MCSE( \widehat{RMSE} ) = \frac{1}{2 \times RMSE} \sqrt{\frac{Var((T-\theta)^2)}{R}}.
\tag{10.3}\] The delta method approach can be applied to many other quantities of interest as well.
The MCSE for the true standard error \(S_T\) of \(T\) does not directly follow the above pattern because the sampling variance \(S_T^2\) of \(T\) is estimated across all the replicates, rather than simply being an average of \(R\) independent estimates. Statistical theory gives an estimated standard error of a variance estimator as \[
MCSE\left(\widehat{\Var}(T)\right) = MCSE\left(S^2_T\right) = S_T^2 \sqrt{\frac{k_T - 1}{R}},
\tag{10.4}\] where \(k_T\) is the sample standardized kurtosis of \(T\): \[
\begin{aligned}
k_T &= \frac{1}{S_T^4} \frac{1}{R}\sum_{r=1}^R \left(T_r - \bar{T}\right)^4.
\end{aligned}
\tag{10.5}\] For a distribution, kurtosis would be \(\kappa_T = E[ (T-E[T])^4 ]/\sigma^4\). The kurtosis is a fourth moment (mean is first, variance is second, skew is third), and measures, in effect, the thickness of the tails of a distribution. The standardized kurtosis is a unitless quantity (note the scaling of \(1/\sigma^4\)); for a normal distribution \(\kappa\) equals 3.
To get the MCSE of the standard error, use the delta method on the variance to get: \[
MCSE\left(S_T\right) \approx \frac{1}{2 S_T} MCSE( S^2_T ) = \frac{S_T}{2}\sqrt{\frac{k_T - 1}{R}}.
\tag{10.6}\]
10.1.1 Connecting bias, RMSE, and SE
\(S_T\) is the sample standard deviation of \(T\), which is, by definition, the true standard error of \(T\). The true MCSE of bias is of the form \(\sigma / \sqrt{R}\), where \(\sigma^2 = \Var(T)\) and thus \(\sigma = SE(T)\). The MCSE of the standard error is of the form \(c \sigma / \sqrt{R}\), where \(c = \sqrt{\kappa - 1}/2\) is a constant depending on the distribution of \(T\). For example, if \(T\) is normally distributed, \(\kappa = 3\) and \(c = 1/\sqrt{2} \approx 0.70\).
For the RMSE, we have a true MCSE of \[
MCSE( \widehat{RMSE} ) = \frac{\sqrt{Var((T-\theta)^2)}}{2 \sigma \sqrt{\sigma^2 + b^2}} \times \frac{\sigma}{\sqrt{R}}
\] where the ugly first term is again going to be a constant \(c\) depending on the distribution of \(T\) (here \(b\) is the bias, i.e. \(E[T]-\theta\)). For example, if \(T\) is normal around \(\theta\), \(c\) collapses to \(1/\sqrt{2}\) (coinciding with the SE, since there is no bias). If \(T\) is normal but shifted by bias \(b\) that is equal to the standard error (this would be a very big bias), we get a constant of \(\sqrt{3}/2\). It turns out this constant can not be bigger than 1 unless \(T\) is substantially skewed (see exercises).
Taking the totality of the above, a reasonable, usually conservative, rule of thumb is the MCSE for a typical performance estimate of a point estimator can be taken as \(\sigma / \sqrt{R}\) (the one for bias). This can be used to quickly get a sense of how precise a simulation will be. For example, the MCSE for \(R=1000\) is going to be about 3% of the true variability of \(T\), \(\sigma\), for bias, SE, and RMSE. For \(R=100\), it is 10%.
10.1.2 Relative-to-parameter performance measures
Section 9.5 discussed circumstances where we might prefer to calculate performance measures in relative rather than absolute terms to some known parameter. For measures that are calculated by dividing a raw measure by the known target parameter, the MCSE for the relative measure is simply the MCSE for the raw measure divided by the target parameter. For instance, the MCSE of relative bias \(\bar{T} / \theta\) is \[
MCSE\left( \frac{\bar{T}}{\theta} \right) = \frac{1}{\theta} MCSE(\bar{T}) = \frac{S_T}{\theta \sqrt{R}}.
\tag{10.7}\] MCSEs for relative variance and relative RMSE follow similarly.
Importantly, measures relative to an unknown quantity, such as the true standard error or the performance of another method, is more tricky. See Section 10.3.
10.1.3 Robust performance measures
In Section 9.1.2 we described several alternative performance measures for evaluating point estimators. MCSEs for these less conventional measures can be obtained using results from the theory of robust statistics (Hettmansperger and McKean 2010; Maronna et al. 2006).
McKean and Schrader (1984) proposed a standard error estimator for the sample median from a continuous but not necessarily normal distribution, derived from a non-parametric confidence interval for the sample median. We use their approach to compute a MCSE for \(M_T\), the sample median of \(T\). Let \(c = \left\lceil(R + 1) / 2 - 1.96 \times \sqrt{R/4}\right\rfloor\), where the inner expression is rounded to the nearest integer. Then \[
MCSE\left(M_T\right) = \frac{T_{(R + 1 - c)} - T_{(c)}}{2 \times 1.96}.
\tag{10.8}\] A Monte Carlo standard error for the median absolute deviation can be computed following the same approach, but substituting the order statistics of \(E_r = | T_r - \theta|\) in place of those for \(T_r\).
For the trimmed mean bias with trimming proportion \(p\), a MCSE is \[
MCSE\left(\tilde{T}_{\{p\}}\right) = \sqrt{\frac{U_p}{R}},
\tag{10.9}\] where \[
U_p = \frac{1}{(1 - 2p)R}\left( pR\left(T_{(pR)} - \tilde{T}_{\{p\}}\right)^2 + pR\left(T_{((1-p)R + 1)} - \tilde{T}_{\{p\}}\right)^2 + \sum_{r=pR + 1}^{(1 - p)R} \left(T_{(r)} - \tilde{T}_{\{p\}}\right)^2 \right) .
\](Maronna et al. 2006, Eq. 2.85). The additional uncertainty due to the thresholds being estimated is ignorable.
Performance measures based on winsorization include winsorized bias, winsorized standard error, and winsorized RMSE. MCSEs for these measures can be computed using the same formuals as for the conventional measures of bias, standard error, and RMSE, but using sample moments of \(\hat{X}_r\) in place of the sample moments of \(T_r\). The additional uncertainty from the thresholds being estimated is ignorable.
10.2 MCSEs for confidence intervals and hypothesis tests
Performance measures for confidence intervals and hypothesis tests are simple compared to those we have described for point and variance estimators, and come directly from the theory of estimating the proportion in a population. First, the MCSE for the estimated rejection rate is \[
MCSE(r_\alpha) = \sqrt{\frac{r_\alpha ( 1 - r_\alpha)}{R}}.
\tag{10.10}\] This MCSE uses the estimated rejection rate to approximate its Monte Carlo error. When evaluating the validity of a test, we might expect the rejection rate to be fairly close to the nominal \(\alpha\) level, in which case we could compute a MCSE using \(\alpha\) in place of \(r_\alpha\), taking \(\sqrt{\alpha(1 - \alpha) / R}\). When evaluating power, we will not usually know the neighborhood of the rejection rate in advance of the simulation. However, a conservative upper bound on the MCSE can be derived by observing that MCSE is maximized when \(r_\alpha = \frac{1}{2}\), giving \[
MCSE(r_\alpha) \leq \frac{1}{2 \sqrt{R}}.
\]
When evaluating confidence interval performance, we focus on coverage rates and expected widths. MCSEs for the estimated coverage rate work similarly to those for rejection rates. If the coverage rate is expected to be in the neighborhood of the intended coverage level \(\beta\), then we can approximate the MCSE as \[
MCSE(\widehat{\text{Coverage}}(A,B)) = \sqrt{\frac{\beta(1 - \beta)}{R}}.
\tag{10.11}\] Alternately, Equation 10.11 could be computed using the estimated coverage rate \(\widehat{\text{Coverage}}(A,B)\) in place of \(\beta\).
Finally, the expected confidence interval width is estimated as \(\bar{W}\), with MCSE \[
MCSE(\bar{W}) = \sqrt{\frac{S_W^2}{R}},
\tag{10.12}\] where \(S_W^2\) is the sample variance of \(W_1,...,W_R\), the widths of the confidence interval from each replication.
10.3 MCSEs for ratio-based and non-standard performance measures
Estimating the MCSE of relative performance measures for variance estimators is complicated by the appearance of an estimated quantity in the denominator of the ratio. For instance, the relative bias of \(V\) is estimated as the ratio \(\bar{V} / S_T^2\), and both the numerator and denominator are estimated quantities that will include some Monte Carlo error. To properly account for the Monte Carlo uncertainty of the ratio, one can use formulas for the standard errors of ratio estimators. Alternately, we can use general uncertainty approximation techniques such as the jackknife (Boos and Osborne 2015) or bootstrap (Efron and Tibshirani 1994). We advocate for these more flexible approaches, and discuss them both in the following.
10.3.1 The jackknife for relative variance estimators
The jackknife involves calculating a statistic of interest repeatedly, each time excluding one observation. The variance of this set of one-left-out statistics usually serves as a reasonable approximation to the actual sampling variance of the statistic calculated from the full sample.
Say we have some performance measure \(M\). To do the jackknife, calculate \(M_{(j)}\), for \(j = 1,...,R\), as the performance calculated on the \(R\) simulation replicates excluding replicate \(j\). Then take the MCSE of \(M\) as \[
MCSE\left( M \right) = \sqrt{\frac{1}{R} \sum_{j=1}^R \left( M_{(j)} - M\right)^2}.
\]
For example, to apply the jackknife to assess the MCSEs of the calibration, or relative bias, of a variance estimator, \(M = \bar{V}_{(j)} / S_{T(j)}^2\), we will need to compute several statistics repeatedly. Let \(\bar{V}_{(j)}\) and \(S_{T(j)}^2\) be the average variance estimate and the variance estimate calculated from the set of replicates that excludes replicate \(j\). The relative bias estimate, excluding replicate \(j\), would then be \(\bar{V}_{(j)} / S_{T(j)}^2\). Calculating all \(R\) versions of this relative bias estimate and taking the variance of these \(R\) versions yields a jackknife MCSE: \[
MCSE\left( \frac{ \bar{V}}{S_T^2} \right) = \sqrt{\frac{1}{R} \sum_{j=1}^R \left(\frac{\bar{V}_{(j)}}{S_{T(j)}^2} - \frac{\bar{V}}{S_T^2}\right)^2}.
\tag{10.13}\] Similarly, a MCSE for the relative standard error of \(V\) is \[
MCSE\left( \frac{ S_V}{S_T^2} \right) = \sqrt{\frac{1}{R} \sum_{j=1}^R \left(\frac{S_{V(j)}}{S_{T(j)}^2} - \frac{S_V}{S_T^2}\right)^2},
\tag{10.14}\] where \(S_{V(j)}\) is the sample variance of \(V_1,...,V_R\), omitting replicate \(j\).
Jackknife calculation would be cumbersome if we did all of the above by brute force. Happily, a few algebra tricks provide a much quicker way. The tricks come from observing that \[
\begin{aligned}
\bar{V}_{(j)} &= \frac{1}{R - 1}\left(R \bar{V} - V_j\right) \\
S_{V(j)}^2 &= \frac{1}{R - 2} \left[(R - 1) S_V^2 - \frac{R}{R - 1}\left(V_j - \bar{V}\right)^2\right] \\
S_{T(j)}^2 &= \frac{1}{R - 2} \left[(R - 1) S_T^2 - \frac{R}{R - 1}\left(T_j - \bar{T}\right)^2\right]
\end{aligned}
\] These formulas can be used to avoid re-computing the mean and sample variance from every subsample. Instead, all we need to do is calculate the overall mean and overall variance, and then do a small adjustment with each jackknife iteration.
Jackknife methods are useful for approximating MCSEs of other performance measures beyond just those for variance estimators. For instance, the jackknife is a convenient alternative for computing the MCSE of the standard error or (raw) RMSE of a point estimator, which avoids the need to compute skewness or kurtosis. However, Boos and Osborne (2015) notes that the jackknife does not work for performance measures involving percentiles, including medians; for such statistics, bootstrapping techniques (discussed next) usually remains valid.
10.3.2 The bootstrap for complex performance measures
An alternative to the jackknife for estimating MCSEs is the bootstrap (Efron and Tibshirani 1994). The bootstrap involves repeatedly resampling the set of simulation replications with replacement, calculating the performance measure of interest for each resample, and then calculating the standard deviation of the performance measures across the resamples.
To illustrate, we calculate a bootstrap MCSE for two quantities from our running cluster RCT example: the relative improvement in variance for Aggregation vs. Linear Regression, and the relative bias of Aggregation’s variance estimator (i.e., how well calibrated it is).
We first calculate these performance measures on our simulation trials:
As before, we see that Aggregation has variance around 9% smaller than linear regression and that its variance estimator is inflated by around 13%. Let’s make bootstrap confidence intervals on these two quantities to account for Monte Carlo error in our simulation.
We want to resample entire simulation iterations to take within-dataset correlations of the model estimates into account. To do this, we first pivot our raw simulation results to wide format:
The sample_n() function draws a sample of 1000 rows from the runW dataset, sampling with replacement so that some rows will be repeated multiple times. This results in a bootstrapped dataset, run_star, on which we calculate performance measures. The whole process is repeated 500 times using repeat_and_stack(), resulting in 1000 bootstrap iterations of the performance measures.
For simplicity, we are only looking at the aggregation estimator (although we could extend the code to bootstrap all of the performance measures for all of the estimators). Once we have 500 bootstrap iterations, we can evaluate how much our performance measures change across iterations:
Interpreting the confidence interval in the second row, Aggregation has a lower variance than linear regression, with the upper end of the MCSE-based confidence interval being 97% (less than 100%). Second, we have evidence that the Aggregation variance estimator is systematically too high, with the lower end of the confidence interval being above 1 (but only just). If we had run fewer simulation iterations, we may well not have been able to make these claims with certainty.
The bootstrap is a very general tool. It can be applied to assess MCSE for nearly any performance measure you might encounter.
10.4 Doing it all at once with the simhelpers package
The simhelpers package provides several functions for calculating most of the performance measures that we have reviewed, along with MCSEs for each. The functions are easy to use. The package also includes some example datasets, including a dataset called welch_res that contains simulation results for the conventional t-test and Welch’s t-test for unequal variances, similar to the case study in Chapter 3. Here is the raw simulation output:
We first assess the rejection rate for the traditional \(t\)-test based on the subset of simulation results with sample sizes of \(n_1 = n_2 = 50\) and a mean difference of 0, using \(\alpha\) levels of .01 and .05:
The column labeled K_rejection reports the number of replications used to calculate the performance measures. The MCSE columns are the Monte Carlo uncertainty for the measures.
Here is the coverage rate calculated for the same condition:
The performance functions are designed to be used within a tidyverse-style workflow, including on grouped datasets. For instance, we can calculate the performance measures for all the methods for our given scenario:
Later, when we analyze multifactor simulations, this grouping becomes even more powerful: we will group by the simulation factors and evaluate performance for all our scenarios at once.
10.4.1 Quick confidence interval coverage
In Section 6.1.2, we created the function r_bivariate_Poisson() for simulating data from a bivariate Poisson distribution. In Section 7.1, we developed a function called r_and_z() for computing confidence intervals for Pearson’s correlation using Fisher’s \(z\) transformation. In Section 8.4, we created a simulation driver function, sim_r_Poisson() from these components. We can use this function to generate 1000 simulated confidence intervals:
bvp_CIs <-sim_r_Poisson( 1000, N =30, rho =0.4, mu1 =8, mu2 =14 )head(bvp_CIs, n =3)
It will be useful to have a function that evaluates the coverage and width of these confidence intervals based on a simulated dataset. We can create such a function using calc_coverage():
10.4.2 Quick performance calculations for our cluster RCT example
In Section 9.1.1, we computed performance measures for three point estimators of the school-level average treatment effect in a cluster RCT. We next carry out the same calculations using the calc_absolute() function from simhelpers, which also provides MCSEs. Examining the MCSEs is useful to ensure that our choice of 1000 replications of the simulation is sufficient to provide reasonably precise estimates of the performance measures. We have:
The MCSEs are small relative to the linear regression bias term and all of the SEs (stddev) and RMSEs. Results based on 1000 replications seems adequate to support our conclusions about the gross trends identified. We cannot rule out the possibility that the aggregation estimator and multilevel modeling estimator are slightly biased: they could have true bias of as much as 0.014 (two MCSEs).
For our variance estimators, we can use the calc_relative_var() function to compute performance measures and MCSEs using the jackknife:
We get essentially the same results as what we found in Section 10.3.2 using bootstrapping.
10.5 Concluding thoughts
For most classic performance measures, calculating a MCSE is a straightforward process most easily done using a software package. In most cases, uncertainty here is like uncertaintly anywhere: it goes down with the square root of the number of trials run. Under some metrics, this means running 1000 trials often is good enough. Going to 10,000 will shrink MCSEs by around \(1/\sqrt{10}\), or about a third. Similarly, dropping to 100 will increase them by around a factor of 3.
Some performance measures are harder to get MCSEs for. You should still try. We provide two tools for this, the Jackknife and the bootstrap. Even if an MCSE is not exactly right, if it is within a factor of 2 it can still be very helpful in understanding how much your performance estimates are moving around due to chance vs. something real.
Exercises
Exercise 10.1 (Jackknife calculation of MCSEs for RMSE) The following code generates 100 replications of a simulation of three average treatment effect estimators in a cluster RCT, using a simulation driver function we developed in Section 8.3 using components described in Section 6.6 and Section 7.2.
Compute the RMSE of each estimator, and use the jackknife technique described in Section 10.3 to compute a MCSE for the RMSE. Check your results against the results from calc_absolute() in the simhelpers package.
Exercise 10.2 (Jackknife calculation of MCSEs for RMSE ratios) Continuing from Exercise 10.1, compute the ratio of the RMSE of each estimator to the RMSE of the linear regression estimator. Use the jackknife technique to compute a MCSE for these RMSE ratios.
Exercise 10.3 (Calculate simplifications of the MCSEs) When calculating the monte-carlo standard error of the RMSE, we assumed \(T \sim N( \theta, \sigma^2 )\) and then stated that \(Var[ (T-\theta)^2 ] = 2\sigma^2\).
Derive this expression.
Now assume \(T \sim N( \theta + b, \sigma^2 )\), i.e., that \(T\) has a bias. What does \(Var[ (T-\theta)^2 ]\) equal now?
Using (b), simplify the expression for the MCSE of the RMSE in terms of kurtosis, variance, and bias parameters.
Expand \(Var[ (T-\theta)^2 ]\) for any \(T\) with defined moments up to the fourth moment. Show that the MCSE of \(c \times S_T / \sqrt{R}\) can only have a scaling constant larger than 1 if there is substantial skew in \(T\).
Boos, Dennis D., and Jason A. Osborne. 2015. “Assessing Variability of Complex Descriptive Statistics in Monte Carlo Studies Using Resampling Methods.”International Statistical Review 83 (2): 228–38. https://doi.org/10.1111/insr.12087.
Efron, Bradley, and Robert J Tibshirani. 1994. An Introduction to the Bootstrap. Chapman; Hall/CRC.
Hettmansperger, Thomas P., and Joseph W. McKean. 2010. Robust Nonparametric Statistical Methods. 2nd ed. Chapman & Hall / CRC Monographs on Statistics & Applied Probability. Taylor and Francis.
Maronna, Ricardo A., R. Douglas Martin, and Vı́ctor J. Yohai. 2006. Robust Statistics: Theory and Methods. Wiley Series in Probability and Statistics. J. Wiley.
McKean, Joseph W., and Ronald M. Schrader. 1984. “A Comparison of Methods for Studentizing the Sample Median.”Communications in Statistics - Simulation and Computation 13 (6): 751–73. https://doi.org/10.1080/03610918408812413.
To be precise, the formulas that we give are estimators for the Monte Carlo standard errors of the performance measure estimators. Our presentation does not emphasize this point because the performance measures will usually be estimated using a large number of replications from an independent and identically distributed process, so the distinction between true and estimated Monte Carlo standard errors will not be consequential. One could write \(MCSE\) and \(\widehat{MCSE}\) to distinguish between the two, but we avoid this notation for simplicity.↩︎
Oehlert (1992) provides a friendly introduction to the delta method, or see Pustejovsky (2018). The delta method approximation says (with some conditions), that if we assume \(X \sim N\left( \phi, \sigma_X^2 \right)\), then we can approximate the distribution of \(g(X)\) for some continuous function \(g(\cdot)\) as \[ g(X) \sim N\left( g(\phi), \;\; g'(\phi)^2 \times \sigma_X^2 \right),\] where \(g'(\phi)\) is the derivative of \(g(\cdot)\) evaluated at \(\phi\). In practice, \(\phi\) is an unknown parameter, but we can estimate the variance of \(g(X)\) by substituting \(X\) in place of \(\phi\). Thus, an approximation of the variance of \(g(X)\) is \(g'(X)^2 \times \sigma_X^2\). If \(g(X)\) is an estimator, this means an approximate SE for \(g(X)\) is \[ SE( g(X) ) \approx \left| g'(\mu_X) \right| \times SE(X).\] To find the MCSE of \(RMSE\), then, simply apply this delta method approximation to \(X = MSE\) with \(g(x) = \sqrt{x}\) and \(g'(x) =\frac{1}{2\sqrt{x}}\).↩︎