Simultaneous Confidence Bands: Theory, Implementation, and an Application to SVARs

It answers the question of which confidence band to report by default when a researcher needs to cover an entire vector of estimated quantities simultaneously – for example, a structural impulse-response function across many horizons – rather than reporting separate pointwise intervals that only cover each parameter one at a time. The paper works in a general framework where a possibly nonlinear transformation theta = h(mu) of an asymptotically normal underlying estimator mu-hat (mu-hat satisfying root-n(mu-hat - mu) converging to a normal distribution) is the object to be covered; a simultaneous 1 - alpha confidence band is a Cartesian product of intervals, one per component of theta, that jointly covers the whole vector with asymptotic probability at least 1 - alpha. The framework nests VAR impulse-response analysis as a leading case, since impulse responses at many horizons are nonlinear transformations of a finite set of underlying VAR coefficients, and the transformation’s Jacobian can even make the asymptotic covariance singular when there are more horizons of interest (k) than underlying parameters (p).

The authors show that pointwise, Bonferroni, Sidak, projection, and sup-t bands are all special cases of a single “one-parameter class,” in which every component interval is the pointwise estimate plus-or-minus its own standard error times one common scalar critical value c. Formally, the band is the product of intervals [theta-hat_j - sigma-hat_j * c, theta-hat_j + sigma-hat_j * c] for j = 1,…,k, where sigma-hat_j is the delta-method pointwise standard error of theta-hat_j; the only thing that distinguishes the named procedures is which value of c they use (e.g., a chi-square pointwise critical value, a Bonferroni-adjusted one using alpha/k, or a Sidak-adjusted one using (1-alpha)^(1/k)). Because every candidate in Table 1 fits this one template, the choice among them reduces to the choice of critical value, which the paper’s ordering result (Section 3.3) ranks as sup-t <= Sidak <= Bonferroni, with the theta-projection and mu-projection bands typically wider still for alpha <= 0.5.

The sup-t band uses, as its critical value c, the (1 - alpha) quantile of the maximum absolute studentized coordinate of the estimator’s joint asymptotic normal distribution – the smallest c consistent with exact asymptotic simultaneous coverage – which by construction makes it the narrowest member of the one-parameter class that still achieves the nominal coverage. Its asymptotic coverage probability equals P(max_j |Sigma_jj^(-1/2) V_j| <= c) for V jointly normal with the estimator’s asymptotic covariance Sigma, evaluated at c = q_{1-alpha}(Sigma), the (1-alpha) quantile of that maximum. The gain over Bonferroni/Sidak comes specifically from exploiting the correlation across the theta-hat_j’s – the paper notes the sup-t advantage is largest exactly when components are highly correlated, “as is typical for impulse responses at adjacent horizons” – and the sup-t band’s width grows only at the slow rate of the square root of log(k) in the number of parameters k, whereas the projection bands’ width is far more sensitive to k.

The paper gives two implementations that are asymptotically equivalent but suit different settings: a “plug-in” simulation algorithm using the estimated asymptotic covariance, and a bootstrap-or-Bayes algorithm that draws directly from the bootstrap or posterior distribution of the underlying parameter. In the plug-in version (Algorithm 1), the researcher computes the delta-method covariance estimate Sigma-hat = h-dot(mu-hat) * Omega-hat * h-dot(mu-hat)’, simulates many draws from N(0, Sigma-hat), and takes the empirical (1-alpha) quantile of the maximum studentized draw as the critical value. In the bootstrap/Bayes version (Algorithm 2), the researcher draws many mu-hat replications from the bootstrap or posterior distribution, maps each through h(.) to get a theta-hat replication, and calibrates a common quantile-trimming level so that the resulting rectangle of equal-tailed intervals contains at least fraction 1-alpha of the draws; the resulting Bayesian band has exact finite-sample simultaneous Bayesian credibility (up to simulation error), while the plug-in band is not guaranteed to deliver an asymptotic refinement over it (footnote 11).

The paper offers a decision-theoretic justification (Section 4, Proposition 1): among all translation-equivariant simultaneous confidence bands, the sup-t band is the unique minimizer of worst-case regret, where regret is measured relative to the best possible band under any degree-one-homogeneous loss function applied to the vector of interval lengths. Because different readers of a published impulse-response figure might care about different features of the band – its width at one particular horizon, its width averaged across horizons, and so on – a band that is optimal only for one specific loss function could perform poorly for another reader’s implicit preferences. The sup-t band’s worst-case-regret optimality means it never performs arbitrarily worse than the best band tailored to any such loss function, which the authors offer as the argument for treating it as the default choice absent more specific information about how the band will be used; for a fully specified known loss function, the authors note that Freyberger and Rai’s (2018) numerically optimal band could instead be preferred (Section 4.2.4).

Section 5 replicates a Gertler-Karadi (2015)-style monthly U.S. SVAR – industrial production, CPI, a one-year government bond yield, and the excess bond premium (Gilchrist-Zakrajsek 2012), estimated in levels with 12 lags over July 1979-June 2012 (external-instrument data from January 1990), with p = 206 underlying VAR/instrument parameters and k = 37 horizons of interest (0-36 months) – and constructs confidence bands for the impulse responses to a monetary policy shock under two identification schemes. The first is a recursive/Cholesky scheme (industrial production and CPI do not respond to the monetary shock on impact; the bond yield does not respond to the excess bond premium on impact); the second is an external-instrument scheme using FF4, a three-month-ahead federal funds futures surprise measured in a short window around FOMC announcements. This design lets the paper compare sup-t, Bonferroni, Sidak, and other bands on the exact kind of object – a multi-horizon impulse-response function – that motivated the theory.

Yes – the choice of band changes which horizons show a statistically significant output response. The sup-t band is roughly 35% narrower than the Sidak and Bonferroni bands in the external-instrument specification at 68% confidence. At that 68% simultaneous confidence level, the plug-in sup-t band excludes zero for the industrial-production response at horizons of roughly 13-36 months, which lets the authors reject, at some horizon in that range, the null that the monetary shock has no effect on output; the Bonferroni band, being wider, does not permit that rejection at the same nominal level. The comparison also runs the other way: a positive industrial-production response that looks pointwise statistically significant at the 2-month horizon loses its significance once the sup-t band’s simultaneous multiple-comparison adjustment is applied, illustrating that pointwise significance and simultaneous significance are genuinely different standards. Comparing the three sup-t implementations (plug-in, bootstrap, Bayes) shows they are similar at short horizons but diverge somewhat at longer horizons under the recursive scheme, with the authors suggesting the Bayesian band may be preferable when the data are highly persistent.

A simulation study of bivariate VARs under recursive or external-instrument identification finds the sup-t band 20-25% narrower than the Bonferroni/Sidak bands at 68% confidence, and 10-20% narrower at 90% confidence, with projection bands described as unnecessarily wide throughout. The one qualification concerns persistence: for highly persistent simulated data, only the Bayesian sup-t implementation is reported to deliver satisfactory finite-sample simultaneous coverage, suggesting the plug-in and bootstrap versions can be less reliable specifically in that regime even though all three are asymptotically equivalent.

The authors note an extension to “generalized familywise error rate control,” in which the band is only required to cover at least k - m of the k parameters (rather than all k), obtained by replacing the maximum operator in Algorithm 2 with the (m+1)-th order statistic – a less stringent, multiple-testing-style coverage criterion (Section 6.1). On limitations: the core theory is developed for point-identified models with a continuously differentiable transformation h(.); it does not directly cover partially identified models such as sign-restriction SVARs, though the authors suggest the Bayesian sup-t band may still be applied there for subjective Bayesian analysis. They also flag that the sup-t band’s width, while growing only at the slow root-log(k) rate, does still grow with the number of parameters, and that the bootstrap implementation is not guaranteed to offer an asymptotic refinement over the plug-in version. Finally, for a researcher who does know the specific loss function relevant to their application, the worst-case-regret optimality of the sup-t band is not the same as being optimal for that one loss function, and a tailored alternative (Freyberger and Rai 2018) may do better.