Sensitivity Analysis Meta-Analysis: Leave-One-Out

Every meta-analysis rests on an implicit assumption: that no single study is so dominant that removing it would overturn the conclusions. Leave-one-out sensitivity analysis tests that assumption directly. By systematically omitting one study at a time and re-estimating the pooled effect, you produce a clear picture of how much each study drives your result. This procedure is one of the most requested sensitivity checks in peer review, and reviewers for journals indexed in PubMed, Cochrane, and PROSPERO routinely expect to see it reported.

When the pooled effect stays stable across all iterations, your conclusion is robust. When one omission shifts the estimate dramatically, you have an influential study that demands explanation. This guide walks through every aspect of leave-one-out analysis, from visual interpretation to advanced influence diagnostics, with a worked example you can replicate using our free robustness check tool or the metafor package in R.

What Makes a Study Influential

Three mechanisms create influential studies, and understanding them helps you explain your findings to reviewers rather than simply reporting numbers.

Precision leverage occurs when a study has an exceptionally small variance, giving it high statistical weight. A large randomized controlled trial in a pool of smaller observational studies will often dominate the pooled estimate simply because its confidence interval is narrow. Under a random-effects model, precision leverage is partially dampened by the between-study variance component (tau-squared), but it still matters.

Effect size discordance occurs when a study's point estimate is far from the other studies in the pool. A study reporting a standardized mean difference of 1.8 in a pool where the other nine studies range from 0.3 to 0.7 will pull the pooled estimate upward. When that study is removed, the pooled effect drops noticeably.

Heterogeneity contribution is more subtle: a study can inflate tau-squared, changing how all weights are distributed under a random-effects model. Removing such a study not only shifts the point estimate but also narrows the confidence interval for every remaining study because the between-study variance decreases. This is the mechanism most often missed by researchers who focus only on point estimates.

Try our free leave-one-out calculator to identify all three types of influential studies instantly.

How Leave-One-Out Analysis Works

For a meta-analysis with k studies: remove study 1, re-fit the model on k-1 studies, record the pooled effect, confidence interval, p-value, and heterogeneity statistics. Restore study 1, remove study 2. Continue until every study has been omitted once.

The output is a table or forest-style plot showing k rows of pooled estimates, each representing the analysis with one study excluded. In R, the metafor package (Viechtbauer, 2010) provides the leave1out() function that automates this procedure for any fitted rma() model object.

Running the Analysis with Our Free Tool

Navigate to the robustness check tool and enter your study labels, effect sizes, and standard errors. The tool runs the full leave-one-out procedure automatically and displays a leave-one-out forest plot with color coding for studies where exclusion changes significance.

The one-click export provides the equivalent R code using the leave1out() function from the metafor package.

How to Read a Leave-One-Out Forest Plot

A leave-one-out forest plot looks similar to a standard forest plot but communicates different information. Each row represents the pooled result after excluding one study, not the individual study's effect. Here is how to read it systematically.

The vertical reference line marks the overall pooled estimate from the full model (all k studies included). Each row's point estimate shows where the pooled effect lands when that specific study is removed. If all point estimates cluster tightly around the reference line, the meta-analysis is stable.

Confidence intervals on each row represent the precision of the reduced-k model. Watch for rows where the confidence interval crosses the null (zero for mean differences, one for odds ratios) when the full model's interval did not. That crossing signals a significance change, the most important finding in leave-one-out analysis.

Color coding in our online sensitivity analysis highlights rows where removing a study changes statistical significance. Red rows indicate that excluding that study makes the result non-significant. Green rows (less common) indicate that excluding a study strengthens significance by removing heterogeneity.

Use the interactive forest plot builder alongside the leave-one-out plot to compare each study's individual effect against the pooled estimate and visually identify which studies are discordant.

Interpreting Results

When Results Are Robust

If all k pooled estimates cluster tightly around the original estimate and confidence intervals consistently include or exclude zero across all iterations, your conclusion is robust. Report this explicitly. Reviewers value a clear statement such as: "Leave-one-out sensitivity analysis confirmed that no single study altered the direction, magnitude, or statistical significance of the pooled effect."

When One Study Changes Significance

Investigate the influential study along four dimensions: sample size and population, methodological quality, effect size and direction, and publication context.

After investigation, present both the full-model and leave-one-out estimates as co-primary results, or conduct a subgroup analysis separating the outlying study. Document whether the influential study differed in risk of bias, population characteristics, intervention dosage, or follow-up duration.

Use the forest plot software to visualize the study's position relative to the rest of the pool.

When Multiple Studies Are Influential

If removing several different studies each changes significance, the meta-analysis has a fragility problem. Report it honestly and recommend further primary research. Calculate the fragility index to quantify how many studies would need to change their event counts to reverse the result, and present this alongside the leave-one-out findings.

Significance Change Detection

Two criteria define a meaningful change: a clinically meaningful shift in the point estimate (a change exceeding the minimally important difference for the outcome), and a change in statistical significance (confidence interval changes from excluding to including the null, or vice versa).

The interactive funnel plot builder helps identify outliers before running leave-one-out analysis, giving you a visual preview of which studies sit far from the regression line.

When Leave-One-Out Analysis Is Insufficient

Leave-one-out analysis answers one question: what happens to the pooled effect when each study is removed individually? It does not capture jointly influential study pairs, and it does not quantify the specific mechanism of influence. For a deeper diagnosis, you need formal influence diagnostics.

Cook's Distance

Cook's distance measures how much all fitted values change when a study is removed. It combines leverage (how unusual a study's precision is) with residual magnitude (how far the study's effect is from the model prediction). A study with a high Cook's distance is influential through both mechanisms simultaneously. In the metafor package, this is available through the cooks.distance() method on a fitted rma() object.

DFBETAS

DFBETAS quantifies how much each model coefficient changes (in standard error units) when a study is deleted. For a simple random-effects meta-analysis with one pooled estimate, DFBETAS tells you whether a study shifts the intercept by more than a threshold (commonly 2 divided by the square root of k). Studies exceeding this threshold are considered influential.

Hat Values (Leverage)

Hat values measure a study's leverage, which is its potential to influence the pooled estimate based on precision alone. A study with a hat value much larger than the average (which equals the number of coefficients divided by k) has high leverage. High leverage combined with a large residual produces influence. Hat values are available through the hatvalues() method in metafor.

DFFITS

DFFITS measures the change in the fitted value for a study when it is deleted, standardized by the study's standard error. It is the individual-study analog of Cook's distance and follows a threshold of 2 times the square root of the number of coefficients divided by k.

Baujat Plot as a Complement to Leave-One-Out

The Baujat plot (Baujat et al., 2002) provides a two-dimensional view of study influence that leave-one-out analysis alone cannot offer. The x-axis shows each study's contribution to the overall heterogeneity statistic (Q), and the y-axis shows each study's influence on the pooled effect.

Studies in the upper-right quadrant are the most problematic: they contribute substantially to heterogeneity and pull the pooled estimate. Studies in the lower-right quadrant contribute to heterogeneity but do not shift the pooled result much, suggesting they add noise without changing the conclusion. Studies in the upper-left quadrant shift the result despite not contributing much to heterogeneity, indicating precision leverage.

Read our detailed guide on Galbraith plots and Baujat plots for heterogeneity diagnostics for a full walkthrough of visual diagnostic strategies, including when to use each plot type.

To generate a Baujat plot in R with metafor, call baujat(model) on your fitted rma() object. Our meta-analysis sensitivity calculator also displays Baujat-style information when you run the full diagnostics panel.

Influence Diagnostics in metafor: The inf.rma() Function

The metafor package by Viechtbauer (2010) provides a comprehensive influence diagnostics framework through the influence() function (also accessible as inf.rma() in earlier versions). Viechtbauer and Cheung (2010) formalized these diagnostics for meta-analysis, adapting established regression diagnostics to the meta-analytic context.

Running a Full Influence Analysis

library(metafor)

# Fit random-effects model
res <- rma(yi, vi, data = dat)

# Run full influence diagnostics
inf <- influence(res)

# View all diagnostics in a single plot
plot(inf)

# Access individual measures
inf$inf      # rstudent, dffits, cook.d, cov.r, tau2.del, QE.del, hat, weight

The influence() function returns a structured object containing externally standardized residuals (rstudent), DFFITS, Cook's distance, covariance ratio (cov.r), tau-squared when deleted (tau2.del), Q-statistic when deleted (QE.del), hat values, and percentage weights. The plot() method displays all of these in a multi-panel diagnostic plot, making it easy to identify studies that are flagged across multiple measures.

Interpreting the Diagnostic Plot

The default plot from plot(influence(res)) shows eight panels. Look for studies that appear as outliers in multiple panels simultaneously. A study flagged only on hat values (high leverage but small residual) may not be problematic. A study flagged on rstudent, Cook's distance, and DFFITS simultaneously is almost certainly influential and warrants investigation.

The covariance ratio (cov.r) deserves special attention. Values below 1 indicate that removing the study decreases the variance of the pooled estimate, meaning the study was inflating uncertainty. Values well above 1 indicate that removing the study increases variance, meaning the study was stabilizing the estimate.

How to Handle an Influential Study

Identifying an influential study is only the first step. The Cochrane Handbook recommends a structured response rather than automatic exclusion.

Option 1: Retain and Report Transparently

If the influential study is methodologically sound and clinically relevant, keep it in the primary analysis. Report the leave-one-out result as a planned sensitivity analysis and discuss why the study differs. This is the most common and most defensible approach.

Option 2: Conduct Subgroup or Meta-Regression Analysis

If the influential study differs on a pre-specified characteristic (different population, different intervention dose, different comparator, higher risk of bias), conduct a subgroup analysis or meta-regression to test whether that characteristic explains the heterogeneity. This turns an influential outlier into a moderator finding, which is scientifically more valuable than simple exclusion.

Option 3: Present a Sensitivity Table

Create a sensitivity analysis table showing the pooled estimate under multiple scenarios: all studies included, influential study excluded, restricted to low risk-of-bias studies, and restricted to the pre-specified primary outcome only. This approach gives readers and reviewers the full picture without forcing a single analytic decision.

Option 4: Remove with Justification

Remove the influential study only if there is a substantive methodological reason (critical risk of bias, protocol deviation, wrong population) that justifies exclusion independently of the sensitivity analysis result. Never remove a study solely because it changes your result. Reviewers will flag this as outcome-driven exclusion.

If you need help deciding how to handle influential studies in your meta-analysis, our team provides expert guidance on meta-analysis methodology including full sensitivity and influence diagnostic reporting.

Reporting Leave-One-Out Results Under PRISMA 2020

The PRISMA 2020 statement includes specific items relevant to sensitivity analysis reporting. Here are the checklist items that apply directly.

Item 13a (Synthesis methods): Describe all sensitivity analyses planned in the protocol, including leave-one-out analysis. State whether the analysis was pre-specified or post-hoc.

Item 13d (Sensitivity analyses): Describe any sensitivity analyses conducted to assess robustness of the synthesized results. This is where you describe the leave-one-out procedure.

Item 22 (Results of syntheses): Present results of each synthesis, including leave-one-out forest plots or summary tables. For each influential study identified, report the change in pooled estimate, confidence interval, and heterogeneity.

Item 23d (Reporting biases): If leave-one-out analysis reveals that removing a single small study with an extreme effect changes the result, discuss whether this is consistent with publication bias or small-study effects.

In practice, reviewers expect a leave-one-out forest plot in the supplementary materials (or main text for high-impact findings) and a sentence in the results section stating whether any study altered the conclusion. Our guide on sensitivity analysis for systematic reviews covers the full reporting framework across all sensitivity analysis types.

Worked Example: Ten-Study Meta-Analysis with One Outlier

Consider a random-effects meta-analysis of ten randomized controlled trials examining the effect of a behavioral intervention on anxiety scores, measured as standardized mean differences.

Study Data

Study Evans reports an SMD of 1.45, far above the range of the other nine studies (0.29 to 0.55). The full-model pooled estimate is SMD = 0.52 (95% CI: 0.35 to 0.69, p < 0.001), with I-squared = 72%.

Leave-One-Out Results

Removing each study in turn produces these pooled estimates:

Interpretation

Removing Evans shifts the pooled SMD from 0.52 to 0.42, a clinically relevant reduction of 0.10 units. The direction and significance do not change, but I-squared drops from 72% to 18%, revealing that Evans was the primary source of heterogeneity. The influence() diagnostics confirm: Evans has the highest Cook's distance (2.31), the largest externally standardized residual (3.87), and sits in the upper-right quadrant of the Baujat plot.

The correct response here is Option 1 (retain and report transparently) combined with Option 2 (investigate why Evans differs). On closer examination, Evans used a higher intervention dose and a younger population, explaining the larger effect. A meta-regression with dose as a moderator would strengthen the analysis.

You can replicate this example using our free sensitivity analysis by entering the ten studies and viewing the leave-one-out forest plot, influence diagnostics, and Baujat plot.

Combining Leave-One-Out with Cumulative Meta-Analysis

Cumulative meta-analysis adds studies one at a time (usually in chronological order) and shows how the pooled estimate evolves as evidence accumulates. Combined with leave-one-out analysis, it answers two complementary questions: "Has the evidence stabilized over time?" (cumulative) and "Does any single study drive the current estimate?" (leave-one-out).

When a cumulative meta-analysis shows the estimate stabilizing after the sixth study but the leave-one-out analysis flags the second study as influential, you know that the early influential study has been "diluted" by subsequent evidence. This is a stronger robustness finding than either analysis alone.

In metafor, generate both analyses with cumul(res) and leave1out(res), then present them side by side. For a detailed guide on cumulative approaches, see our article on heterogeneity and I-squared in meta-analysis, which covers how cumulative analysis tracks heterogeneity over time.

Pre-Specified vs Post-Hoc Sensitivity Analysis

The distinction between pre-specified and post-hoc sensitivity analysis has important implications for how reviewers interpret your results and how much weight editors give them.

Pre-Specified (Protocol-Level)

Sensitivity analyses declared in a registered protocol (PROSPERO, OSF, or a published protocol paper) carry the highest credibility. State in your protocol that you will conduct leave-one-out sensitivity analysis and, if any study changes the pooled significance, you will investigate that study's characteristics and present both estimates. This pre-commitment prevents the appearance of data-driven analytic decisions.

Post-Hoc (Reviewer-Requested or Exploratory)

When reviewers request leave-one-out analysis during peer review, it becomes a post-hoc addition. This is perfectly acceptable, but label it clearly. Write: "At the request of a reviewer, we conducted a post-hoc leave-one-out sensitivity analysis to assess the robustness of the primary estimate." This transparency protects you from the criticism that you ran the analysis selectively.

Protocol Implications

If your protocol did not mention sensitivity analysis but you conducted one anyway, some reviewers may question whether you ran multiple analyses and reported only the favorable result. The solution is honesty: report all sensitivity analyses you performed, regardless of the result. If leave-one-out analysis showed the result was fragile, reporting that finding strengthens rather than weakens your paper because it demonstrates scientific integrity.

When Reviewers Request Additional Sensitivity Analyses

Reviewer requests for sensitivity analysis typically fall into predictable categories. Knowing what reviewers commonly ask for helps you prepare a comprehensive sensitivity analysis section during the initial submission, reducing revision rounds.

"Please conduct leave-one-out sensitivity analysis." This is the most common request. If you have already included it, point the reviewer to the relevant figure and table. If you have not, add it using the procedure described in this guide.

"Please assess whether the high risk-of-bias study drives the result." This is a targeted version of leave-one-out analysis. Present the leave-one-out result for that specific study and, if appropriate, a subgroup analysis restricted to low and moderate risk-of-bias studies.

"Please provide influence diagnostics beyond leave-one-out." This request calls for the full influence() panel from metafor: Cook's distance, DFBETAS, hat values, and externally standardized residuals. Present the multi-panel diagnostic plot in supplementary materials.

"Please justify retaining the outlier study." Cite the study's methodological quality, its relevance to the review question, and the pre-specified nature of the sensitivity analysis. Explain the mechanism behind its divergent result (different population, dose, or comparator) and present the meta-regression if applicable.

If you are facing challenging reviewer requests on your meta-analysis sensitivity analyses, get pricing for your project from our team for expert statistical support and reviewer response drafting.

Key Takeaways

• Leave-one-out sensitivity analysis omits one study at a time and re-estimates the pooled effect, producing k estimates that reveal each study's individual influence on the conclusion.
• Influential studies arise through precision leverage, effect size discordance, or disproportionate contribution to between-study variance (tau-squared).
• A leave-one-out forest plot should be read by comparing each row's confidence interval to the full-model reference line, watching for significance changes.
• When leave-one-out analysis is insufficient, use formal influence diagnostics: Cook's distance, DFBETAS, hat values, and the covariance ratio, all available through the influence() function in metafor (Viechtbauer & Cheung, 2010).
• The Baujat plot (Baujat et al., 2002) complements leave-one-out analysis by showing each study's contribution to both heterogeneity and the pooled effect on a single graph.
• Handle influential studies through transparent reporting, subgroup analysis, sensitivity tables, or justified exclusion, but never remove a study solely because it changes the result.
• Report leave-one-out results under PRISMA 2020 Items 13a, 13d, 22, and 23d, with clear labeling of pre-specified vs post-hoc analyses.
• Combining leave-one-out with cumulative meta-analysis provides complementary evidence: temporal stability plus individual study influence.
• Pre-specify sensitivity analyses in your protocol to maximize credibility and reduce reviewer pushback during peer review.
• Our free influence diagnostics tool runs the full leave-one-out procedure, flags significance changes, and generates publication-ready plots and R code.