Risk of Bias Systematic Review: A Full Guide

Risk of bias in a systematic review is a domain-based evaluation of whether systematic errors in study design, conduct, or reporting may distort results. Unlike quality scoring, risk of bias assessment examines specific domains, selection, performance, detection, attrition, and reporting bias, using validated tools such as RoB 2 (for RCTs), ROBINS-I (for non-randomized studies), and the Newcastle-Ottawa Scale (for observational studies).

Every systematic review must include a formal risk of bias assessment of its included studies. The Cochrane Handbook (Higgins et al., 2023, Chapter 8) mandates domain-based evaluation because a single high-risk domain can invalidate an otherwise well-conducted study. Whether you are reviewing randomized controlled trials, cohort studies, or qualitative research, selecting the correct bias assessment tool and applying it consistently determines whether your review's conclusions can be trusted. Need to assess risk of bias? Use our free online risk of bias tool for randomized controlled trials or ROBINS-I assessment tool for non-randomized studies.

Risk of Bias Versus Reporting and Methodological Quality

Risk of bias is not the same as reporting quality or methodological quality. Reporting quality evaluates whether a study describes what was done (assessed by tools like CONSORT or STROBE). Methodological quality is a broader, less precise term that can encompass everything from study design to statistical analysis. Risk of bias specifically asks: did systematic errors in the study's design, conduct, or analysis produce results that deviate from the truth?

The Cochrane Handbook identifies five core bias domains that apply across study designs:

• Selection bias, systematic differences between groups at baseline, caused by flawed randomization or allocation concealment
• Performance bias, systematic differences in care provided to groups, typically caused by lack of blinding of participants and personnel
• Detection bias, systematic differences in how outcomes are measured, caused by lack of blinding of outcome assessors
• Attrition bias, systematic differences in withdrawals and dropouts between groups, leading to incomplete outcome data
• Reporting bias, systematic differences between reported and unreported results, including selective outcome reporting

Each bias domain is assessed independently because problems in one domain can completely undermine a study's findings regardless of how well other domains were handled.

Why Risk of Bias Assessment Matters

Risk of bias assessment directly determines whether your systematic review's conclusions are credible. Without it, readers and guideline developers cannot judge whether the studies supporting your findings are trustworthy.

Three authoritative frameworks require formal risk of bias evaluation. The Cochrane Handbook mandates it for all Cochrane reviews and provides detailed guidance on tool selection and application. PRISMA 2020 (Page et al., 2021) requires authors to describe the risk of bias methods used and present results for each included study. Most peer-reviewed journals now reject systematic reviews that lack risk of bias assessment, making it a practical requirement for publication.

Risk of bias is the first domain evaluated in the GRADE framework for assessing certainty of evidence. When studies contributing to a body of evidence have high risk of bias, GRADE downgrades the certainty of evidence, potentially moving it from "high" to "moderate" or lower. This directly affects whether clinical guidelines classify a recommendation as strong or conditional.

Choosing the Right Risk of Bias Tool

Which risk of bias tool should I use? Use RoB 2 for randomized controlled trials, ROBINS-I for non-randomized studies of interventions, Newcastle-Ottawa Scale for observational cohort and case-control studies, and JBI checklists for qualitative or other study types.

The decision depends entirely on the study design of your included studies. Many systematic reviews include multiple study designs, requiring you to use more than one tool. The table below maps each tool to its intended use:

RoB 2 -- For Randomized Controlled Trials

RoB 2 (revised Cochrane risk of bias tool) is the standard for assessing randomized controlled trials. It replaced the original Cochrane risk of bias tool in 2019 and provides a more structured, signaling-question-based approach. RoB 2 assesses risk of bias in randomized controlled trials across five domains, each evaluated through a series of signaling questions that guide the assessor to a domain-level judgment.

ROBINS-I -- For Non-Randomized Studies of Interventions

ROBINS-I (Risk Of Bias In Non-randomized Studies of Interventions) was developed for studies that compare health outcomes across groups but lack randomization. ROBINS-I assesses risk of bias in non-randomized studies by evaluating seven domains organized into pre-intervention, at-intervention, and post-intervention categories. Because non-randomized studies lack the built-in protection against confounding that randomization provides, ROBINS-I is necessarily more complex than RoB 2 and demands careful consideration of each study's analytical approach to confounding control. For a full walkthrough, see our ROBINS-I assessment guide.

Newcastle-Ottawa Scale -- For Observational Studies

The Newcastle-Ottawa Scale is widely used for cohort and case-control studies. It is simpler than ROBINS-I and uses a star-based scoring system rather than domain-level judgments. NOS is particularly common in public health and epidemiology reviews where observational designs predominate. Its ease of use makes it accessible to reviewers without extensive methodological training, though this simplicity comes at the cost of less detailed domain-level insight compared to ROBINS-I.

JBI Checklists -- For Qualitative and Other Study Types

The JBI Checklist (Joanna Briggs Institute) provides JBI critical appraisal tools for study types that RoB 2, ROBINS-I, and NOS do not cover. These include qualitative studies, cross-sectional studies, prevalence studies, case reports, and diagnostic accuracy studies. Each checklist is tailored to the methodological features of its target study design. JBI checklists use a Yes/No/Unclear/Not applicable format and are freely available from the JBI website, making them the most versatile option for mixed-methods systematic reviews.

Assessing observational studies? Use our Newcastle-Ottawa Scale calculator or JBI critical appraisal tool.

RoB 2 -- Domain-by-Domain Overview

RoB 2 evaluates five domains for each RCT, with each domain containing signaling questions that lead to a structured judgment. Cochrane Handbook recommends domain-based risk of bias assessment over summary quality scores, because a single high-risk domain can invalidate an otherwise well-conducted study (Higgins et al., 2023, Chapter 8).

Domain 1: Randomization process, This domain evaluates whether the allocation sequence was truly random and whether allocation concealment was adequate. Signaling questions ask whether the sequence was generated using a validated method (computer-generated, random number tables) and whether participants and recruiters could foresee assignments. Inadequate randomization introduces selection bias at baseline.

Domain 2: Deviations from intended interventions, This domain assesses whether participants, caregivers, or study personnel were aware of intervention assignments and whether any deviations from the protocol occurred. Blinding of participants and personnel is the primary safeguard. The domain also considers whether an intention-to-treat analysis was used, which preserves the benefits of randomization even when protocol deviations occur.

Domain 3: Missing outcome data, This domain evaluates whether outcome data were available for all or nearly all randomized participants. High dropout rates, differential attrition between groups, or inappropriate handling of missing data can introduce attrition bias. The signaling questions assess both the proportion of missing data and whether missingness is likely related to the outcome.

Domain 4: Measurement of the outcome, This domain examines whether the method of outcome measurement was appropriate and whether outcome assessors were blinded to intervention status. Subjective outcomes (pain scales, clinician-rated scores) are more vulnerable to detection bias than objective outcomes (mortality, laboratory values).

Domain 5: Selection of the reported result, This domain evaluates whether the reported results were selected from multiple eligible analyses of the data. Reporting bias includes switching primary outcomes, selecting favorable subgroup analyses, or choosing between multiple measurement time points. Pre-registration of protocols (e.g., on ClinicalTrials.gov or PROSPERO) is strong evidence against selective reporting.

Each domain receives a judgment of Low risk, Some concerns, or High risk, reached by answering the domain's signalling questions (Yes, Probably yes, Probably no, No, or No information) and following the tool's algorithm. The overall judgment is Low only when every domain is Low, and it is High when any domain is High. The shorthand "overall follows the worst domain" misses one rule: the RoB 2 guidance (version of 22 August 2019, riskofbias.info) also allows an overall High judgment when a trial raises Some concerns in multiple domains in a way that substantially lowers confidence in the result.

Worked Example: One Trial Assessed With RoB 2

Suppose a review includes a hypothetical trial that randomized 240 adults with chronic low back pain 1:1 to a supervised exercise program or usual care. The result being assessed is self-reported pain on a 0 to 10 numerical rating scale at 12 weeks, and the review team's aim is the effect of assignment to the intervention. The trial report describes a computer-generated sequence, allocation through a central web system, similar baseline characteristics, no crossover between groups, every participant analysed in the group they were randomized to, and a statistical analysis plan finalized before unblinding. Pain data were available for 108 of 120 participants in the exercise group (90.0%) and 96 of 120 in usual care (80.0%), so 204 of 240 overall (85.0%), with no sensitivity analysis for the missing data. Participants knew their allocation and rated their own pain.

Two answers decide most of this assessment. In Domain 3, the algorithm gives Some concerns because missingness could depend on pain level but probably did not; had the answer to 3.4 been Probably yes, the domain would be High. Domain 4 works the same way: if participants held strong beliefs about exercise, 4.5 could reasonably be answered Probably yes, which makes the domain High. Write the reason for each of these answers into the support-for-judgment column, because they are the answers a second reviewer is most likely to dispute.

The overall default is Some concerns, because no domain is High and two raise concerns. The reviewers still have to decide whether concerns in two domains substantially lower confidence in this pain result. If they judge that they do, the overall judgment becomes High. Record that reasoning in the risk of bias table, because it decides whether the trial stays in a sensitivity analysis limited to low risk of bias trials, and it feeds the risk of bias domain of GRADE for the pain outcome.

Need comprehensive risk of bias assessment for your systematic review? Our methodologists apply validated tools (RoB 2, ROBINS-I, Newcastle-Ottawa) with dual-reviewer calibration and transparent reporting. get a free ballpark research estimate, or see our professional systematic review support.

ROBINS-I -- Domain-by-Domain Overview

ROBINS-I evaluates seven bias domains organized into three temporal categories that reflect when the bias could have been introduced relative to the intervention. ROBINS-I assesses risk of bias in non-randomized studies through a more granular framework than RoB 2, because non-randomized designs are inherently more vulnerable to confounding and selection effects.

Pre-intervention domains:

• Bias due to confounding, Were important prognostic factors balanced between groups? Were appropriate statistical methods used to control for confounders?
• Bias in selection of participants, Was selection into the study or into the intervention/comparator group related to both the intervention and the outcome?

At-intervention domain:

• Bias in classification of interventions, Was intervention status well defined and determined prospectively?

Post-intervention domains:

• Bias due to deviations from intended interventions, Were there deviations that reflected differences between groups? Were co-interventions balanced?
• Bias due to missing data, Were outcome data complete? Was attrition related to intervention status or prognostic factors?
• Bias in measurement of outcomes, Were outcome assessors blinded? Were measurement methods comparable across groups?
• Bias in selection of the reported result, Were results selected from multiple analyses?

ROBINS-I uses a five-level judgment scale: Low risk, Moderate risk, Serious risk, Critical risk, and No information. The scale is intentionally more granular than RoB 2 because non-randomized studies face more heterogeneous threats to validity. A judgment of "critical" risk means the study is too problematic to provide any useful evidence for the review question.

Newcastle-Ottawa Scale -- Scoring Guide

The Newcastle-Ottawa Scale evaluates observational study quality through a star-based system with a maximum of 9 stars across three categories. NOS evaluates observational study quality using a simpler approach than ROBINS-I, which is both its strength (ease of use) and its limitation (less granular assessment).

Selection (maximum 4 stars):

• Representativeness of the exposed cohort (or case definition for case-control studies)
• Selection of the non-exposed cohort (or control selection)
• Ascertainment of exposure
• Demonstration that outcome of interest was not present at start of study (cohort) or same method of ascertainment for cases and controls

Comparability (maximum 2 stars):

• Comparability of cohorts (or cases and controls) on the basis of design or analysis, this category awards up to 2 stars for controlling the most important confounders

Outcome (maximum 3 stars):

• Assessment of outcome (or exposure for case-control)
• Sufficient follow-up length
• Adequacy of follow-up (low dropout rate)

Common thresholds classify NOS scores as: Good quality (7-9 stars), Fair quality (4-6 stars), Poor quality (0-3 stars). However, these cutoffs are not officially validated by the tool developers, and Cochrane advises caution when using summary scores for any risk of bias tool.

When should you use NOS instead of ROBINS-I? NOS is appropriate when your review includes observational studies that are not studying interventions, for example, studies examining risk factors, prognosis, or prevalence. ROBINS-I is specifically designed for studies comparing interventions and should be preferred in that context. For a detailed walkthrough, see our Newcastle-Ottawa Scale guide.

How to Present Risk of Bias Results

Risk of bias results should be presented using visual displays (traffic light plots and summary bar charts), narrative description, and integration into sensitivity analyses. The systematic review follows PRISMA 2020 reporting guidelines for risk of bias presentation, which require both a description of methods used and study-level results.

Traffic light plots display individual domain judgments for each included study. Each cell shows a colored circle, green (low risk), yellow (some concerns/moderate), red (high risk/serious/critical), creating an at-a-glance visual summary. Traffic light plots are the standard output from RoB 2 and ROBINS-I and can be generated using the robvis R package or Cochrane's RevMan software.

Summary bar charts show the proportion of studies at each risk of bias level for each domain. These plots answer the question: across all included studies, which domains are most problematic? A bar chart showing that 80% of studies have high risk of bias in the "blinding" domain tells a different story than one where risk is evenly distributed.

Summary tables provide a compact presentation of domain-level judgments across all studies. For reviews with many included studies, tables may be placed in supplementary materials with the traffic light plot in the main text.

Sensitivity analysis by risk of bias is a critical analytical step. Always repeat your primary meta-analysis excluding high-risk-of-bias studies. If the pooled effect changes substantially, this indicates that the overall result may be driven by biased studies, a finding that must be reported in the Discussion section. Sensitivity analysis tests result robustness and is one of the most informative analyses in any systematic review. For detailed data handling, see our guide on data extraction best practices.

Common Risk of Bias Assessment Mistakes

Five errors undermine risk of bias assessments more than any others, and each one threatens the validity of your systematic review's conclusions.

Using the wrong tool for the study design. Applying RoB 2 to a cohort study or Newcastle-Ottawa Scale to a randomized trial produces meaningless results. RoB 2 is designed exclusively for RCTs. ROBINS-I is for non-randomized studies of interventions. NOS covers observational studies. JBI checklists handle qualitative and other designs. If your review includes mixed designs, you must use multiple tools.

Treating risk of bias as a quality score. Risk of bias assessment is domain-based assessment, not a summation exercise. Averaging domain judgments into a single number obscures critical information. A study with "low risk" in four domains and "high risk" in one domain is fundamentally different from a study with "some concerns" across all domains, but both might receive similar numeric scores. Cochrane explicitly recommends against summary scores.

Single-reviewer assessment. Having only one person assess risk of bias introduces subjectivity and reduces reliability. Cochrane recommends at least two independent reviewers assessing every study, with disagreements resolved through discussion or a third reviewer. In practice, inter-rater agreement for risk of bias assessment ranges from fair to moderate (kappa 0.40-0.60), which underscores why dual assessment is essential.

Not conducting sensitivity analysis by risk of bias level. Completing risk of bias assessment without using the results in your analysis is a missed opportunity. Sensitivity analysis by risk of bias, repeating the meta-analysis with and without high-risk studies, reveals whether your conclusions depend on potentially biased evidence.

Applying RoB 2 to non-randomized studies. RoB 2 was designed for trials with randomization. Applying it to non-randomized studies means you are not evaluating the most important sources of bias (confounding, selection into intervention groups). Use ROBINS-I instead. For guidance on the correct tool, see our ROBINS-I assessment guide.

How Risk of Bias Feeds into GRADE

Risk of bias is the first of five domains in the GRADE framework for assessing certainty of evidence, and it is often the domain with the most impact on the overall certainty rating. The GRADE framework assesses certainty of evidence across risk of bias, inconsistency, indirectness, imprecision, and publication bias.

When the majority of studies contributing to an outcome have high risk of bias, GRADE recommends downgrading the certainty of evidence by one level (e.g., from "high" to "moderate"). If risk of bias is particularly severe or pervasive, downgrading by two levels may be warranted. This downgrade directly affects clinical recommendations, a body of evidence rated "low certainty" supports only conditional (weak) recommendations, regardless of the size of the effect estimate.

The connection between risk of bias and GRADE means that your domain-level judgments have consequences beyond the systematic review itself. Guideline panels, health technology assessment agencies, and clinical decision-makers all use GRADE certainty of evidence ratings to determine how much confidence to place in your findings. Flawed risk of bias assessment, whether too lenient or too strict, propagates through to every downstream decision.

For Cochrane reviews, risk of bias results feed directly into the Summary of Findings table, which presents the GRADE assessment for each outcome. This table is often the most-read component of a Cochrane review, making accurate risk of bias assessment essential to its credibility. For a complete walkthrough of how GRADE works, see our GRADE framework guide.

Research Gold's systematic review service includes expert risk of bias assessment using the correct tool for your study designs, with results integrated into GRADE certainty ratings and presented in publication-ready Summary of Findings tables. For a comprehensive overview of the entire systematic review process, see our complete systematic review guide explained.

Tool selection and rigor expectations vary by framework. See how risk of bias assessment differs in Cochrane vs non-Cochrane reviews.

Once assessments are complete, you need clear visualizations. Our guide to creating risk of bias summary charts covers traffic-light tables and bar charts for publication.

Choosing the right tool depends on your study designs. See our head-to-head comparison of RoB 2, NOS, JBI, and GRADE with a decision flowchart.