Quality assessment is not optional in a systematic review. Every included study must be evaluated for methodological rigor, and the tools you choose must match the study designs in your review. Selecting the wrong instrument can undermine your entire evidence synthesis and invite major revisions from peer reviewers. This guide provides a thorough side-by-side comparison of seven quality assessment tools you are most likely to encounter: RoB 2, the Newcastle-Ottawa Scale (NOS), JBI checklists, GRADE, QUADAS-2, AMSTAR 2, and ROBINS-I.
All of these are available as free online instruments on Research Gold: RoB 2 assessment tool, Newcastle-Ottawa Scale, JBI Critical Appraisal, GRADE Evidence Tool, and ROBINS-I Tool.
Get both tools as ready worksheets in the RoB 2 and Newcastle-Ottawa templates.
What These Tools Actually Measure
Understanding what each tool evaluates prevents the most common mistake in quality assessment: using the wrong instrument for your study design.
RoB 2 and NOS measure study-level risk of bias, meaning how likely it is that the methods introduced systematic error. ROBINS-I also measures study-level bias but targets non-randomized studies of interventions with far more granularity than NOS.
JBI checklists measure methodological quality, especially for qualitative and mixed-methods research where bias operates differently. QUADAS-2 measures risk of bias and applicability concerns in diagnostic test accuracy studies. AMSTAR 2 evaluates the methodological quality of other systematic reviews, making it essential for umbrella reviews.
GRADE measures evidence certainty at the outcome level across a body of studies. Risk of bias findings from the other tools feed into GRADE as one of five domains. GRADE is never a replacement for study-level assessment. It builds on top of it.
Mixing these up creates serious problems. NOS is an input to GRADE, not a substitute. QUADAS-2 cannot replace RoB 2 for intervention studies. Reviewers will flag misapplication immediately.
RoB 2: The Standard for Randomized Controlled Trials
Developed by Sterne et al. (2019) and endorsed by Cochrane, RoB 2 evaluates five domains: randomization process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of the reported result. Each domain receives a judgment of Low risk, Some concerns, or High risk.
The current version uses structured signaling questions within each domain to guide assessors, improving inter-rater reliability substantially over the original Cochrane tool.
When to use RoB 2: Any systematic review that includes randomized controlled trials. Mandatory for Cochrane reviews. Use a separate tool for any non-randomized studies.
Use our free Risk of Bias Tool for structured assessments, or read the full RoB 2 assessment guide for randomized controlled trials.
Newcastle-Ottawa Scale: Cohort and Case-Control Studies
Developed by Wells et al., NOS uses a star-rating system across three categories: Selection, Comparability, and Outcome/Exposure. Maximum score is 9 stars. Common thresholds: 7+ high quality, 5-6 moderate, below 5 low.
Key limitation: the star system creates a false sense of precision. NOS is best used for domain-level comparison rather than total-score ranking.
NOS versus ROBINS-I: NOS is faster (10 to 15 minutes) and simpler. ROBINS-I is more thorough (30 to 60 minutes) and preferred when you need granularity comparable to RoB 2.
Our Newcastle-Ottawa Scale tool calculates star ratings automatically.
JBI Checklists: Qualitative, Cross-Sectional, and Mixed-Methods
The Joanna Briggs Institute provides 13 critical appraisal checklists covering designs that NOS and RoB 2 do not handle: qualitative research, cross-sectional studies, case reports, case series, and prevalence studies.
Each item is rated Yes, No, Unclear, or Not applicable with no aggregate score. JBI avoids numerical scoring because a single number cannot capture the multidimensional nature of methodological quality in qualitative research.
When to use JBI: Studies outside RoB 2 and NOS scope. Especially valuable for mixed-methods systematic reviews and scoping reviews.
Our JBI Critical Appraisal tool covers all major checklist types and exports results for your appendices.
QUADAS-2: Diagnostic Test Accuracy Studies
Developed by Whiting et al. (2011), QUADAS-2 is the standard tool for diagnostic test accuracy studies. It has four domains: patient selection, index test, reference standard, and flow and timing. Each domain is assessed for risk of bias, and the first three are also assessed for applicability concerns.
Diagnostic accuracy studies have unique bias sources: verification bias, spectrum bias, and incorporation bias all require signaling questions that RoB 2 does not contain.
When to use QUADAS-2: Reviews comparing imaging modalities, laboratory tests, screening instruments, and AI-based diagnostic algorithms.
AMSTAR 2: Assessing the Quality of Systematic Reviews
Developed by Shea et al. (2017), AMSTAR 2 evaluates the methodological quality of systematic reviews themselves. It contains 16 items, of which 7 are critical domains. Failing on even one critical domain results in a Critically Low confidence rating. The four categories are High, Moderate, Low, and Critically Low.
When to use AMSTAR 2: Umbrella reviews, overviews of reviews, guideline development, and research proposals demonstrating weaknesses in prior reviews.
ROBINS-I: Non-Randomized Studies of Interventions
Developed by the Cochrane Bias Methods Group, ROBINS-I evaluates seven domains: confounding, participant selection, intervention classification, deviations from interventions, missing data, outcome measurement, and reported result selection.
Each domain is judged as Low, Moderate, Serious, Critical, or No information. The overall judgment takes the worst domain rating. Cochrane reviews now require ROBINS-I for non-randomized studies rather than NOS.
Use our free ROBINS-I Tool for structured assessments with all seven domains.
