JBI Critical Appraisal Tool

Critical appraisal is the systematic evaluation of research evidence to judge its trustworthiness, relevance, and applicability, and no single instrument can adequately assess every study design. The JBI critical appraisal tool, developed and maintained by the Joanna Briggs Institute (Aromataris & Munn, 2020), addresses this challenge by providing a suite of design-specific checklists that share a common scoring philosophy while tailoring their questions to the methodological features that matter most for each study type. The JBI suite covers randomized controlled trials, cohort studies, analytical cross-sectional studies, prevalence studies, case reports, case series, and, uniquely among major appraisal frameworks, qualitative research. This JBI checklist online currently includes the randomized controlled trial, cohort, analytical cross-sectional, and qualitative checklists. This breadth makes the JBI system indispensable for mixed-methods and comprehensive systematic reviews that synthesize evidence from heterogeneous study designs. The CASP (Critical Appraisal Skills Programme) checklists serve as a complementary appraisal framework that is particularly popular in the United Kingdom and in health services research, offering streamlined question sets for qualitative studies, cohort studies, and RCTs. For reviews that combine quantitative and qualitative evidence, the Mixed Methods Appraisal Tool (MMAT) developed by Hong et al. (2018) provides a single instrument capable of appraising all empirical study designs within one unified framework.

The qualitative study appraisal tool within the JBI framework deserves particular attention because qualitative evidence plays an increasingly important role in evidence synthesis. The JBI Checklist for Qualitative Research contains 10 items addressing the congruity between the stated philosophical perspective, the methodology, the research question, the data collection methods, the data analysis approach, and the interpretation of results. It also evaluates whether the researcher's cultural and theoretical positioning is clearly stated, a reflexivity assessment that is absent from quantitative appraisal tools and is essential for judging the credibility of interpretive research. When synthesizing qualitative findings across studies, the CERQual (Confidence in Evidence from Reviews of Qualitative Research) approach provides a structured method for rating the certainty of qualitative evidence, analogous to the GRADE framework used for quantitative reviews. For quantitative designs, the JBI RCT checklist assesses randomization, allocation concealment, blinding, and completeness of follow-up, while the cohort checklist evaluates group comparability, exposure measurement, and confounding control. Each item is answered as Yes, No, Unclear, or Not Applicable, and the proportion of Yes responses yields a summary quality score.

The JBI Manual for Evidence Synthesis (Aromataris & Munn, 2020) recommends that two independent reviewers complete the critical appraisal for every included study, with disagreements resolved by discussion or referral to a third reviewer. JBI does not define a cutoff score; the checklists end with an overall decision to include, exclude, or seek further information. Some review teams count Yes responses and set their own threshold, but that is their convention rather than JBI practice, and reviewers should look beyond any count and consider which specific items were rated No or Unclear. A study meeting most criteria but failing on allocation concealment may introduce more bias than one with more unmet criteria that are only minor reporting gaps. PRISMA 2020 (Page et al., 2021) requires transparent reporting of all critical appraisal results, and the Cochrane Handbook (Higgins et al., 2023) emphasizes that quality assessment findings should directly inform the interpretation of review results, including decisions about GRADE certainty of evidence ratings. SUMARI (JBI's System for the Unified Management, Assessment and Review of Information) integrates critical appraisal directly into the review management workflow, allowing teams to conduct JBI assessments within the same platform used for screening and extraction.

Selecting the most appropriate appraisal instrument requires matching the tool to the study design and the level of granularity needed. For randomized trials where domain-level bias judgments are required by journal or funder guidelines, the Cochrane RoB 2 assessment tool provides a five-domain evaluation with traffic light visualization. For non-randomized studies of interventions, the ROBINS-I bias assessment framework offers seven domains with signaling questions tailored to observational comparative designs. When a simpler star-based scoring system is preferred for cohort or case-control studies, the Newcastle-Ottawa Scale calculator provides a widely recognized alternative. The JBI checklists complement all of these tools by covering design categories that the others do not address, including qualitative research, prevalence studies, and case series, making them an essential component of any comprehensive systematic review toolkit. Once your appraisal is complete, record all quality scores alongside study characteristics in your data extraction template to keep all evidence in a single, auditable document.