Newcastle-Ottawa Scale Calculator

Observational studies, including cohort, case-control, and cross-sectional designs, form the backbone of evidence in many systematic reviews, particularly when randomized controlled trials are impractical or unavailable. The Newcastle-Ottawa Scale calculator implements the star-based scoring system developed by Wells et al. (2000) at the Universities of Newcastle (Australia) and Ottawa (Canada) to provide a standardized, reproducible method for appraising the methodological quality of these study designs. A NOS score tool allows reviewers to assign up to 9 stars across three categories: Selection (up to 4 stars), Comparability (up to 2 stars), and Outcome for cohort studies or Exposure for case-control studies (up to 3 stars). Each star represents a specific methodological criterion that the study satisfies, and the total score serves as a summary indicator of cohort study quality assessment that can be reported alongside effect estimates in meta-analyses. For cross-sectional studies, which fall outside the original NOS scope, a modified version adapted by Herzog et al. (2013) extends the star-based framework to a maximum of 10 stars and addresses the methodological concerns specific to cross-sectional designs, including sample size justification, response rate, and standardized outcome measurement. This calculator includes that adaptation as a dedicated tab, clearly labelled as an adaptation rather than the official scale. Because a scoping review by Kelly et al. (2024) found that two of Herzog's items, sample size and statistical test, measure reporting quality and statistical precision rather than risk of bias, the cross-sectional tab also offers a risk-of-bias focus toggle that excludes those two items and rescales the total to 8 stars, so reviewers can report either the familiar quality score or a stricter internal-validity appraisal. And because many published reviews adapt the scale to their own question, the Custom Instrument tab lets a review team define its own domains, items, and star values, score studies against that definition, and share the instrument as a JSON file so every reviewer appraises with an identical item set.

The Selection category evaluates whether the exposed and non-exposed cohorts (or cases and controls) were drawn from comparable populations and whether exposure or outcome ascertainment methods were reliable. The Comparability category, the most heavily weighted per-star domain, assesses whether the study controlled for the most important confounders, typically age, sex, and other variables central to the research question. The Outcome (or Exposure) category examines how the outcome was measured, whether follow-up was sufficiently long and complete, and whether objective assessment methods were used. The Cochrane Handbook for Systematic Reviews of Interventions (Higgins et al., 2023) acknowledges NOS as one of several validated tools for observational study quality, while PRISMA 2020 (Page et al., 2021) requires all systematic reviews to present results of quality or risk of bias assessments for every included study. It is worth noting that AMSTAR 2 (Shea et al., 2017) assesses the quality of systematic reviews themselves rather than primary studies. This is a distinct purpose that should not be confused with the NOS, which evaluates individual observational study methodology. Systematic review management platforms such as Covidence and DistillerSR now include built-in quality assessment modules that support NOS scoring alongside other appraisal tools, streamlining the workflow for review teams.

A common interpretation framework classifies studies scoring 7-9 stars as high quality, 4-6 as moderate quality, and 0-3 as low quality, although these thresholds are conventions rather than empirically validated cutoffs. Best practice involves using NOS scores in sensitivity analyses, for instance restricting a meta-analysis to high-quality studies to determine whether the pooled effect remains robust. NOS scores can also serve as a covariate in meta-regression, testing whether study quality moderates the treatment effect across studies. In dose-response meta-analyses, quality-stratified pooling by NOS score is particularly important because poorly designed studies may distort the shape of the exposure-response curve. Similarly, funnel plot asymmetry can be stratified by NOS score to disentangle publication bias from small-study effects driven by lower methodological quality. Because inter-rater reliability for NOS has been reported as moderate in validation studies (Lo et al., 2014), dual independent scoring followed by consensus resolution is essential. Reviewers should calculate and report the initial agreement rate, ideally using Cohen's kappa as an inter-rater reliability measure, in their methods section.

Choosing the right quality assessment tool depends on the study design and the level of detail required. NOS provides a relatively quick, star-based scoring system well-suited for large reviews with many observational studies. For a more granular, domain-based evaluation of non-randomized comparative studies, the ROBINS-I bias assessment for non-randomized studies uses signaling questions across seven domains with four judgment levels. Randomized trials should be appraised with the Cochrane RoB 2 risk of bias tool rather than NOS, as the two instruments address fundamentally different bias mechanisms. For reviews incorporating qualitative or prevalence data, the JBI critical appraisal checklists offer design-specific item sets from the Joanna Briggs Institute (Aromataris & Munn, 2020). Regardless of which instrument you choose, documenting the complete scoring rationale in your data extraction template builder ensures transparency and reproducibility across your review team.