Every systematic review that involves human raters making categorical decisions requires a measure of agreement that goes beyond simple percentage overlap. Cohen's kappa, introduced by Jacob Cohen in 1960, corrects for the agreement you would expect by chance alone, producing a statistic that reflects only the genuine concordance between raters. Without this correction, two reviewers who both exclude 95% of records will appear to agree 90% of the time purely by coincidence, masking real disagreements that affect the validity of your review.
This guide covers the full practical workflow: when kappa is appropriate, how to calculate it by hand, how to use our free interrater agreement calculator, how to interpret results using the Landis and Koch (1977) benchmarks, and how to handle the many situations where kappa can mislead you. Whether you are screening titles and abstracts, resolving full-text disagreements, or measuring consistency during data extraction, this resource gives you the statistical foundation and reporting language you need. To do the dual screening itself, our free two-reviewer screening tool with kappa agreement scores agreement and lists every conflict.
When Cohen's Kappa Applies
Kappa is appropriate when you have exactly two raters assigning the same set of items to the same nominal categories. The classic use case in a systematic review is dual screening: two independent reviewers each label a record as "include" or "exclude." Because those labels are categorical and the raters are fixed, Cohen's kappa is the correct statistic.
Kappa does not apply when you have continuous measurements (use the intraclass correlation coefficient instead), when categories are ordinal and the distance between disagreements matters (use weighted kappa), or when three or more raters classify items simultaneously (use Fleiss' kappa). Using the wrong statistic undermines the credibility of your reliability reporting.
The Relationship Between Raw Agreement and Kappa
Raw percent agreement simply divides the number of concordant decisions by the total number of items. It ignores the fact that raters will agree on some items by pure chance. Kappa subtracts this expected agreement from the observed agreement, then divides by the maximum possible improvement over chance:
Kappa = (Po - Pe) / (1 - Pe)
Where Po is observed agreement and Pe is expected agreement under independence. This formula means kappa can be negative (agreement worse than chance), zero (agreement equals chance), or positive up to 1.0 (perfect agreement).
Calculating Kappa by Hand: A 2x2 Contingency Table Walkthrough
Understanding the manual calculation builds intuition about what kappa actually measures. Suppose two reviewers screen 200 records for a systematic review on intervention effectiveness. Their decisions form a 2x2 contingency table:
| Reviewer B: Include | Reviewer B: Exclude | Row Total | |
|---|---|---|---|
| Reviewer A: Include | 30 | 8 | 38 |
| Reviewer A: Exclude | 12 | 150 | 162 |
| Column Total | 42 | 158 | 200 |
Step 1: Calculate observed agreement (Po). The diagonal cells represent concordant decisions. Po = (30 + 150) / 200 = 0.90.
Step 2: Calculate expected agreement (Pe). For the "include/include" cell, the expected count under independence is (38 x 42) / 200 = 7.98. For the "exclude/exclude" cell, the expected count is (162 x 158) / 200 = 127.98. So Pe = (7.98 + 127.98) / 200 = 0.6798.
Step 3: Calculate kappa. Kappa = (0.90 - 0.6798) / (1 - 0.6798) = 0.2202 / 0.3202 = 0.688.
This result falls in the substantial agreement range on the Landis and Koch scale. You can verify this result instantly using our Cohen's kappa calculator, which also provides the standard error and 95% confidence interval.
Step 4: Calculate the standard error. The standard error for kappa allows you to construct a confidence interval. For the example above, the 95% confidence interval would be approximately 0.57 to 0.81, meaning you can be confident the true agreement level is at least moderate and possibly almost perfect.
Using the Kappa Calculator
Navigate to the Cohen's kappa calculator and enter your data as a contingency table or item-by-item ratings. The output includes: observed agreement (Po), expected agreement (Pe), Cohen's kappa, standard error and 95% confidence interval, Landis and Koch benchmark category, prevalence index, and bias index.
The calculator handles the arithmetic and lets you focus on interpretation. For studies with ordinal rating scales, you can toggle between unweighted, linear weighted, and quadratic weighted kappa directly in the tool interface.
Interpreting Kappa: The Landis and Koch Scale
The most widely cited benchmarks for kappa come from Landis and Koch (1977). While these thresholds were originally proposed as guidelines rather than rigid cutoffs, they have become the standard reference in health sciences and systematic review methodology.
| Kappa Value | Strength of Agreement |
|---|---|
| Below 0.00 | Poor |
| 0.00 to 0.20 | Slight |
| 0.21 to 0.40 | Fair |
| 0.41 to 0.60 | Moderate |
| 0.61 to 0.80 | Substantial |
| 0.81 to 1.00 | Almost Perfect |
For systematic review screening, kappa at or above 0.61 (substantial agreement) is generally the minimum acceptable level.
A kappa below 0.41 indicates that inclusion criteria are ambiguous or raters have not calibrated consistently. Re-examine criteria definitions and conduct a calibration exercise before proceeding with full screening.
Minimum Acceptable Kappa Thresholds by Journal and Field
Different disciplines and journals set different bars for what counts as adequate agreement. Cicchetti (1994) proposed an alternative framework that is commonly used in clinical and psychological research:
| Kappa Value | Cicchetti Classification |
|---|---|
| Below 0.40 | Poor |
| 0.40 to 0.59 | Fair |
| 0.60 to 0.74 | Good |
| 0.75 to 1.00 | Excellent |
Cochrane reviews expect at least substantial agreement (kappa 0.61 or above) for screening decisions. The Cochrane Handbook recommends that review teams report kappa for both title/abstract screening and full-text screening separately.
Clinical journals (BMJ, JAMA, Lancet) typically expect kappa above 0.70, and many editors consider values below 0.60 a reason to question the reliability of the review process. Psychology and education journals often cite Cicchetti's thresholds and consider 0.75 or above as the gold standard.
Nursing and public health journals tend to accept the Landis and Koch framework and consider substantial agreement (0.61+) sufficient, particularly when the prevalence index is high and raw agreement is documented alongside kappa.
When in doubt, check the specific reporting guidelines of your target journal and the relevant Cochrane or JBI methodology manual for your review type.
Weighted Kappa for Ordinal Data
When your rating categories have a natural order, not all disagreements are equally serious. A rater who assigns "high risk of bias" when the correct answer is "low risk" has made a larger error than one who assigns "unclear" instead of "low." Weighted kappa accounts for this by penalizing distant disagreements more heavily than adjacent ones.
Linear Versus Quadratic Weighting
Linear weights assign penalties proportional to the distance between categories. If you have three ordered categories (low, unclear, high), disagreement between low and unclear receives a penalty of 1, while disagreement between low and high receives a penalty of 2.
Quadratic weights assign penalties proportional to the squared distance. The low-to-high disagreement is penalized four times as heavily as the low-to-unclear disagreement. Quadratic weighting produces values that are mathematically equivalent to the intraclass correlation coefficient for a two-way random effects model, making it the preferred choice when you want direct comparability with ICC results.
When to Use Each Weighting Scheme
Use unweighted kappa for binary decisions: include versus exclude, yes versus no, present versus absent. This is the correct choice for title/abstract screening and full-text eligibility decisions.
Use linear weighted kappa when all adjacent-category disagreements carry equal practical importance, such as rating patient satisfaction on a 5-point scale.
Use quadratic weighted kappa for risk-of-bias assessments (low, some concerns, high) and quality appraisal tools where extreme disagreements are far more consequential than near-miss disagreements. This is the most common weighting scheme in systematic review methodology.
Our Cohen's kappa calculator supports all three options, so you can compare results across weighting schemes for the same dataset.
