Finding a second reviewer is only the first step. Training that reviewer to apply your eligibility criteria consistently is what produces the high inter-rater agreement that journals expect.
Create a screening manual before any screening begins. This document should define each inclusion and exclusion criterion with at least two concrete examples of studies that meet and do not meet the criterion. Ambiguous scenarios should be addressed explicitly. For instance, if your inclusion criterion specifies "randomized controlled trials," your manual should clarify whether cluster-randomized trials, crossover designs, and quasi-randomized studies are included or excluded.
Run a pilot screening round on a random sample of 30 to 50 records. Both reviewers screen this batch independently, then compare decisions. Disagreements in the pilot round reveal criteria that are poorly defined, overly subjective, or interpreted differently by each reviewer. Revise your inclusion and exclusion criteria based on the pilot findings before proceeding to full screening.
Calculate pilot kappa after the pilot round. If kappa falls below 0.60, the criteria need refinement and additional discussion before moving forward. If kappa is 0.60 to 0.80, minor calibration may be sufficient. If kappa exceeds 0.80, proceed with confidence.
Document decision rules for edge cases encountered during the pilot. These rules become an appendix to your screening manual and ensure that similar studies are handled consistently throughout the review. Common edge cases include studies published as conference abstracts only, studies reporting relevant outcomes as secondary endpoints, and studies conducted in populations that overlap with but do not exactly match your inclusion criteria.
Schedule regular calibration meetings during the screening phase. After every 200 to 300 records, compare a random sample of decisions to confirm that agreement remains stable. Drift in criteria application is common in long screening phases, and periodic calibration prevents it.
Solo researchers, early-career faculty without a review team, and graduate students working on their first systematic review all face the same challenge: where to find a qualified second reviewer. Several options exist, each with distinct advantages and limitations.
A departmental colleague is the most accessible option. A faculty member or fellow graduate student in your department can serve as a second reviewer if they understand the subject area and commit to the time required. The advantage is cost (typically free), but the limitation is availability. Colleagues have their own research priorities, and screening 3,000 records is a significant time commitment that competes with their own deadlines.
A research assistant or graduate student can be trained as a second reviewer, particularly for title-and-abstract screening where clinical expertise is less critical than consistent criteria application. The Cochrane Handbook permits non-expert screeners at the title-and-abstract stage as long as they are properly trained and supervised, with clinical judgment reserved for full-text decisions.
A co-author from another institution adds methodological credibility to the review team while distributing the workload geographically. Multi-institutional author teams are viewed favorably during peer review and reduce the risk that a single institution's perspective biases the study selection.
Struggling to find a second reviewer? You are not alone. Many researchers, especially solo PhD students and early-career investigators, face this exact barrier. Research Gold provides professional second-reviewer services where an experienced methodologist screens your records independently using your protocol and eligibility criteria. You receive documented screening decisions, a completed kappa calculation, and a conflict resolution log ready for your methods section. request your scoping call or learn more about our Research Gold systematic review.
A professional systematic review service provides a second reviewer as part of a comprehensive methodology package. This option is particularly valuable when you need not just screening support but also extraction methods overview, risk of bias assessment, and statistical analysis. Professional services maintain trained reviewers who work exclusively on evidence synthesis, ensuring consistent quality and turnaround without the coordination challenges of volunteer collaborators.
Online research communities such as systematic review methodology groups on ResearchGate and LinkedIn occasionally connect solo researchers with volunteer second reviewers. However, the reliability and availability of volunteer matches vary widely, and there is no quality assurance mechanism for volunteer screeners.
Screening Workflow: Using Covidence and Rayyan for Blind Dual Screening
Blind screening means that each reviewer makes inclusion and exclusion decisions without seeing the other reviewer's choices. This independence is essential for calculating a valid inter-rater reliability statistic. Two software platforms dominate systematic review screening workflows.
Covidence is the screening platform recommended by Cochrane. It imports records directly from database searches, removes duplicates, and presents each record to both reviewers in a blinded interface. Reviewer 1 and Reviewer 2 see the same title, abstract, and keywords but cannot view each other's decisions until both have voted. Covidence automatically identifies conflicts (where one reviewer includes and the other excludes) and routes them to a resolution queue. The platform generates screening statistics, including kappa, that you can export directly for your methods section.
Rayyan is a free alternative developed by the Qatar Computing Research Institute. Rayyan uses a similar blinded screening workflow with a "blind mode" toggle that hides collaborator decisions during active screening. It supports unlimited collaborators, integrates with reference managers like EndNote and Zotero, and uses machine learning to suggest relevant records based on your early screening decisions. Rayyan's conflict resolution interface highlights disagreements for discussion. Research Gold also offers a free browser-based screening tool that supports the same independent two-reviewer workflow, ranks records by relevance, and computes Cohen's kappa on the reconciled decisions without any account or installation.
The screening workflow in both platforms follows the same evidence-based process. First, import deduplicated records from your database search. Second, enable blind mode so that both reviewers screen independently. Third, each reviewer applies the eligibility criteria to every record, voting include, exclude, or maybe. Fourth, after both reviewers complete screening, disable blind mode and review the conflict list. Fifth, resolve conflicts through discussion, applying documented decision rules, and consult a third reviewer for persistent disagreements. Sixth, export the completed screening data, including agreement statistics and the PRISMA flow diagram numbers.
Label your exclusion reasons during screening. Both Covidence and Rayyan allow you to tag each excluded record with a standardized reason (wrong population, wrong intervention, wrong study design, wrong outcome, duplicate, and similar categories). These labels populate the "reasons for exclusion" section of your PRISMA flow diagram and demonstrate to reviewers that your decisions were systematic rather than arbitrary.
