Reference Deduplication Tool

A reference deduplication tool solves one of the most time-consuming bottlenecks in the systematic review workflow: identifying and removing records that appear in search results from multiple bibliographic databases. The Cochrane Handbook for Systematic Reviews of Interventions (Higgins et al., 2023) requires reviews to search a minimum of two databases, and most high-quality reviews search four or more, including PubMed/MEDLINE, Embase, CINAHL, Cochrane CENTRAL, Web of Science, and Scopus. Because major biomedical journals are indexed across multiple platforms, the same article frequently appears in several database exports. Empirical studies by Jiang et al. (2014) and Kwon et al. (2015) report that database overlap rates range from 10% to over 50% depending on the topic and databases searched. Running the same search across PubMed, Embase, and CINAHL typically yields 30–60% duplicates because major biomedical journals are indexed in all three platforms simultaneously. Screening platforms such as Covidence, Rayyan, and EPPI-Reviewer include built-in deduplication features, but their matching algorithms vary in sensitivity and may miss near-duplicates that specialized tools catch. Without systematic deduplication, these duplicate records inflate the total record count, waste screening time, and produce inaccurate numbers for the PRISMA 2020 flow diagram's identification phase. This tool allows researchers to remove duplicate citations efficiently using a layered matching approach that catches both exact and near-duplicate records.

The gold standard for exact deduplication is DOI (Digital Object Identifier) matching, because each DOI uniquely identifies a single published work regardless of how its metadata is formatted across databases. However, DOIs are not universally available. Older publications, conference abstracts, grey literature, and preprints may lack DOIs entirely. For records without DOIs, this RIS file deduplicator employs Jaccard title similarity, which computes the ratio of shared words to total unique words between two titles after normalization (lowercasing, punctuation removal, stopword handling). A Jaccard threshold of 0.85 balances sensitivity against false positives: it catches duplicates where one database includes a subtitle and another omits it, or where special characters render differently across export formats. The third matching layer, author-year-title start matching, addresses cases where abbreviated titles or encoding issues reduce Jaccard similarity below the threshold but the first author surname, publication year, and initial title words clearly identify the same study. Rathbone et al. (2015), in their evaluation of deduplication methods for systematic reviews, found that combining DOI matching with title similarity and author-year heuristics achieves >95% recall for true duplicate detection. The Bramer method (2016) formalized a systematic, stepwise deduplication approach specifically designed for systematic reviews, prioritizing DOI and PMID matching before progressively relaxing title and author criteria. CrossRef's DOI validation API can further verify that DOIs resolve to the same published record, catching cases where formatting errors produce syntactically different DOI strings for identical articles.

The practical workflow for systematic review deduplication begins immediately after search execution. Export all database search results in a consistent format. RIS is recommended because it preserves the most metadata fields, including DOIs, author lists, abstracts, and journal identifiers. While reference managers such as Zotero, Mendeley, and EndNote offer built-in deduplication, their algorithms are optimized for personal library management rather than systematic review rigor, and they may miss duplicates with inconsistent metadata or merge records that are actually distinct publications (e.g., conference abstracts versus full papers from the same study). Before deduplication, ensure your searches are finalized by building them with our search strategy builder, which constructs comprehensive Boolean queries with concept mapping and synonym expansion. If you need to run the same search across multiple databases, our database search translator converts your PubMed search into equivalent Embase, CINAHL, Cochrane, and Web of Science syntax, ensuring consistent scope while adapting field tags and operators. After deduplication, the number of unique records becomes the denominator for your screening phase, which you can run free in our systematic review screening tool that ranks every record by relevance before a reviewer decides.

Accurate deduplication counts are essential for PRISMA 2020 reporting compliance. The updated PRISMA flow diagram (Page et al., 2021) requires separate reporting of records identified from databases and registers, records removed before screening (including duplicates and records marked as ineligible by automation tools), and the number of unique records screened. Generate your flow diagram with our PRISMA flow diagram generator, which incorporates the PRISMA 2020 template with the updated identification pathways. For researchers who want to define their eligibility criteria before screening the deduplicated set, our inclusion and exclusion criteria builder helps formalize the population, intervention, comparator, outcome, and study design filters that will guide title-and-abstract and full-text screening decisions. Together, these tools create a reproducible, transparent pipeline from search execution through deduplication to study selection, meeting the methodological standards expected by Cochrane, JBI, and major peer-reviewed journals.