Find and remove duplicate citations from your systematic review search results. Paste references in RIS, BibTeX, or CSV format and detect duplicates using DOI matching, Jaccard title similarity, and author-year-title matching.
Paste your exported references in RIS, BibTeX, or CSV format into the text area. The tool auto-detects the format and parses all citation fields including title, authors, year, journal, DOI, volume, and pages.
The tool groups potential duplicates into sets based on three matching strategies: exact DOI match (strongest signal), title similarity above 85% (Jaccard word-level), and author-year-title start matching. Each match type is clearly labeled.
Review each duplicate set and confirm which references to remove. Duplicates are pre-selected for removal (keeping the first occurrence), but you can override any selection to keep specific versions you prefer.
After removing duplicates, export your cleaned reference list as RIS or CSV format. Copy the summary statistics (total imported, duplicates found, unique remaining) for your PRISMA flow diagram reporting.
Systematic reviews search multiple databases to ensure comprehensive retrieval, but substantial overlap exists between PubMed, Embase, CINAHL, and Cochrane. Studies show 20-40% overlap is typical, and without deduplication reviewers waste time screening the same study multiple times. Removing duplicates early in the workflow directly reduces screening burden and improves efficiency.
Digital Object Identifiers (DOIs) provide a unique, persistent identifier for each published article. When two references share the same DOI, they are definitively the same publication regardless of formatting differences. Always prioritize DOI matching as the first deduplication step. However, not all records include DOIs, especially older publications or grey literature, so additional matching strategies are needed.
The same study title can appear differently across databases due to capitalization, punctuation, special characters, or subtitle inclusion. Jaccard similarity at the word level is robust to these variations: it computes the intersection over union of word sets from two titles after normalization. A threshold of 85% balances sensitivity (catching true duplicates) with specificity (avoiding false positives from genuinely different studies with similar titles).
The PRISMA 2020 flow diagram requires reporting the number of records identified from each database, the number of duplicates removed, and the number of unique records screened. Accurate deduplication counts are essential for transparent reporting. Keep a record of which references were flagged as duplicates and which were removed to support reproducibility and audit trails in your systematic review.
A reference deduplication tool solves one of the most time-consuming bottlenecks in the systematic review workflow: identifying and removing records that appear in search results from multiple bibliographic databases. The Cochrane Handbook for Systematic Reviews of Interventions (Higgins et al., 2023) requires reviews to search a minimum of two databases — and most high-quality reviews search four or more, including PubMed/MEDLINE, Embase, CINAHL, Cochrane CENTRAL, Web of Science, and Scopus. Because major biomedical journals are indexed across multiple platforms, the same article frequently appears in several database exports. Empirical studies by Jiang et al. (2014) and Kwon et al. (2015) report that database overlap rates range from 10% to over 50% depending on the topic and databases searched — running the same search across PubMed, Embase, and CINAHL typically yields 30–60% duplicates because major biomedical journals are indexed in all three platforms simultaneously. Screening platforms such as Covidence, Rayyan, and EPPI-Reviewer include built-in deduplication features, but their matching algorithms vary in sensitivity and may miss near-duplicates that specialized tools catch. Without systematic deduplication, these duplicate records inflate the total record count, waste screening time, and produce inaccurate numbers for the PRISMA 2020 flow diagram's identification phase. This tool allows researchers to remove duplicate citations efficiently using a layered matching approach that catches both exact and near-duplicate records.
The gold standard for exact deduplication is DOI (Digital Object Identifier) matching, because each DOI uniquely identifies a single published work regardless of how its metadata is formatted across databases. However, DOIs are not universally available — older publications, conference abstracts, grey literature, and preprints may lack DOIs entirely. For records without DOIs, this RIS file deduplicator employs Jaccard title similarity, which computes the ratio of shared words to total unique words between two titles after normalization (lowercasing, punctuation removal, stopword handling). A Jaccard threshold of 0.85 balances sensitivity against false positives: it catches duplicates where one database includes a subtitle and another omits it, or where special characters render differently across export formats. The third matching layer — author-year-title start matching — addresses cases where abbreviated titles or encoding issues reduce Jaccard similarity below the threshold but the first author surname, publication year, and initial title words clearly identify the same study. Rathbone et al. (2015), in their evaluation of deduplication methods for systematic reviews, found that combining DOI matching with title similarity and author-year heuristics achieves >95% recall for true duplicate detection. The Bramer method (2016) formalized a systematic, stepwise deduplication approach specifically designed for systematic reviews, prioritizing DOI and PMID matching before progressively relaxing title and author criteria. CrossRef's DOI validation API can further verify that DOIs resolve to the same published record, catching cases where formatting errors produce syntactically different DOI strings for identical articles.
The practical workflow for systematic review deduplication begins immediately after search execution. Export all database search results in a consistent format — RIS is recommended because it preserves the most metadata fields, including DOIs, author lists, abstracts, and journal identifiers. While reference managers such as Zotero, Mendeley, and EndNote offer built-in deduplication, their algorithms are optimized for personal library management rather than systematic review rigor, and they may miss duplicates with inconsistent metadata or merge records that are actually distinct publications (e.g., conference abstracts versus full papers from the same study). Before deduplication, ensure your searches are finalized by building them with our search strategy builder, which constructs comprehensive Boolean queries with concept mapping and synonym expansion. If you need to run the same search across multiple databases, our database search translator converts your PubMed search into equivalent Embase, CINAHL, Cochrane, and Web of Science syntax, ensuring consistent scope while adapting field tags and operators. After deduplication, the number of unique records becomes the denominator for your screening phase.
Accurate deduplication counts are essential for PRISMA 2020 reporting compliance. The updated PRISMA flow diagram (Page et al., 2021) requires separate reporting of records identified from databases and registers, records removed before screening (including duplicates and records marked as ineligible by automation tools), and the number of unique records screened. Generate your flow diagram with our PRISMA flow diagram generator, which incorporates the PRISMA 2020 template with the updated identification pathways. For researchers who want to define their eligibility criteria before screening the deduplicated set, our inclusion and exclusion criteria builder helps formalize the population, intervention, comparator, outcome, and study design filters that will guide title-and-abstract and full-text screening decisions. Together, these tools create a reproducible, transparent pipeline from search execution through deduplication to study selection — meeting the methodological standards expected by Cochrane, JBI, and major peer-reviewed journals.
When conducting a systematic review, researchers typically search multiple databases (PubMed, Embase, CINAHL, Cochrane, Web of Science, Scopus) to ensure comprehensive coverage. Because journals are indexed across multiple databases, the same study often appears in several search results. Without deduplication, duplicate records inflate the apparent number of unique citations, waste screening time, and can lead to inaccurate PRISMA flow diagram counts. Studies report that database overlap can range from 10% to over 50% depending on the topic and databases searched, making deduplication a critical early step in the systematic review workflow.
Fuzzy matching uses approximate string comparison algorithms to identify references that are similar but not identical. This tool employs three strategies: (1) Exact DOI matching, which is the strongest signal since DOIs are unique identifiers; (2) Title similarity using Jaccard word-level comparison, which computes the ratio of shared words to total unique words between two titles after normalization (lowercasing, punctuation removal); and (3) Author-year-title start matching, which combines the first author surname, publication year, and the first five words of the title. These layered strategies catch duplicates even when formatting differs between databases, such as variations in author name order, abbreviated journal names, or minor title differences.
This tool supports three widely used bibliographic formats: RIS (.ris), which is the standard export format for EndNote, Zotero, Mendeley, and most database search results; BibTeX (.bib), commonly used with LaTeX and tools like JabRef; and CSV (.csv), a universal spreadsheet format with columns for title, author, year, journal, and DOI. RIS format uses tagged fields (TY, TI, AU, PY, JO, DO), BibTeX uses @article entries with named fields, and CSV expects a header row followed by data rows. You can paste the contents of any of these formats directly into the tool.
Near-duplicates require careful manual review. The tool flags references with title similarity above 85% (Jaccard index), but some flagged pairs may be genuinely different studies, such as separate publications from the same research group with similar titles, or different versions of a study (conference abstract vs. full paper). For each flagged pair, review the matching fields highlighted in the results: check whether the DOIs differ, whether author lists are substantially different, or whether the journal and year point to distinct publications. When in doubt, keep both references and resolve during full-text screening rather than risk excluding a relevant study.
Best practices include: (1) Export all database search results in a consistent format (RIS is recommended as it preserves the most metadata); (2) Run automated deduplication first to remove exact DOI matches; (3) Review fuzzy matches manually, paying attention to title similarity scores and author-year combinations; (4) Document the number of duplicates removed for your PRISMA flow diagram (identification phase); (5) Use a reference manager (EndNote, Zotero, Mendeley) as a secondary check after automated deduplication; (6) Have a second reviewer verify a random sample of removal decisions; (7) Keep a log of all removal decisions for reproducibility. The Cochrane Handbook recommends reporting the total number of records identified, the number of duplicates removed, and the number of unique records screened.
Studies report 30–60% overlap between PubMed and Embase, depending on the topic. Biomedical clinical trials have higher overlap, while pharmacology and European research show more Embase-unique records. This overlap means deduplication is essential after multi-database searching. Jiang et al. (2014) found that searching both databases retrieves 5–20% unique studies from each that would be missed by searching only one.
Always deduplicate before screening. Screening duplicates wastes reviewer time and can inflate your PRISMA flow diagram numbers. Deduplicate after exporting all database results but before importing into your screening tool (e.g., Covidence, Rayyan). Report both the pre-deduplication total and the number of duplicates removed in your PRISMA flow diagram under the Identification phase.
Match on DOI first (most reliable), then title similarity (using fuzzy matching with a threshold like Jaccard ≥ 0.85), author last names, publication year, and journal/source. DOI matching catches exact duplicates, while fuzzy title matching catches records with minor variations (e.g., different capitalization, special characters, or truncated titles). Volume and page numbers provide additional confirmation.
After deduplicating your references, generate a PRISMA flow diagram to document your identification and screening numbers. Building your search from scratch? Use our search strategy builder to construct Boolean search strings with synonym expansion for PubMed, Cochrane, and Embase. If you need to adapt your search across databases, the database search translator converts syntax between PubMed, Embase, CINAHL, and other major platforms.
Our information specialists can manage your entire search and deduplication workflow, run comprehensive multi-database searches, deduplicate results, and prepare your reference library for screening with full PRISMA documentation.