Screening tool benchmark: relevance ranking on whole systematic reviews
This benchmark replays the screening tool's relevance ranking on every review dataset of the two public SYNERGY collections, each record carrying the inclusion decision of the reviewers who ran the original review. It measures how much reading the ranking saves before 95% of the relevant records have been found. The saving is stated as work saved over sampling at 95% recall (WSS@95): the share of the pile left unread, less the 5% that reading in random order would leave unread as well. Across the 26 reviews of SYNERGY 1.0 (169,288 records), the median saving was 65.3%, and it ran from 3.7% on the weakest review to 92.7% on the strongest. Across the 114 reviews of SYNERGY+ 3.0 (168,969 records), the median was 70.6% (-5.0% to 93.5%). Every review is in the tables below, the weak results included. The run is dated 2026-11-13.
What was measured, and on what
The part of the tool under test is Rank by relevance with Learn from my decisions: the active-learning ranking that reorders the unscreened pile after your include and exclude calls. The benchmark runs the tool's own ranking code, unmodified, with the default ranking model. It makes no use of the AI screening assistant, which is a separate feature covered further down.
The data are two public collections of screening piles from published systematic reviews, each record labelled with the review's final inclusion decision. Both are released under the CC0 1.0 licence, and the records (titles and abstracts taken from OpenAlex) are frozen inside each release, so a rerun reads the same text.
- SYNERGY, version 1.0 (doi:10.34894/HE6NAQ): 26 reviews. Every record of every review is kept, including the 4,765 of 169,288 records (2.8%) that have no abstract. This is the whole screening pile at its real prevalence: 2,834 relevant records in total.
- SYNERGY+, version 3.0 (doi:10.34894/DDCVCV): 119 review folders. Each review was built the way the dataset's own software builds it: only open access records that carry an abstract of at least 20 words or 100 characters are kept, and a review needs at least 3 included records after that filter. A review here is therefore the open access part of the original pile, with no record lacking an abstract, and its prevalence is that of this part.
The two collections overlap: 22 of the 26 SYNERGY 1.0 reviews are also in SYNERGY+ 3.0 (matched by the DOI of the review's publication), with a different set of records. The two tables are two views of partly the same material and should not be added together.
How the benchmark was run
- Each review starts as an unscreened pile of titles and abstracts. No eligibility criteria are typed in, so the ranking has only the decisions to learn from.
- One relevant and one irrelevant record are drawn at random and given to the ranking as the first two decisions, as in our earlier benchmark report. These two seed records are left out of every figure.
- The ranking orders the remaining records. The record at the top is "screened": its true label is revealed and added to the decisions. The ranking retrains after every record for the first 25 records and then after each further 5% of records screened, the schedule the tool's own simulation uses.
- This continues until every relevant record has been found; the rest of the pile is read in the last ranked order.
- The whole simulation is repeated 10 times per review with fixed random seeds 1 to 10, which change the two seed records. Each table cell reports the median of the 10 runs and, under it, the lowest and the highest run.
The simulations took 75 minutes of computing in total. The results of every run, review by review and seed by seed, are stored in one file from which this page is rendered, so the page cannot say anything the run did not produce.
The four figures and their formulas
Write N for the records left after the two seed records, R for the relevant records among them, and n95 for the number of records read, in ranked order, at the moment 95% of R (rounded up to a whole record) has been found. The definitions are those of the tool's simulation and of our earlier report.
- Work saved at 95% recall
- Work saved over sampling at 95% recall (WSS@95) = (N − n95) / N − 0.05. It is the share of the pile you did not have to read, compared with reading in random order, which finds 95% of the relevant records only after reading 95% of everything. Zero means no better than random order; a negative value means worse. It can never exceed 0.95 minus the share of the pile taken up by the relevant records that have to be found.
- Read to reach 95% recall
- n95 / N: the share of the pile read when 95% of the relevant records had been found.
- Found after 10% and after 20%
- The share of R found once the first 10% (or 20%) of N, rounded up to a whole record, had been read.
Results for every review: SYNERGY 1.0
26 reviews in alphabetical order, as the dataset lists them. Each review name links to the publication of the original review. On a narrow screen the table scrolls sideways.
| Review | Records | Relevant | No abstract | Work saved at 95% recall | Read to reach 95% recall | Found after 10% | Found after 20% |
|---|---|---|---|---|---|---|---|
| Appenzeller-Herzog 2019 | 2,873 | 260.91% | 67623.5% | 63.6%60.6 to 71.5 | 31.4%23.5 to 34.4 | 72.0%0.0 to 76.0 | 88.0%84.0 to 92.0 |
| Bos 2018 | 4,878 | 100.21% | 621.3% | 85.0%85.0 to 90.7 | 10.0%4.3 to 10.0 | 100.0%88.9 to 100.0 | 100.0%every run |
| Brouwer 2019 | 38,114 | 620.16% | 4261.1% | 92.0%91.8 to 92.4 | 3.0%2.6 to 3.2 | 100.0%every run | 100.0%every run |
| Chou 2003 | 1,908 | 150.79% | 1105.8% | 3.7%3.6 to 3.8 | 91.3%91.2 to 91.4 | 85.7%every run | 85.7%every run |
| Chou 2004 | 1,630 | 90.55% | 18911.6% | 13.0%13.0 to 33.4 | 82.0%61.6 to 82.0 | 25.0%12.5 to 37.5 | 25.0%12.5 to 50.0 |
| Donners 2021 | 258 | 155.81% | 83.1% | 11.0%10.6 to 11.0 | 84.0%84.0 to 84.4 | 75.0%64.3 to 85.7 | 85.7%every run |
| Hall 2012 | 8,793 | 1041.18% | 00.0% | 92.7%92.6 to 92.7 | 2.3%2.3 to 2.4 | 99.0%every run | 99.0%every run |
| Jeyaraman 2020 | 1,175 | 968.17% | 20.2% | 51.8%51.6 to 51.9 | 43.2%43.1 to 43.4 | 55.8%53.7 to 56.8 | 75.3%73.7 to 76.8 |
| Leenaars 2019 | 5,812 | 170.29% | 490.8% | 92.7%92.6 to 93.0 | 2.3%2.0 to 2.4 | 100.0%every run | 100.0%every run |
| Leenaars 2020 | 7,216 | 5838.08% | 3765.2% | 56.8%55.4 to 58.5 | 38.2%36.5 to 39.6 | 51.8%50.3 to 59.6 | 67.9%67.0 to 85.9 |
| Meijboom 2021 | 882 | 374.20% | 576.5% | 75.0%74.5 to 75.9 | 20.0%19.1 to 20.5 | 61.1%41.7 to 63.9 | 97.2%94.4 to 97.2 |
| Menon 2022 | 975 | 747.59% | 60.6% | 21.1%21.0 to 21.1 | 73.9%73.9 to 74.0 | 52.1%50.7 to 56.2 | 72.6%every run |
| Moran 2021 | 5,214 | 1112.13% | 430.8% | 9.5%8.8 to 9.6 | 85.5%85.4 to 86.2 | 16.4%6.4 to 20.9 | 35.5%26.4 to 38.2 |
| Muthu 2021 | 2,719 | 33612.36% | 240.9% | 35.8%35.3 to 38.4 | 59.2%56.6 to 59.7 | 33.7%29.0 to 37.9 | 56.6%55.2 to 59.1 |
| Nelson 2002 | 366 | 8021.86% | 82.2% | 35.0%34.3 to 37.0 | 60.0%58.0 to 60.7 | 27.2%24.1 to 29.1 | 50.0%44.3 to 54.4 |
| Oud 2018 | 952 | 202.10% | 151.6% | 69.8%69.2 to 81.6 | 25.2%13.4 to 25.8 | 68.4%63.2 to 84.2 | 94.7%94.7 to 100.0 |
| Radjenovic 2013 | 5,935 | 480.81% | 00.0% | 88.2%88.0 to 88.4 | 6.8%6.6 to 7.0 | 97.9%every run | 97.9%every run |
| Sep 2021 | 271 | 4014.76% | 10.4% | 14.0%13.2 to 15.4 | 81.0%79.6 to 81.8 | 25.6%20.5 to 30.8 | 55.1%51.3 to 56.4 |
| Smid 2020 | 2,627 | 271.03% | 1104.2% | 82.9%82.7 to 85.4 | 12.1%9.6 to 12.3 | 92.3%88.5 to 96.2 | 100.0%every run |
| van de Schoot 2018 | 4,544 | 380.84% | 711.6% | 89.2%87.4 to 89.6 | 5.8%5.4 to 7.6 | 100.0%every run | 100.0%every run |
| van der Valk 2021 | 725 | 8912.28% | 101.4% | 48.4%48.3 to 54.6 | 46.6%40.4 to 46.7 | 37.5%28.4 to 38.6 | 62.5%62.5 to 63.6 |
| van der Waal 2022 | 1,970 | 331.68% | 130.7% | 67.0%66.7 to 75.0 | 28.0%20.0 to 28.3 | 87.5%87.5 to 90.6 | 93.8%93.8 to 96.9 |
| van Dis 2020 | 9,128 | 720.79% | 1661.8% | 69.2%68.9 to 69.3 | 25.8%25.7 to 26.1 | 73.2%71.8 to 76.1 | 90.1%90.1 to 93.0 |
| Walker 2018 | 48,375 | 7621.58% | 1,5333.2% | 42.8%41.0 to 44.4 | 52.2%50.6 to 54.0 | 75.2%74.4 to 76.3 | 86.9%86.2 to 87.4 |
| Wassenaar 2017 | 7,668 | 1111.45% | 4295.6% | 76.3%75.5 to 77.4 | 18.7%17.6 to 19.5 | 86.8%85.5 to 89.1 | 95.9%95.5 to 97.3 |
| Wolters 2018 | 4,280 | 190.44% | 3818.9% | 68.1%67.9 to 83.8 | 26.9%11.2 to 27.1 | 88.9%88.9 to 94.4 | 94.4%94.4 to 100.0 |
No review of this collection was left out.
Results for every review: SYNERGY+ 3.0
114 reviews in alphabetical order, as the dataset lists them. Each review name links to the publication of the original review. On a narrow screen the table scrolls sideways.
| Review | Records | Relevant | No abstract | Work saved at 95% recall | Read to reach 95% recall | Found after 10% | Found after 20% |
|---|---|---|---|---|---|---|---|
| Abgaz 2023 | 288 | 113.82% | 00.0% | 80.3%75.4 to 80.7 | 14.7%14.3 to 19.6 | 60.0%50.0 to 70.0 | 100.0%every run |
| Adamo 2021 | 1,151 | 161.39% | 00.0% | 66.5%65.9 to 67.1 | 28.5%27.9 to 29.1 | 73.3%66.7 to 80.0 | 80.0%every run |
| Ali 2024 | 515 | 305.83% | 00.0% | 53.9%52.5 to 53.9 | 41.1%41.1 to 42.5 | 39.7%34.5 to 48.3 | 65.5%58.6 to 69.0 |
| Anmarkrud 2021 | 593 | 162.70% | 00.0% | 48.1%48.0 to 87.9 | 46.9%7.1 to 47.0 | 86.7%86.7 to 100.0 | 93.3%93.3 to 100.0 |
| Aouad 2024 | 714 | 304.20% | 00.0% | 64.9%62.1 to 65.1 | 30.1%29.9 to 32.9 | 86.2%72.4 to 86.2 | 89.7%every run |
| Appenzeller-Herzog 2019 | 582 | 81.37% | 00.0% | 74.7%73.6 to 74.8 | 20.3%20.2 to 21.4 | 57.1%14.3 to 57.1 | 85.7%every run |
| Attai 2022 | 1,726 | 1357.82% | 00.0% | 54.7%54.6 to 55.0 | 40.3%40.0 to 40.4 | 54.5%53.0 to 57.5 | 79.9%76.9 to 81.3 |
| Bakker-Jacobs 2022 | 549 | 81.46% | 00.0% | 46.4%46.0 to 52.8 | 48.6%42.2 to 49.0 | 42.9%0.0 to 42.9 | 57.1%57.1 to 85.7 |
| Bech 2019 | 556 | 152.70% | 00.0% | 32.2%32.0 to 32.2 | 62.8%62.8 to 63.0 | 71.4%71.4 to 78.6 | 92.9%every run |
| Bindoli 2024a | 1,449 | 644.42% | 00.0% | 71.2%70.6 to 71.4 | 23.8%23.6 to 24.4 | 66.7%66.7 to 74.6 | 89.7%87.3 to 90.5 |
| Bindoli 2024b | 341 | 72.05% | 00.0% | 73.5%72.6 to 78.2 | 21.5%16.8 to 22.4 | 50.0%every run | 83.3%83.3 to 100.0 |
| Boersma-van Dam 2024 | 578 | 244.15% | 00.0% | 72.1%70.9 to 77.6 | 22.9%17.4 to 24.1 | 63.0%60.9 to 65.2 | 87.0%87.0 to 95.7 |
| Bos 2018 | 2,241 | 60.27% | 00.0% | 91.7%91.7 to 93.3 | 3.3%1.7 to 3.3 | 100.0%every run | 100.0%every run |
| Bosch 2021 | 1,716 | 221.28% | 00.0% | 85.5%85.0 to 85.5 | 9.5%9.5 to 10.0 | 95.2%every run | 95.2%every run |
| Bosch 2023 | 2,341 | 190.81% | 00.0% | 84.5%every run | 10.5%every run | 88.9%every run | 100.0%every run |
| Brons 2024 | 172 | 2514.53% | 00.0% | 5.0%3.8 to 46.2 | 90.0%48.8 to 91.2 | 25.0%16.7 to 37.5 | 54.2%25.0 to 58.3 |
| Brouwer 2019 | 11,440 | 140.12% | 00.0% | 91.3%91.2 to 91.8 | 3.7%3.2 to 3.8 | 100.0%every run | 100.0%every run |
| Burska 2023 | 3,247 | 1584.87% | 00.0% | 75.5%74.0 to 76.1 | 19.5%18.9 to 21.0 | 78.3%74.5 to 80.9 | 95.5%94.9 to 96.2 |
| Butink 2023 | 3,090 | 300.97% | 00.0% | 78.3%78.2 to 78.8 | 16.7%16.2 to 16.8 | 86.2%86.2 to 89.7 | 100.0%every run |
| Chakraborty 2023 | 2,233 | 70.31% | 00.0% | 77.7%77.6 to 81.3 | 17.3%13.7 to 17.4 | 66.7%50.0 to 66.7 | 100.0%every run |
| Chueca 2023 | 609 | 162.63% | 00.0% | -0.4%-0.6 to -0.4 | 95.4%95.4 to 95.6 | 40.0%26.7 to 60.0 | 66.7%66.7 to 73.3 |
| Cinquin 2018 | 290 | 93.10% | 00.0% | 41.9%38.4 to 42.2 | 53.1%52.8 to 56.6 | 25.0%12.5 to 62.5 | 62.5%25.0 to 75.0 |
| Clarinval 2021 | 446 | 194.26% | 00.0% | 60.1%59.9 to 70.5 | 34.9%24.5 to 35.1 | 61.1%50.0 to 77.8 | 88.9%every run |
| Clark 2021 | 199 | 31.51% | 00.0% | 93.5%93.5 to 94.0 | 1.5%1.0 to 1.5 | 100.0%every run | 100.0%every run |
| Cozim-Melges 2024 | 630 | 9515.08% | 00.0% | 25.3%24.6 to 29.6 | 69.7%65.4 to 70.4 | 34.0%10.6 to 36.2 | 50.0%39.4 to 52.1 |
| de Matteis 2024a | 140 | 32.14% | 00.0% | 79.1%77.6 to 82.0 | 15.9%13.0 to 17.4 | 50.0%every run | 100.0%every run |
| de Matteis 2024b | 553 | 244.34% | 00.0% | 80.7%79.9 to 82.8 | 14.3%12.2 to 15.1 | 87.0%87.0 to 91.3 | 95.7%95.7 to 100.0 |
| Deckers 2022 | 106 | 32.83% | 00.0% | 74.8%74.8 to 85.4 | 20.2%9.6 to 20.2 | 50.0%50.0 to 100.0 | 100.0%every run |
| Dolinska 2022 | 3,231 | 150.46% | 00.0% | 86.5%86.4 to 86.5 | 8.5%8.5 to 8.6 | 100.0%every run | 100.0%every run |
| Donners 2021 | 119 | 1411.76% | 00.0% | 70.2%63.4 to 70.2 | 24.8%24.8 to 31.6 | 61.5%61.5 to 69.2 | 84.6%every run |
| Dornauer 2023 | 259 | 51.93% | 00.0% | 87.2%87.2 to 93.4 | 7.8%1.6 to 7.8 | 100.0%every run | 100.0%every run |
| Eggmann 2023 | 266 | 93.38% | 00.0% | 77.0%76.4 to 78.3 | 18.0%16.7 to 18.6 | 50.0%25.0 to 87.5 | 100.0%every run |
| Endedijk 2021 | 1,092 | 898.15% | 00.0% | 63.6%61.1 to 64.5 | 31.4%30.5 to 33.9 | 47.7%36.4 to 47.7 | 83.0%80.7 to 84.1 |
| Farisogullari 2023 | 1,439 | 594.10% | 00.0% | 84.4%83.4 to 85.3 | 10.6%9.7 to 11.6 | 94.8%93.1 to 98.3 | 98.3%98.3 to 100.0 |
| Fejza 2023 | 1,499 | 453.00% | 00.0% | 63.9%63.9 to 64.1 | 31.1%30.9 to 31.1 | 61.4%54.5 to 68.2 | 88.6%86.4 to 90.9 |
| Ferreira 2022 | 223 | 31.35% | 00.0% | 67.9%67.9 to 81.4 | 27.1%13.6 to 27.1 | 50.0%0.0 to 50.0 | 50.0%50.0 to 100.0 |
| Filges 2018 | 925 | 202.16% | 00.0% | 66.7%66.6 to 79.9 | 28.3%15.1 to 28.4 | 84.2%78.9 to 94.7 | 94.7%94.7 to 100.0 |
| Filges 2022 | 781 | 60.77% | 00.0% | 3.9%3.7 to 81.8 | 91.1%13.2 to 91.3 | 60.0%60.0 to 80.0 | 80.0%80.0 to 100.0 |
| Fong 2021 | 489 | 449.00% | 00.0% | 58.7%58.4 to 58.9 | 36.3%36.1 to 36.6 | 72.1%72.1 to 74.4 | 93.0%every run |
| Giesen 2021 | 1,293 | 110.85% | 00.0% | 42.9%41.3 to 48.1 | 52.1%46.9 to 53.7 | 10.0%0.0 to 30.0 | 20.0%10.0 to 60.0 |
| Greca 2023 | 328 | 247.32% | 00.0% | 63.7%63.4 to 64.9 | 31.3%30.1 to 31.6 | 39.1%21.7 to 56.5 | 69.6%65.2 to 73.9 |
| Hall 2011 | 340 | 41.18% | 00.0% | 92.8%92.0 to 93.2 | 2.2%1.8 to 3.0 | 100.0%every run | 100.0%every run |
| Hamaker 2022 | 1,313 | 130.99% | 00.0% | 80.4%79.7 to 83.4 | 14.6%11.6 to 15.3 | 75.0%75.0 to 91.7 | 100.0%every run |
| Hanlon 2022 | 77 | 67.79% | 00.0% | 69.7%68.3 to 77.7 | 25.3%17.3 to 26.7 | 50.0%20.0 to 80.0 | 80.0%80.0 to 100.0 |
| Harms 2024 | 840 | 485.71% | 00.0% | 21.0%20.7 to 21.1 | 74.0%73.9 to 74.3 | 31.9%19.1 to 34.0 | 56.4%46.8 to 57.4 |
| Hilfiker 2017 | 7,573 | 7499.89% | 00.0% | 69.2%68.1 to 69.6 | 25.8%25.4 to 26.9 | 62.6%60.7 to 63.1 | 91.0%90.4 to 91.6 |
| Kapuka 2021 | 151 | 85.30% | 00.0% | 56.7%56.1 to 78.2 | 38.3%16.8 to 38.9 | 57.1%14.3 to 71.4 | 85.7%85.7 to 100.0 |
| Kerschbaumer 2022 | 2,605 | 291.11% | 00.0% | 82.8%82.2 to 86.2 | 12.2%8.8 to 12.8 | 89.3%89.3 to 96.4 | 96.4%96.4 to 100.0 |
| Kerschbaumer 2024a | 1,949 | 291.49% | 00.0% | 65.4%65.4 to 84.8 | 29.6%10.2 to 29.6 | 85.7%85.7 to 92.9 | 85.7%85.7 to 96.4 |
| Kerschbaumer 2024b | 1,146 | 151.31% | 00.0% | 74.2%74.0 to 76.0 | 20.8%19.0 to 21.0 | 71.4%42.9 to 78.6 | 85.7%85.7 to 100.0 |
| Lauper 2021 | 5,691 | 1372.41% | 00.0% | 60.1%59.9 to 63.6 | 34.9%31.4 to 35.1 | 78.7%74.3 to 80.9 | 86.0%83.8 to 86.8 |
| Leenaars 2019 | 1,395 | 70.50% | 00.0% | 92.6%92.6 to 93.0 | 2.4%2.0 to 2.4 | 100.0%every run | 100.0%every run |
| Leenaars 2020 | 1,956 | 19910.17% | 00.0% | 39.0%36.7 to 55.0 | 56.0%40.0 to 58.3 | 41.2%32.3 to 42.9 | 61.4%59.1 to 75.3 |
| Lewowski 2021 | 329 | 92.74% | 00.0% | 78.5%78.5 to 78.8 | 16.5%16.2 to 16.5 | 87.5%every run | 100.0%every run |
| Low 2023 | 556 | 366.47% | 00.0% | 38.5%38.0 to 38.5 | 56.5%56.5 to 57.0 | 38.6%25.7 to 42.9 | 74.3%62.9 to 80.0 |
| Maciel 2024 | 274 | 3111.31% | 00.0% | 51.6%50.9 to 52.0 | 43.4%43.0 to 44.1 | 60.0%50.0 to 63.3 | 86.7%83.3 to 86.7 |
| Marques 2021 | 713 | 162.24% | 00.0% | 57.3%57.3 to 78.5 | 37.7%16.5 to 37.7 | 86.7%6.7 to 86.7 | 93.3%93.3 to 100.0 |
| Meijboom 2021 | 464 | 214.53% | 00.0% | 76.4%74.4 to 76.4 | 18.6%18.6 to 20.6 | 70.0%10.0 to 70.0 | 100.0%90.0 to 100.0 |
| Mejean 2024 | 486 | 6613.58% | 00.0% | 50.8%38.4 to 54.9 | 44.2%40.1 to 56.6 | 43.1%33.8 to 47.7 | 66.2%64.6 to 67.7 |
| Menon 2022 | 485 | 357.22% | 00.0% | 4.9%4.5 to 4.9 | 90.1%90.1 to 90.5 | 41.2%35.3 to 47.1 | 60.3%50.0 to 67.6 |
| Mohamed 2023 | 2,699 | 802.96% | 00.0% | 78.4%78.0 to 79.5 | 16.6%15.5 to 17.0 | 81.0%77.2 to 81.0 | 97.5%97.5 to 100.0 |
| Monschau 2025 | 7,384 | 510.69% | 00.0% | 89.8%89.3 to 89.9 | 5.2%5.1 to 5.7 | 100.0%every run | 100.0%every run |
| Moran 2020 | 1,992 | 391.96% | 00.0% | 33.0%32.8 to 38.9 | 62.0%56.1 to 62.2 | 52.6%50.0 to 55.3 | 76.3%76.3 to 78.9 |
| Moseng 2024 | 3,556 | 381.07% | 00.0% | 68.6%68.4 to 68.7 | 26.4%26.3 to 26.6 | 54.1%51.4 to 75.7 | 86.5%86.5 to 89.2 |
| Muthu 2020 | 923 | 202.17% | 00.0% | 11.5%11.4 to 39.3 | 83.5%55.7 to 83.6 | 36.8%5.3 to 52.6 | 55.3%36.8 to 63.2 |
| Nobrega-Dos Santos 2023 | 277 | 82.89% | 00.0% | 85.2%84.8 to 86.3 | 9.8%8.7 to 10.2 | 100.0%every run | 100.0%every run |
| Noetel 2021 | 444 | 61.35% | 00.0% | 86.2%85.5 to 89.3 | 8.8%5.7 to 9.5 | 100.0%every run | 100.0%every run |
| Oliveira 2021 | 184 | 179.24% | 00.0% | 9.8%9.8 to 68.1 | 85.2%26.9 to 85.2 | 37.5%31.3 to 56.3 | 62.5%62.5 to 93.8 |
| Oud 2018 | 243 | 52.06% | 00.0% | 70.9%70.1 to 84.6 | 24.1%10.4 to 24.9 | 50.0%0.0 to 100.0 | 50.0%50.0 to 100.0 |
| Parodis 2023a | 4,125 | 521.26% | 00.0% | 76.5%76.3 to 76.7 | 18.5%18.3 to 18.7 | 90.2%90.2 to 92.2 | 96.1%96.1 to 98.0 |
| Parodis 2023b | 1,679 | 331.97% | 00.0% | 78.3%78.1 to 78.5 | 16.7%16.5 to 16.9 | 81.3%81.3 to 87.5 | 96.9%every run |
| Pijls 2017 | 54 | 35.56% | 00.0% | 91.2%89.2 to 91.2 | 3.8%3.8 to 5.8 | 100.0%every run | 100.0%every run |
| Pijls 2018 | 301 | 165.32% | 00.0% | 79.6%79.6 to 83.0 | 15.4%12.0 to 15.4 | 86.7%80.0 to 86.7 | 100.0%every run |
| Pinos-Cisneros 2023 | 109 | 2119.27% | 00.0% | 40.8%38.9 to 42.7 | 54.2%52.3 to 56.1 | 15.0%15.0 to 35.0 | 40.0%20.0 to 50.0 |
| Quevedo 2023 | 199 | 189.05% | 00.0% | 65.6%65.6 to 68.6 | 29.4%26.4 to 29.4 | 61.8%23.5 to 64.7 | 82.4%82.4 to 88.2 |
| Ramiro 2015 | 338 | 185.33% | 00.0% | 86.7%86.1 to 86.7 | 8.3%8.3 to 8.9 | 100.0%every run | 100.0%every run |
| Rinne 2021 | 815 | 19423.80% | 00.0% | 36.8%36.7 to 36.8 | 58.2%58.2 to 58.3 | 30.1%29.5 to 31.1 | 56.0%55.4 to 57.5 |
| Roberts 2021 | 883 | 30.34% | 00.0% | 83.6%81.6 to 86.8 | 11.4%8.2 to 13.4 | 50.0%50.0 to 100.0 | 100.0%every run |
| Ruggeri 2019 | 2,265 | 110.49% | 00.0% | 85.1%85.1 to 86.7 | 9.9%8.3 to 9.9 | 100.0%every run | 100.0%every run |
| Sanchez-Acedo 2023 | 345 | 205.80% | 00.0% | 0.0%0.0 to 20.1 | 95.0%74.9 to 95.0 | 15.8%5.3 to 21.1 | 21.1%10.5 to 36.8 |
| Sanchez-Alvarez 2023 | 2,289 | 803.50% | 00.0% | 75.5%75.2 to 81.1 | 19.5%13.9 to 19.8 | 82.3%81.0 to 84.8 | 96.2%96.2 to 97.5 |
| Sanchez-Gomez 2024 | 385 | 71.82% | 00.0% | 80.6%80.1 to 80.6 | 14.4%14.4 to 14.9 | 83.3%every run | 100.0%every run |
| Santos 2018 | 488 | 71.43% | 00.0% | 89.0%89.0 to 89.4 | 6.0%5.6 to 6.0 | 100.0%every run | 100.0%every run |
| Seghers 2022 | 1,759 | 160.91% | 00.0% | 71.4%70.2 to 81.1 | 23.6%13.9 to 24.8 | 80.0%80.0 to 86.7 | 80.0%80.0 to 100.0 |
| Sep 2021 | 147 | 2819.05% | 00.0% | 10.2%8.1 to 16.4 | 84.8%78.6 to 86.9 | 27.8%3.7 to 37.0 | 44.4%22.2 to 48.1 |
| Sepriano 2020 | 1,908 | 291.52% | 00.0% | 73.0%72.9 to 73.1 | 22.0%21.9 to 22.1 | 89.3%every run | 92.9%every run |
| Sepriano 2022 | 1,760 | 573.24% | 00.0% | 71.4%67.8 to 75.5 | 23.6%19.5 to 27.2 | 47.3%37.5 to 58.9 | 80.4%60.7 to 96.4 |
| Smid 2019 | 770 | 111.43% | 00.0% | 87.5%86.4 to 89.1 | 7.5%5.9 to 8.6 | 100.0%every run | 100.0%every run |
| Taschner 2024 | 1,323 | 342.57% | 00.0% | 21.1%20.7 to 21.1 | 73.9%73.9 to 74.3 | 42.4%33.3 to 48.5 | 57.6%54.5 to 60.6 |
| Tektonidou 2019 | 1,251 | 584.64% | 00.0% | 72.5%72.3 to 72.7 | 22.5%22.3 to 22.7 | 78.1%77.2 to 80.7 | 94.7%every run |
| Theobald 2021 | 70 | 912.86% | 00.0% | 12.6%11.2 to 28.8 | 82.4%66.2 to 83.8 | 25.0%0.0 to 37.5 | 37.5%0.0 to 62.5 |
| Toffalini 2021 | 328 | 226.71% | 00.0% | 62.2%62.2 to 67.4 | 32.8%27.6 to 32.8 | 69.0%57.1 to 71.4 | 81.0%81.0 to 85.7 |
| Tumkaya 2018 | 481 | 398.11% | 00.0% | 47.8%47.2 to 57.8 | 47.2%37.2 to 47.8 | 26.3%21.1 to 52.6 | 42.1%39.5 to 71.1 |
| van Ballegooijen 2024 | 3,295 | 872.64% | 00.0% | 66.5%65.9 to 66.6 | 28.5%28.4 to 29.1 | 68.6%67.4 to 69.8 | 83.7%83.7 to 84.9 |
| van de Schoot 2025 | 6,948 | 1001.44% | 00.0% | 83.8%83.2 to 83.9 | 11.2%11.1 to 11.8 | 94.9%93.9 to 94.9 | 99.0%every run |
| van der Valk 2021 | 362 | 4612.71% | 00.0% | 48.3%48.1 to 48.6 | 46.7%46.4 to 46.9 | 37.8%35.6 to 40.0 | 62.2%62.2 to 66.7 |
| van der Waal 2022 | 1,158 | 171.47% | 00.0% | 50.6%49.2 to 50.7 | 44.4%44.3 to 45.8 | 93.8%25.0 to 93.8 | 93.8%every run |
| van Dis 2019 | 3,154 | 200.63% | 00.0% | 55.9%54.9 to 56.1 | 39.1%38.9 to 40.1 | 52.6%15.8 to 57.9 | 73.7%73.7 to 78.9 |
| van Heugten Breurkes 2022 | 303 | 134.29% | 00.0% | 58.8%58.5 to 77.1 | 36.2%17.9 to 36.5 | 50.0%25.0 to 58.3 | 91.7%66.7 to 100.0 |
| van Hoorn 2020 | 1,154 | 151.30% | 00.0% | 62.4%62.3 to 62.4 | 32.6%32.6 to 32.7 | 85.7%78.6 to 85.7 | 85.7%every run |
| Verdugo-Castro 2022 | 1,245 | 100.80% | 00.0% | 64.9%64.8 to 71.9 | 30.1%23.1 to 30.2 | 44.4%22.2 to 55.6 | 77.8%66.7 to 77.8 |
| Viguie 2020 | 1,978 | 1045.26% | 00.0% | 81.5%81.4 to 81.8 | 13.5%13.2 to 13.6 | 89.3%87.4 to 89.3 | 96.1%every run |
| Walker 2018 | 11,735 | 640.55% | 00.0% | 71.4%71.3 to 71.4 | 23.6%23.6 to 23.7 | 82.5%every run | 92.1%every run |
| Ward 2020 | 374 | 82.14% | 00.0% | 74.3%74.0 to 74.3 | 20.7%20.7 to 21.0 | 85.7%every run | 85.7%every run |
| Wassenaar 2017 | 1,523 | 231.51% | 00.0% | 81.9%81.5 to 81.9 | 13.1%13.1 to 13.5 | 90.9%86.4 to 90.9 | 100.0%every run |
| Webers 2022 | 8,078 | 841.04% | 00.0% | 76.7%75.8 to 77.2 | 18.3%17.8 to 19.2 | 90.4%90.4 to 92.8 | 95.2%every run |
| Wijnen 2024 | 399 | 61.50% | 00.0% | 26.5%24.5 to 38.6 | 68.5%56.4 to 70.5 | 20.0%0.0 to 20.0 | 40.0%0.0 to 40.0 |
| Williamson 2023 | 258 | 51.94% | 00.0% | 88.8%88.8 to 91.9 | 6.3%3.1 to 6.3 | 100.0%every run | 100.0%every run |
| Wolters 2018 | 1,683 | 90.53% | 00.0% | -5.0%every run | 100.0%every run | 56.3%50.0 to 87.5 | 87.5%every run |
| Xu 2022 | 960 | 212.19% | 00.0% | 70.9%70.3 to 71.2 | 24.1%23.8 to 24.7 | 50.0%50.0 to 60.0 | 80.0%75.0 to 85.0 |
| Zakeri-Nasrabadi 2023a | 627 | 91.44% | 00.0% | 86.0%84.4 to 86.0 | 9.0%9.0 to 10.6 | 100.0%87.5 to 100.0 | 100.0%every run |
| Zakeri-Nasrabadi 2023b | 1,720 | 462.67% | 00.0% | 68.4%68.3 to 68.5 | 26.6%26.5 to 26.7 | 82.2%82.2 to 86.7 | 88.9%88.9 to 91.1 |
| Zanframundo 2022 | 1,334 | 403.00% | 00.0% | 69.5%69.5 to 78.4 | 25.5%16.6 to 25.5 | 69.2%69.2 to 74.4 | 94.9%94.9 to 97.4 |
| Zinsser 2022 | 125 | 32.40% | 00.0% | 90.9%85.2 to 90.9 | 4.1%4.1 to 9.8 | 100.0%every run | 100.0%every run |
5 reviews of this collection were not simulated. None was left out because of its result:
- Alunno 2020 (5,868 records in the download). The dataset's own build rule leaves it out: 1 included record remains after its open access and abstract filter, fewer than the 3 it requires.
- Chou 2003 (1,866 records in the download). The dataset's own build rule leaves it out: 1 included record remains after its open access and abstract filter, fewer than the 3 it requires.
- Demirkaya 2015 (1,032 records in the download). The dataset's own build rule leaves it out: 0 included records remain after its open access and abstract filter, fewer than the 3 it requires.
- Erer 2015 (1,249 records in the download). The dataset's own build rule leaves it out: 0 included records remain after its open access and abstract filter, fewer than the 3 it requires.
- Radjenovic 2013 (5,845 records in the download). The dataset's own build rule leaves it out: 0 included records remain after its open access and abstract filter, fewer than the 3 it requires.
Overall medians
One figure per review (its median across the 10 runs), then the median, the lowest and the highest of those figures across the reviews of a collection. Every review counts once, whatever its size.
| Collection | Reviews | Records | Work saved at 95% recall | Read to reach 95% recall | Found after 10% | Found after 20% |
|---|---|---|---|---|---|---|
| SYNERGY 1.0 | 26 | 169,288 | 65.3%3.7 to 92.7 | 29.7%2.3 to 91.3 | 74.1%16.4 to 100.0 | 89.1%25.0 to 100.0 |
| SYNERGY+ 3.0 | 114 | 168,969 | 70.6%-5.0 to 93.5 | 24.4%1.5 to 100.0 | 69.1%10.0 to 100.0 | 89.7%20.0 to 100.0 |
- SYNERGY 1.0: the ranking saved 80% or more of the reading on 7 of 26 reviews, between 50% and 80% on 9, and less than 50% on 10, of which 5 under 20%.
- SYNERGY+ 3.0: the ranking saved 80% or more of the reading on 30 of 114 reviews, between 50% and 80% on 58, and less than 50% on 26, of which 10 under 20%.
Where the ranking was weakest
The ranking does not help equally everywhere. These are the reviews with the lowest saving in each collection, lowest first, with the lowest and highest run in brackets. A figure at or below zero means the ranking did no better than reading in random order.
SYNERGY 1.0
- Chou 2003: work saved at 95% recall 3.7% (3.6 to 3.8); 15 relevant among 1,908 records; 85.7% of the relevant records found after the first 20% of the pile. Fewer than 20 relevant records, so 95% recall here means every one of them.
- Moran 2021: work saved at 95% recall 9.5% (8.8 to 9.6); 111 relevant among 5,214 records; 35.5% of the relevant records found after the first 20% of the pile.
- Donners 2021: work saved at 95% recall 11.0% (10.6 to 11.0); 15 relevant among 258 records; 85.7% of the relevant records found after the first 20% of the pile. Fewer than 20 relevant records, so 95% recall here means every one of them. 1 of its relevant records has no abstract.
- Chou 2004: work saved at 95% recall 13.0% (13.0 to 33.4); 9 relevant among 1,630 records; 25.0% of the relevant records found after the first 20% of the pile. Fewer than 20 relevant records, so 95% recall here means every one of them.
- Sep 2021: work saved at 95% recall 14.0% (13.2 to 15.4); 40 relevant among 271 records; 55.1% of the relevant records found after the first 20% of the pile.
- Menon 2022: work saved at 95% recall 21.1% (21.0 to 21.1); 74 relevant among 975 records; 72.6% of the relevant records found after the first 20% of the pile.
SYNERGY+ 3.0
- Wolters 2018: work saved at 95% recall -5.0% (every run); 9 relevant among 1,683 records; 87.5% of the relevant records found after the first 20% of the pile. Fewer than 20 relevant records, so 95% recall here means every one of them. 1 of its relevant records is labelled as not in English, in a pile where 98.7% of the records are labelled English.
- Chueca 2023: work saved at 95% recall -0.4% (-0.6 to -0.4); 16 relevant among 609 records; 66.7% of the relevant records found after the first 20% of the pile. Fewer than 20 relevant records, so 95% recall here means every one of them.
- Sanchez-Acedo 2023: work saved at 95% recall 0.0% (0.0 to 20.1); 20 relevant among 345 records; 21.1% of the relevant records found after the first 20% of the pile. Fewer than 20 relevant records, so 95% recall here means every one of them. 1 of its relevant records is labelled as not in English, in a pile where 93.6% of the records are labelled English.
- Filges 2022: work saved at 95% recall 3.9% (3.7 to 81.8); 6 relevant among 781 records; 80.0% of the relevant records found after the first 20% of the pile. Fewer than 20 relevant records, so 95% recall here means every one of them. 1 of its relevant records is labelled as not in English, in a pile where 98.2% of the records are labelled English.
- Menon 2022: work saved at 95% recall 4.9% (4.5 to 4.9); 35 relevant among 485 records; 60.3% of the relevant records found after the first 20% of the pile.
- Brons 2024: work saved at 95% recall 5.0% (3.8 to 46.2); 25 relevant among 172 records; 54.2% of the relevant records found after the first 20% of the pile.
- Oliveira 2021: work saved at 95% recall 9.8% (9.8 to 68.1); 17 relevant among 184 records; 62.5% of the relevant records found after the first 20% of the pile. Fewer than 20 relevant records, so 95% recall here means every one of them. 4 of its relevant records are labelled as not in English, in a pile where 96.2% of the records are labelled English.
- Sep 2021: work saved at 95% recall 10.2% (8.1 to 16.4); 28 relevant among 147 records; 44.4% of the relevant records found after the first 20% of the pile. Relevant records are 19.0% of this pile, so no ranking could have saved more than 77.1%.
Four things can be read from the data, and nothing more is claimed. First, in a review with fewer than 20 relevant records, 95% recall means finding every one, so a single record that the ranking places late decides the figure. That is the case for 8 of the 14 reviews listed above, and 4 of them had already found at least 80% of their relevant records within the first 20% of the pile. Over the whole benchmark it applies to 7 of the 26 reviews of SYNERGY 1.0 and 61 of the 114 reviews of SYNERGY+ 3.0. Second, where a relevant record has no abstract, the ranking has only its title to go on; the list says where that was the case. In SYNERGY 1.0, the 7 reviews with at least one relevant record lacking an abstract have a median saving of 56.8%; the other 19 have 68.1%. Third, a pile in which many records are relevant leaves little to save, whatever the ranking does; the list gives that ceiling where it is below 80%. Fourth, a relevant record written in another language than the rest of the pile shares few words with the records already found, and the ranking places it late; SYNERGY+ 3.0 labels each record's language, and the list says where a relevant record is not in English. The 19 reviews of that collection with at least one such record have a median saving of 48.1%; the other 95 have 72.5%. These two comparisons describe this run; the groups differ in other ways as well. For the weak results that none of these four covers, the data here do not show a cause, and we do not offer one.
How this run compares with our earlier report
Our benchmark report of July 2026 measured the ranking on five corpora; four of them are also in SYNERGY 1.0. The two runs differ in three ways. The earlier one fetched titles and abstracts from OpenAlex itself, while this one uses the records frozen inside the dataset. The earlier one screened in batches of 25, supplied five or six criteria terms and averaged three seeds; this one uses the tool's own retraining schedule, no criteria, and the median of 10 seeds. And the ranking code has changed since July.
| Corpus | Earlier report (mean of 3 seeds) | Earlier protocol replayed on the SYNERGY records | This run (median of 10 seeds) | Records, earlier and now |
|---|---|---|---|---|
| van de Schoot 2018 (post-traumatic stress disorder) | 76.5%seeds: 71.8, 86.8, 71.0 | 88.4%seeds: 87.5, 89.8, 87.7 | 89.2%87.4 to 89.6 | 4,502now 4,544 |
| Smid 2020 | 69.2%seeds: 63.2, 63.0, 81.5 | 83.4%seeds: 82.8, 84.5, 83.0 | 82.9%82.7 to 85.4 | 2,594now 2,627 |
| Hall 2012 | 92.7% | 92.7%seeds: 92.7, 92.7, 92.7 | 92.7%92.6 to 92.7 | 8,757now 8,793 |
| Radjenovic 2013 | 88.3% | 88.1%seeds: 88.1, 88.0, 88.1 | 88.2%88.0 to 88.4 | 5,912now 5,935 |
The third column separates the causes. It replays the earlier protocol (batches of 25, the earlier criteria terms, seeds 11, 42, 2026) on the SYNERGY records with today's ranking code. On the same records the two protocols differ by at most 0.9 percentage points across the four corpora, so the protocol accounts for little of the difference with the earlier report.
The records account for most of it. On the post-traumatic stress disorder corpus, our July retrieval returned 4,502 records, of which 1,423 (31.6%) had no abstract; the SYNERGY records number 4,544, of which 71 (1.6%) have none. On the July records, this page's protocol gives a median of 85.7% (70.9 to 87.0), and the earlier protocol with today's code gives 71.7%, 71.7%, 71.2% for the three seeds, where the report printed 71.8%, 86.8%, 71.0%. The July build of the ranking code, run again on those records, gives 71.8%, 86.8%, 71.0%, so the earlier figures reproduce and the seed that moved did so because the ranking code has changed since July. The earlier figure was a mean over runs that fall into two groups, near 71% and near 87%, depending on the seed.
On Smid 2020, the earlier report noted that roughly 24% of the records it retrieved had no abstract. Of the 2,627 SYNERGY records, 110 (4.2%) have none. On Hall 2012 and Radjenovic 2013, where none of the SYNERGY records lacks an abstract, this run is within 0.1 percentage points of the earlier report.
The practical reading: the figures on this page describe the ranking when nearly every record has an abstract. When many abstracts are missing, as in our own July retrieval, expect less.
AI screening: what the existing figure is based on
The AI screening assistant was not part of this run, and this page states no whole-review figure for it. The measured figure we publish is worded on our cite page as follows:
On ~200-record labeled samples screened against each review's eligibility criteria, the AI screening assistant kept 229 of 231 relevant records for human review (99.1% recall, 100% on three corpora), with a 3.5% average false-include rate (0% on the PTSD corpus).
That figure rests on labelled samples of about 200 records from each of five corpora, in which the relevant records were deliberately over-represented so that recall could be measured. It is a sample measurement, not a measurement on whole reviews at their real prevalence.
Limits of this benchmark
- A simulation, not a live review. The labels of the original reviewers are replayed; fatigue, changing criteria and disagreement between reviewers are not modelled, and any labelling error in the original review is carried over.
- Final inclusion labels. A record counts as relevant only if the original review included it after reading the full text. Records that passed title and abstract screening but were excluded later count as irrelevant here.
- No written criteria. The tool also uses the criteria a reviewer types in. None were supplied, so these figures describe ranking from decisions alone.
- Abstract coverage is that of the dataset. In SYNERGY 1.0, 2.8% of records have no abstract; in SYNERGY+ 3.0 none do, by the way the collection is built. Our own July retrieval of two of these corpora had far more missing abstracts and ranked worse, as the comparison above shows.
- SYNERGY+ reviews are the open access part of each pile. Records that are not open access, or have no usable abstract, are left out by the dataset's own build rule, so the number of records and the prevalence differ from the original review.
- Small reviews are decided by single records. With fewer than 20 relevant records, 95% recall means all of them.
- 10 seeds. The range across runs is shown for every review; it is a range, not a confidence interval.
- One ranking model. Only the default model was run. The alternative models a reviewer can choose in the tool were not part of this benchmark.
- Not all of it is unseen data. The ranking was developed with the corpora of our earlier report in hand, among them Hall 2012, Radjenovic 2013, Smid 2020 and van de Schoot 2018, so the results on those four are not results on data the ranking had never met.
- Your review will differ. These are measurements on public datasets, not a statement about any other pile of records.
How to rerun it
The data are public: each collection can be downloaded from its DOI above, and every review folder holds the record identifiers with their labels and the frozen records. The protocol on this page is complete: the two seed records, the random seeds 1 to 10, the retraining schedule and the formulas are all stated, and the review names in the tables are the dataset's own.
The simulation itself is part of the screening tool. On any review in which you have made include and exclude decisions, the Simulate button replays the ranking on your decided records with the same code as this benchmark. Set Runs to 10, Seed includes to 1, Seed excludes to 1 and Random seed to 1 to use the settings of this page; it reports work saved at 95% recall for every run. That is also the direct way to see what the ranking would have saved on a pile of your own.
On our side the run is three commands (download with checksum verification, simulate, assemble the results file). The results file records a fingerprint of the ranking code that produced it (c97e75de66b277c8), so a later run can be told apart from this one.
How to cite this benchmark and the data
Research Gold Team. (2026). Whole-review benchmark of the relevance ranking in the Research Gold screening tool on the SYNERGY datasets (run of 2026-11-13). Research Gold. https://researchgold.org/systematic-review-screening-tool/benchmark
De Bruin, J., Ma, Y., Ferdinands, G., Teijema, J., & Van de Schoot, R. (2023). SYNERGY - Open machine learning dataset on study selection in systematic reviews (Version 1.0) [Data set]. DataverseNL. https://doi.org/10.34894/HE6NAQ
van der Kuil, T., Westerbeek, E., de Bruin, J., Fang, Q., & van de Schoot, R. (2026). SYNERGY+: A Large-Scale, Curated Benchmark Dataset of Expert Screening Decisions for Systematic Reviews. Zenodo preprint. https://doi.org/10.5281/zenodo.23687208 (data, version 3.0: https://doi.org/10.34894/DDCVCV)
The reviews themselves belong to their authors: each review name in the tables links to its publication. For the software, see how to cite the screening tool.
The earlier report, with the AI screening and duplicate detection measurements, is archived at doi.org/10.5281/zenodo.21158870 and summarised in the measured accuracy section of the tool page. To see the ranking on your own records, open the screening tool and rank a review; the guide to ranking and screening explains each step.