A systematic review in psychology synthesizes empirical evidence on behavioral phenomena, psychological interventions, and cognitive processes using transparent, reproducible methods that address the unique challenges of behavioral science research. Psychology's emphasis on effect size estimation, its diverse research designs ranging from randomized controlled trials to experience sampling studies, and the ongoing replication crisis make evidence synthesis both critically important and methodologically demanding.
The Replication Crisis and Why Systematic Reviews Matter More Than Ever
The Open Science Collaboration (2015) famously found that only 36% of psychology findings replicated successfully, fundamentally shaking confidence in individual study results. This finding, published in Science, transformed how the field approaches evidence. Individual studies, no matter how well-designed, carry uncertainty. Systematic reviews and meta-analyses aggregate evidence across multiple studies, providing more reliable effect estimates and revealing whether findings hold across diverse samples, settings, and operationalizations.
The replication crisis also introduced important methodological concepts that systematic reviewers must understand. P-hacking (selective reporting of significant results), HARKing (hypothesizing after results are known), and publication bias toward statistically significant findings all inflate effect sizes in the published literature. A rigorous psychology systematic review must actively assess and correct for these distortions using tools like free funnel plot creation tool, Egger's regression, trim-and-fill analysis, and p-curve analysis.
The American Psychological Association (APA) now strongly endorses systematic reviews and meta-analyses as the highest level of evidence for psychological practice guidelines. The APA's JARS-Quant and JARS-Qual reporting standards provide detailed guidance on what to include in quantitative and qualitative synthesis reports.
Searching the Psychology Literature Effectively
Psychology research is distributed across clinical, social, developmental, cognitive, organizational, and educational subfields, each with distinct publication outlets.
PsycINFO (now PsycNET), maintained by the American Psychological Association, is the essential psychology database. Its controlled vocabulary (Thesaurus of Psychological Index Terms) uses discipline-specific descriptors like "Cognitive Behavioral Therapy," "Self Efficacy," and "Emotional Regulation" that differ from MeSH terms in PubMed. Mapping between PsycINFO descriptors and MeSH is necessary when translating your deep dive into search strategy across databases.
PubMed/MEDLINE captures clinical psychology and health psychology research published in medical journals. Embase adds European and pharmacological literature relevant to psychopharmacology reviews.
ERIC (Education Resources Information Center) is essential for educational psychology, school-based interventions, and learning science reviews. Sociological Abstracts and Social Science Citation Index capture social psychology research published outside traditional psychology journals.
ProQuest Dissertations and Theses is uniquely valuable in psychology because doctoral dissertations often contain null results not published in journals, providing a critical check against publication bias explained. Including dissertations can meaningfully change meta-analytic effect estimates.
Build your cross-database search strategy using our handy search strategy builder, which supports translation between PsycINFO descriptors and MeSH vocabulary.
Effect Sizes in Psychology: Choosing and Interpreting the Right Measure
Psychology relies heavily on effect size estimation as the primary currency of meta-analysis. The choice of effect size measure depends on the type of outcome and study design.
Cohen's d and Hedges' g are standard for comparing group means. Hedges' g corrects for small-sample bias and is preferred in meta-analysis. Jacob Cohen (1988) proposed benchmarks of 0.2 (small), 0.5 (medium), and 0.8 (large), but these should be interpreted in context, not as universal thresholds. A small effect on a large-scale public health intervention may be more practically significant than a large effect in a laboratory study.
Pearson's r (correlation coefficient) is the natural effect size for correlational studies, common in personality, social, and developmental psychology. Fisher's z transformation is recommended for meta-analytic pooling, with back-transformation for interpretation.
Odds ratios and risk ratios appear in clinical psychology reviews evaluating treatment response (e.g., remission from depression). These are standard in clinical trials and are interpretable using our our advanced effect size calculator.
The critical issue in psychology is that published effect sizes are inflated compared to true population effects. Schäfer and Schwarz (2019, Frontiers in Psychology) found that effects from studies published without pre-registration (median r = 0.36) were more than twice as large as effects from pre-registered studies (median r = 0.16), a gap consistent with publication bias and questionable research practices. Meta-analysts must account for this through sensitivity analyses and bias-correction methods.
Worked Example: Converting Reported t and F Statistics to Hedges' g
Psychology papers often report a test statistic instead of the means and standard deviations a meta-analysis needs. The conversions below follow Lakens (2013, Frontiers in Psychology) and the variance formula in Borenstein, Hedges, Higgins and Rothstein (2009, Introduction to Meta-Analysis, chapter 4). Suppose a review of a brief mindfulness intervention extracts three studies:
| Study | What the paper reports | Conversion | Cohen's d | Hedges' g (95% CI) |
|---|---|---|---|---|
| A | Two independent groups, t(58) = 2.45, 30 per arm | d = t × √(1/n₁ + 1/n₂) | 0.633 | 0.624 (0.113 to 1.136) |
| B | One-way analysis of variance with two groups, F(1, 78) = 6.30, 40 per arm | t = √F = 2.510, sign taken from the reported means, then as Study A | 0.561 | 0.556 (0.113 to 0.998) |
| C | One group measured before and after, paired t(29) = 3.10, n = 30 | Depends on the pre-post correlation | see below | see below |
Hedges' correction multiplies d by J = 1 − 3/(4(n₁ + n₂) − 9), which is 0.987 for Study A and 0.990 for Study B. The variance of d is (n₁ + n₂)/(n₁n₂) + d²/(2(n₁ + n₂)), and the variance of g is J² times that. The F conversion is valid only when the F test has one numerator degree of freedom (exactly two groups) and no covariates.
Study C is the common trap, because the same paired t gives very different answers:
- d_z = t/√n = 0.566, standardized by the standard deviation of the change scores.
- d_rm = d_z × √(2(1 − r)), standardized by the standard deviation of the raw scores: 0.566 if the pre-post correlation r is 0.5, 0.438 if r = 0.7, and 0.253 if r = 0.9.
- Entering the paired t into the independent-groups formula as if there were two groups of 30 gives 0.800, an inflated number that estimates neither quantity.
Two rules follow. Never pool d_z values with between-group d values as if they measured the same thing; choose one metric and convert every study to it. And when a paper does not report the pre-post correlation, state the value you assume and run a sensitivity analysis across plausible values, because Study C alone moves from 0.253 to 0.566 across r = 0.5 to 0.9.
Research Design Diversity in Psychology Reviews
Unlike medical systematic reviews that primarily synthesize RCTs, psychology reviews routinely include multiple research designs within a single synthesis.
Randomized controlled trials provide the strongest causal evidence for intervention effectiveness. In clinical psychology, randomized controlled trials of cognitive behavioral therapy, acceptance and commitment therapy, and pharmacological treatments follow designs familiar to clinical researchers.
Quasi-experimental designs (non-equivalent control group, interrupted time series) are common in organizational psychology, educational psychology, and community-based intervention research where randomization is impractical. These studies require different read about risk of bias assessment approaches, typically using the ROBINS-I tool rather than RoB 2.
Correlational and observational studies dominate personality, social, and developmental psychology. Reviews synthesizing these designs use correlation-based meta-analysis and must carefully distinguish cross-sectional study designs from longitudinal ones.
Experience sampling and ecological momentary assessment studies are increasing in affective science and health psychology. These designs generate intensive longitudinal data that requires specialized synthesis approaches not yet well-standardized in meta-analytic methodology.
Assessing quality across this design spectrum requires multiple tools. Our online risk of bias tool supports RoB 2 assessment for RCTs, while the free newcastle-ottawa scale calculator handles observational designs.