Propensity Score Matching: Estimating Effects from Observational Data

Propensity score matching is a method for estimating the effect of a treatment or exposure from observational data by pairing treated and untreated participants who had a similar probability of receiving the treatment. The propensity score is that probability, the likelihood of being treated given a person's measured characteristics, and matching on it makes the treated and comparison groups resemble each other on those characteristics. The goal is to approximate the balance a randomized trial would have produced when randomization was never possible.

In a randomized trial, treatment is assigned by chance, so the groups are comparable on everything, measured or not. In observational research, sicker patients may be more likely to receive a treatment, which confounds the comparison: any difference in outcomes mixes the treatment effect with the baseline difference. Propensity score matching addresses the measured part of that problem by constructing comparison groups that look alike on the variables you observed.

When to use propensity score matching

Use propensity score matching when you want to estimate a treatment effect but cannot randomize, which is the normal situation in registry studies, electronic health record analyses, and many cohort designs. It is most appropriate when you have measured the important confounders, when treated and untreated groups overlap enough to find matches, and when a reviewer or your own design demands that the comparison groups be made comparable before outcomes are analyzed.

It is the wrong tool when the variables that drive both treatment and outcome are unmeasured. Matching can only balance what you observed. If an important confounder was never recorded, the estimate remains biased no matter how good the balance looks on paper, a limitation that must be stated plainly rather than hidden.

How the method works

The procedure has a clear sequence. First, model the probability of treatment, usually with a logistic regression that predicts treatment status from the baseline covariates, producing a propensity score for every participant. Second, match treated participants to untreated ones with similar scores, often one-to-one within a tolerance called a caliper, though one-to-many and other schemes exist. Third, and most important, check covariate balance in the matched sample to confirm the groups now resemble each other. Fourth, estimate the treatment effect within the matched sample.

The third step is where careful analysts spend their attention. Balance is assessed with standardized mean differences for each covariate, with values below 0.10 generally taken to indicate adequate balance. Comparing balance before and after matching, rather than relying on the matching procedure to have worked, is the standard a reviewer expects.

Matching is one of several propensity score approaches

Matching is the most intuitive way to use a propensity score, but not the only one. Inverse probability weighting keeps the whole sample and weights each person by the inverse of their probability of the treatment they received, which can be more efficient than discarding unmatched cases. Stratification groups participants into bands of similar scores. Covariate adjustment includes the score directly in an outcome model. Each makes different tradeoffs between bias, efficiency, and the population the estimate generalizes to, and the right choice depends on the data and the question. Weighing these options is exactly what our managed biostatistics consulting does for observational studies.

Reporting standards

A credible propensity score analysis reports the variables in the treatment model, the matching algorithm and caliper, the number of treated participants who could not be matched and were therefore dropped, the balance achieved on every covariate, and a sensitivity analysis exploring how strong an unmeasured confounder would have to be to overturn the result. Observational analyses are also expected to follow established reporting guidance for their study design, the kind of alignment our research methodology support builds in from the start.

Common mistakes

1. Treating matching as a cure for all confounding. It balances measured covariates only. Unmeasured confounders remain, and the limitation must be stated.
2. Reporting an effect without showing balance. A matched analysis is only credible if you demonstrate that the groups became comparable.
3. Ignoring the cases you dropped. Unmatched treated participants change who the estimate applies to, which affects generalizability.
4. Putting the outcome in the propensity model. The treatment model should predict treatment from baseline covariates, not from anything measured after or caused by treatment.
5. Skipping sensitivity analysis. Without it, you cannot say how fragile the conclusion is to the confounders you could not measure.

Which effect are you estimating? Decide the estimand first

Before any matching, decide which average treatment effect you are after, because different propensity methods target different ones. One-to-one matching of treated to untreated participants usually estimates the average treatment effect on the treated: the effect in the kind of people who actually received the treatment. Inverse probability weighting can instead estimate the average treatment effect in the whole population, or, with overlap weighting, the effect in the subpopulation where treated and untreated participants genuinely coexist. These are different numbers that answer different questions, and they can diverge when the effect varies across people. A reviewer who asks "the effect in whom?" is asking you to name your estimand. State it, and choose the method that targets it, rather than reporting whatever the default produced.

Choosing and tuning the match

The treatment model gives every participant a score, but the matching scheme is a separate set of decisions that materially change the answer. The main choices are:

• Nearest-neighbour versus optimal matching. Greedy nearest-neighbour matching is fast; optimal matching minimises the total within-pair distance across the whole sample and usually balances better.
• With or without replacement. Matching with replacement lets a good control serve several treated participants, lowering bias when controls are scarce but requiring frequency weights in the analysis.
• The caliper. Restricting matches to scores within a tolerance avoids poor pairs. A widely cited default is a caliper of 0.2 times the standard deviation of the logit of the propensity score (Austin, 2011).
• The ratio. One-to-many matching (one treated to several controls) lowers variance but raises bias as the later, less similar controls are admitted.

Match on the logit of the propensity score rather than the raw probability, because distances are more stable away from the boundaries of zero and one.

Balance is judged on standardized differences, not p-values

The single most common reviewer objection is a balance check done with significance tests. A t-test or chi-square on each covariate confounds balance with sample size: after matching shrinks the sample, real imbalances can stop being "significant" without having gone away. Judge balance with the standardized mean difference for every covariate, treating values below 0.10 as adequate, and check higher moments and interactions, not just means, because two groups can share a mean and differ in spread. A "love plot" showing the standardized differences before and after matching is the expected figure. The R package cobalt produces exactly this.

Getting the standard errors right

Matched data are not independent: the members of a pair were deliberately made similar, so a naive standard error that assumes independent observations is wrong. Account for the matched structure with an estimator built for it (the Abadie-Imbens standard error for matching), a cluster-robust (sandwich) variance that clusters on the matched set, or a bootstrap that resamples pairs rather than individuals. Matching with replacement additionally requires the frequency weights to enter the variance. Skipping this step produces confidence intervals that are too narrow and a result that looks more certain than the data support.

Positivity, and the doubly robust alternative

Every propensity method rests on positivity (also called common support): each kind of participant must have a non-zero chance of either treatment. Where treated and untreated participants do not overlap, no amount of weighting fixes the gap; inspect the overlap of the score distributions and consider restricting to the region of common support or using overlap weights. Inverse probability weighting is sensitive here, because near-violations create extreme weights that a single participant can dominate; use stabilized weights and consider trimming or truncating the largest ones.

The most robust modern approach does not bet everything on the treatment model. Doubly robust estimators, augmented inverse probability weighting and targeted maximum likelihood estimation, combine the propensity model with an outcome model and stay consistent if either model is correctly specified, not necessarily both. For high-dimensional or hard-to-model confounding, these methods (with cross-fitting when machine learning estimates the nuisance models) are now the defensible default.

Quantifying how much unmeasured confounding would matter

Because matching only balances measured covariates, the honest companion to any estimate is a formal sensitivity analysis for the confounders you could not measure. Two named tools are expected. Rosenbaum bounds report how large a hidden bias (the factor by which an unmeasured confounder would have to change the odds of treatment) would have to be before your conclusion flips. The E-value (VanderWeele and Ding, 2017) reports the minimum strength of association, on the risk-ratio scale, that an unmeasured confounder would need with both treatment and outcome to explain away the observed effect. A large E-value is reassuring; a small one tells the reader the result is fragile.

A worked analysis in R

library(MatchIt)   # matching
library(cobalt)    # balance diagnostics and love plots
library(survey)    # correct standard errors via weights

# 1. Estimate the propensity score and match 1:1 on the logit, with a caliper
m <- matchit(treat ~ age + sex + comorbidity + baseline_severity,
             data = d, method = 'nearest', distance = 'glm',
             link = 'logit', caliper = 0.2, ratio = 1)

# 2. Judge balance on standardized differences (not p-values)
bal.tab(m, un = TRUE, stats = 'mean.diffs')
love.plot(m, threshold = 0.1)

# 3. Estimate the effect in the matched sample with robust standard errors
md  <- match.data(m)
des <- svydesign(ids = ~subclass, weights = ~weights, data = md)
svyglm(outcome ~ treat, design = des, family = quasibinomial)

Bringing it together

Propensity score matching lets observational data support a more credible causal comparison by making treated and untreated groups resemble each other on measured characteristics. Done well, it includes a transparent treatment model, demonstrated balance, honest accounting of dropped cases, and a sensitivity analysis for unmeasured confounding. Done poorly, it dresses a biased comparison in technical language.

If a reviewer has asked for propensity methods, or your observational study needs a defensible treatment effect, our biostatistics analysts build and validate the analysis in reproducible code. Request a quote and tell us about your dataset.