Propensity Scores for Nonprobability Samples

Last updated October 30, 2026·4 min read

An opt-in panel, a river sample from a website, a volunteer registry: these samples arrive with no selection probabilities, because nobody was selected. Their members differ from the population in ways that track the outcomes being measured. Younger, more online, more engaged, more opinionated, in proportions nobody chose. Raw estimates from such samples describe the sample. The question is whether anything defensible can make them describe the population.

The construction

Pseudo-weighting borrows the probability machinery from a sample that has it:

• Take a reference probability sample: a survey drawn with known selection probabilities, carrying the same covariates.
• Stack the two samples with an indicator for which one each case came from.
• Model the indicator with logistic regression on the shared covariates. Each nonprobability case gets a predicted probability of appearing in the nonprobability sample rather than the reference.
• Invert that probability, scaled by the relative sizes, into a pseudo-weight for each nonprobability case. Cases that look like people who rarely volunteer get large weights; cases from over-volunteered profiles get small ones.
• Calibrate the pseudo-weights to population margins with raking, which is why the procedure lives inside the weighting module in Analysis Studio: propensity adjustment first, calibration second, both sets of diagnostics read.

What it corrects, and the boundary it cannot cross

The adjustment corrects selection on the covariates in the model, under the assumption that, given those covariates, appearing in the volunteer sample is unrelated to the outcome. Variables that drive both volunteering and the outcome, and that nobody measured, remain uncorrected, and no diagnostic inside the data can certify them away. That sentence is not boilerplate; it is the finding. A propensity-adjusted estimate is an estimate under a stated assumption, and the report should name the covariates and own the assumption in the same breath as the number.

Two practical consequences follow. The covariate set is the analysis, chosen on subject grounds before outcomes are examined. And the reference sample sets the ceiling: a reference with thin coverage of a population segment gives the model nothing to learn about that segment.

Diagnostics that decide whether the estimate ships

• Weight spread and design effect. Pseudo-weights often vary more than design weights. The Kish design effect prices that spread; an effect that turns 2,000 volunteers into an effective 300 is information about how much the estimate should be trusted, not an embarrassment to hide.
• Covariate balance after weighting. Weighted distributions of the model covariates in the volunteer sample should match the population or reference values closely. Where they do not, the model missed structure the subject knows about, and the covariate set goes back for work.
• Comparison against benchmarks. Where a population value for an outcome-adjacent variable is known from an independent source, the adjusted estimate should land near it. A miss there outranks model elegance.

When to use it, and when to decline

Use it when a nonprobability sample is the data that exist, a credible reference sample shares the needed covariates, and the report can carry the assumption in plain view. Decline to make population claims, and report the sample as a sample, when no reference exists, when the covariates are thin, or when the selection story plainly runs through unmeasured variables. An unadjusted number labeled honestly serves a reader better than an adjusted number whose assumption nobody believes.

How this differs from propensity adjustment for treatment effects

Same surname, different job. In causal analysis (Module 18 in the Studio), the propensity score models exposure, and weighting balances exposed and unexposed groups inside one study. Here the score models sample membership, and weighting balances a volunteer sample against its population. The machinery is shared; the claims are different, and reports should keep the two apart by name.

Run it

The propensity adjustment procedure sits inside Module 11 of Analysis Studio, after the weighting steps it builds on:

Module 11: Survey Weighting and RakingOpen in Analysis Studio →

Sources and method notes

The construction follows the pseudo-weighting literature for nonprobability samples (propensity models against a reference probability sample, followed by calibration), as implemented and gated in the Studio module specification. No worked dataset is quoted in this guide; its figures are definitional.