But What's My Margin of Error? Estimating Sampling Variance When Blending Nonprobability and Probability Samples
Submission ID: 5318

Date: Friday, 8:00 AM to 9:30 AM 
Session: Session G: F8:00 - 9:30 AM

Primary Presenter

Michael Jackson, SSRS

Additional Authors or Round Table Presenters

, ,

, ,

, ,

, ,

, ,

, ,

, ,

, ,

Abstract

The low cost of nonprobability sample sources has prompted continued research into methods of obtaining reasonable population inferences from such samples. A key challenge is that selection into nonprobability samples is related not only to demographics, but also to behavioral and attitudinal characteristics for which external benchmarks do not exist.

“Hybrid” samples, blending data from probability and nonprobability sources, offer one potential solution that balances the low cost of nonprobability sampling with the rigor of probability sampling. Such designs typically use the probability-based portion of the sample to produce “internal benchmarks” for non-demographic characteristics that are correlated with selection into the nonprobability sample. The combined probability-nonprobability sample is then weighted to these internal benchmarks in addition to traditional demographic benchmarks. This approach has been shown to reduce selection bias in some circumstances.

However, an unanswered question is how practitioners can obtain reasonable estimates of sampling variance from hybrid samples. The critical feature for reducing selection bias in such designs—weighting to benchmarks obtained from a portion of the sample itself—adds a source of variability that may be difficult to account for using methods that were originally developed for probability-only samples.

Using simulation methods and real-world hybrid datasets, this presentation will assess the accuracy of standard errors and confidence intervals produced for hybrid samples using existing off-the-shelf variance estimation methods. Simpler methods (adjustment by an unequal weighting effect and Taylor linearization) will be compared to more rigorous but harder-to-implement methods (replication). Sensitivity to key study design features (the choice of weighting variables and the size of the probability portion of the sample) will be assessed.

Results will provide insight into whether and how traditional measures of sampling variability can be applied to hybrid samples, and thus assist practitioners in weighing the benefits and challenges of such designs.