Untangling Representation Error in Address-based Sample Surveys
Submission ID: 5603

Date: Thursday, 10:15 AM to 11:45 AM 
Session: Session D: T10:15 - 11:45 AM

Primary Presenter

Robyn Rapoport, SSRS

Additional Authors or Round Table Presenters

Cameron McPhee, SSRS ,

Robert Manley, SSRS ,

, ,

, ,

, ,

, ,

, ,

, ,

Abstract

Historically, with dual frame RDD surveys, landlines have tended to underrepresent males, since women are more likely to answer the phone in households while we often obtain disproportionately more interviews with men on cellphones.  Interestingly, it seems that some ABS studies using a push-to-web or mixed-mode strategy provide a near-Census split of males and females, while other ABS studies are underrepresenting males in their responding samples.

It is well documented that quasi-random within-household selection methods, such as the last birthday method, are not always complied with (e.g., Olson, Stange, and Smyth, 2014, Olson and Smyth, 2017), leading to one source of potential bias.  In ABS studies this bias may not be systematic and may be directly related to the survey topic. For example, out of 10 ABS push-to-web studies administered by SSRS in 2021, the unweighted bias by gender ranged from near zero to as high as 17 percentage points, with the largest discrepancy appearing with surveys addressing health and health-related topics.

This paper seeks to better understand the drivers of these biases. We will examine a set of national and regional ABS surveys administered within the previous three years to determine if the respondent gender bias can be explained by study design factors including survey topic, mailing method, within-household selection protocol, as well as potential interactions between these study characteristics. By disentangling some of the complexities of this type of error, we will be able to point to some important levers for improving representativeness in ABS studies.

As household surveys continue to shift away from random-digit dial sample frames and interviewer-administered data collection modes towards address-based sample frames and self-administered modes, representation error seems to be changing as well. One notable example of this is gender. Historically, with dual frame RDD surveys, landlines have tended to underrepresent males, since women are more likely to answer the phone in households while we often obtain disproportionately more interviews with men on cellphones.  Interestingly, it seems that some ABS studies using a push-to-web or mixed-mode strategy provide a near-Census split of males and females, while other ABS studies are underrepresenting males in their responding samples.

It is well documented that quasi-random within-household selection methods, such as the last birthday method, are not always complied with (e.g., Olson, Stange, and Smyth, 2014, Olson and Smyth, 2017), leading to one source of potential bias.  In ABS studies this bias may not be systematic and may be directly related to the survey topic. For example, out of 10 ABS push-to-web studies administered by SSRS in 2021, the unweighted bias by gender ranged from near zero to as high as 17 percentage points, with the largest discrepancy appearing with surveys addressing health and health-related topics.

This paper seeks to better understand the drivers of these biases. We will examine a set of national and regional ABS surveys administered within the previous three years to determine if the respondent gender bias can be explained by study design factors including survey topic, mailing method, within-household selection protocol, as well as potential interactions between these study characteristics. By disentangling some of the complexities of this type of error, we will be able to point to some important levers for improving representativeness in ABS studies.