The Nonresponse Challenge to Surveys and Statistics
Special Editors: Douglas S. Massey and Roger Tourangeau
Informed social policymaking relies on survey-based research. Understanding the effects of mass immigration, for example, or responding to the changing nature of the American workforce, or addressing the health and educational needs of contemporary American families requires good data that tell us about American society.
Most publicly reported statistics come from survey samples, not from tabulations of administrative data or actual population counts. For example, almost all of what people commonly refer to as “census data” does not come from an actual enumeration of the U.S. population but from a sample survey that is administered along with the census—the 2010 U.S. Census contained just four questions about the household and seven questions about each household member—all other data now come from the American Community Survey (ACS), which samples about 2 million households. In addition to the census, other federal sample surveys include the Current Population Survey (which produces monthly labor force statistics), the National Survey of Family Growth (birth and childbearing statistics), and the National Health Interview Survey (health statistics). The federal government also supports other surveys indirectly through grants, including the New Immigrant Survey, the Panel Study of Income Dynamics, and the American National Election Study.
But survey researchers are increasingly unable to get people to respond to surveys. This is a real worry because nonresponse can lead to biased research and because nonresponse poses a significant threat to the federal statistical system in its entirety.
This volume of The ANNALS abundantly highlights the critical importance of surveys to contemporary society, addresses the causes and consequences of rising survey nonresponse, and explores ways to address the challenges that nonresponse brings.
Key Observations and Findings
- High rates of nonresponse do not necessarily translate into bias. Research in this volume shows how response rates by themselves are poor indicators of bias (in some cases, raising response rates can even increase bias). But rising nonresponse increases the likelihood of bias, and survey nonresponse is increasing at an alarming rate.
- The survey nonresponse problem is likely going to continue to worsen, and survey costs are likely going to continue to rise. We show that nonresponse is related with the frequency of single-person households, the prevalence of households without children, and long commute times—all seemingly endemic to social life today. The increase in nonresponse is driven by powerful generational changes: younger cohorts are associated with low response and less comfortable with communication techniques favored by survey researchers, such as telephone surveys and face-to-face interviews.
- The simplest way to address nonresponse is to reward respondents financially for their participation. Evidence clearly indicates that financial incentives increase response rates, but this also increases costs.
- Reliance on nonsurvey data can assist in getting quality survey-based estimates. Using administrative records and “paradata” to strengthen and extend survey research are critical ways to adjust survey estimates and reduce budget pressures on the survey organizations. It may also reduce the level of survey fatigue in the general public. Interestingly, the most important limitation on the potential use of administrative systems as data for nonresponse adjustment is legal: federal legislation prohibits the public release of the data collected by the federal government, so it can only be done in-house by the Census Bureau or some other authorized agency.
- Rising costs and likely federal budget cuts will probably mean serious efforts to overhaul one or more major federal surveys (instruments that have not undergone major redesigns in decades) or to consolidate federal data collection into fewer surveys (or both). The current model for many high-quality but costly surveys may soon become unsustainable. Research in this volume shows that response rates for the most difficult and expensive kinds of surveys—longitudinal studies that follow the same respondents and reinterview them over time—have not seen the same increases in nonresponse rates as have been observed for one-time surveys.
- The volume is replete with technical details of how to maximize the likelihood of response and compensate for nonresponse. One promising analytic technique highlighted in the volume is the multi-level integrated database approach, which employs multiple databases to collect as much information as possible about a target population to maximize the accuracy of survey estimates. Data systems integration is definitely a direction in which the field is moving.
Perhaps the greatest threat to the future of survey data is the lack of public recognition of the importance of statistical and scientific surveys in the world today. It is probably time for an industry-wide effort to improve the image of survey research and, critically, to differentiate legitimate social scientific surveys from the onslaught of unwanted solicitations that most Americans are trying so hard to fend off.
Protecting the integrity of survey research also requires better efforts to educate our political leadership and society writ large. Must all become better educated to the critical importance of social surveys in contemporary society, not only because they provide basic information to inform citizen-voters in a democratic society but also because they generate key knowledge for American businesses that enable us to compete effectively in a global, knowledge-based economy. We offer this volume as a first step in a broader and more concerted effort to educate policy-makers and the public about the key importance of social surveys to the healthy functioning of postindustrial society.
Issue Conclusion:
Where Do We Go from Here? Nonresponse and Social Measurement
Written Statement of Paul Emrath:
