Gallup’s Approach to Opt-In Sampling

WASHINGTON, D.C. — Opt-in panel sampling has become a popular and relatively inexpensive method for collecting survey data, but it’s important to understand its benefits and challenges.
Opt-in panels (also known as nonprobability panels) typically have access to hundreds of thousands (if not millions) of participants who can take online surveys. Because of their size and online access, research can usually be fielded quickly and at a lower cost than for more traditional methods, such as telephone, mail or face-to-face data collection. These panels can also be used to collect large sample sizes or to reach low-incidence populations.
However, opt-in panels do not use random selection to recruit panelists; therefore, the probability of selection into the panel — a key statistic in survey sampling and weighting — is unknown. Opt-in panels use a variety of methods to recruit respondents, such as consumer lists (for example, a list of airline loyalty rewards members), professional membership lists or online advertisements.
Most opt-in panels also allow potential respondents to join the panel without an invitation, meaning participants can seek out panels and sign up to join them. Some panels also use river or intercept sampling, whereby people who are online are directed to a survey but are not recruited into a panel.1
Opt-in recruiting and sampling are different from probability-based methods, which include address-based sampling, random-digit-dialing, random-route procedures and probability-based panels. In probability-based sampling, a sample frame is constructed for the population, meaning researchers create or access a list of everyone in the population, such as all household addresses. Information about the percentage of people in the population covered by the frame is known, and every unit in the frame has a known probability of selection.
Once the sample frame is constructed, potential respondents are randomly selected for the survey or are invited to join a probability-based panel. Therefore, unlike nonprobability samples, all members of the target population have a chance of being randomly selected and given the opportunity to participate. Despite some concerns about declining response rates from probability-based samples, all current research has consistently found that probability-based samples produce more accurate estimates than opt-in/nonprobability samples.23
Opt-in online panels have existed in the United States for at least three decades,4 but they have continued to grow in popularity because of several potential benefits:
Recently, however, there have been renewed and growing concerns about the quality of opt-in sampling. Some studies using opt-in sampling have come under scrutiny for implausible or inaccurate results.5 Other organizations, such as Pew,6 have raised concerns about data quality and fraudulent responses when opt-in panels are used, though their findings did not employ methods to attempt to remove poor-quality responses.
Despite the many advantages of opt-in panels, these concerns cannot be dismissed. Potential error stems from many aspects of opt-in design:
At Gallup, we carefully evaluate every study to determine the best method for collecting data. While there are challenges with opt-in sampling, sometimes it is the best “fit for purpose,” given the research objectives. Gallup has been using online opt-in samples for research, including for some public release studies, for more than a decade, and we regularly conduct research experiments to develop methodologies, analytic approaches and implementation strategies that can improve the quality of opt-in data. We bring this expertise to every study that we field.
Through our extensive research with opt-in samples and different panel providers around the world, we have learned that opt-in online data collection is uniquely different from probability-based data collection and must be treated as such. Our research has experimentally tested different methodological strategies for improving data quality and has found that some are more effective than others at mitigating the errors associated with opt-in samples. Over the coming months, Gallup will be releasing a series of methodology blogs that will dive deeper into our opt-in research and share what we have learned and what it may mean for your research. At a high level, our research has found that:
For more than a decade, Gallup has worked extensively with opt-in panel providers and conducted innovative methodological research to ensure we have the facts needed to properly incorporate this methodology into our work. Our research, as well as that conducted by others in the industry, has shown that opt-in panels have unique challenges that can increase the potential for error. However, through this research, we have also found that there are times when opt-in sample is the best solution, given the research objectives, and we have developed methods for minimizing bias when possible.
Finally, while this article reflects Gallup’s current recommendations related to the use of opt-in samples, our guidance may evolve as new methodological innovations are uncovered or new challenges become barriers to conducting quality opt-in research. Gallup will continue to study all issues related to the use of opt-in panels and is committed to sharing our findings.