Impact of Explicit and Hidden Survey Framing on Paid Online Survey Panel Participant Screening Rates, Attentiveness, and Data Quality: An Empirical Examination
Henri Jalo, Markus Makkonen, Henri Pirkkalainen and Marko Seppänen · ICIS 2025 Proceedings (AISeL)
Telling paid online survey takers the topic up front screened out about 12 percent; hiding it screened out about 78 percent, yet answer quality was much the same.
Abstract
Researchers increasingly use paid online panels to collect survey data. However, financial incentives may lead to inappropriate participant self-selection, compromising data quality and relevance. Despite these concerns, the fundamental question of how participants are screened into surveys has received surprisingly scant attention. This study examines the effect of alternative survey framing approaches on screen-out rates, attentiveness, and data quality. Two samples were collected via Prolific: one with participant opt-in based on explicit topic disclosure (n = 1,035), and the other using hidden framing with screening criteria concealed from respondents (n = 695). Analysis revealed a substantial difference in screen-out rates (explicit: 11.87%, hidden: 78.33%), but no significant difference in data quality based on reliability or convergent and discriminant validity. However, common method variance was unevenly distributed in the explicit framing sample and evenly distributed in the hidden framing sample. Based on these findings, we offer practical recommendations for survey researchers.