Details
| Original language | English |
|---|---|
| Pages (from-to) | 1295-1317 |
| Number of pages | 23 |
| Journal | Journal of Survey Statistics and Methodology |
| Volume | 12 |
| Issue number | 5 |
| Early online date | 2 Aug 2024 |
| Publication status | Published - Nov 2024 |
Abstract
Cognitive interviewing in the form of probing is key for developing methodologically sound survey questions. For a long time, probing was tied to the laboratory setting, making it difficult to achieve large sample sizes and creating a time-intensive undertaking for both researchers and participants. Web surveys paved the way for administering probing questions over the Internet in a time- and cost-efficient manner. In so-called web probing studies, respondents first answer a question and then they receive one or more open-ended questions about their response process, with requests for written answers. However, participants frequently provide very short or no answers at all to open-ended questions, in part because answering questions in writing is tedious. This is especially the case when the web survey is completed via a smartphone with a virtual on-screen keypad that shrinks the viewing space. In this study, we examine whether the problem of short and uninterpretable answers in web probing studies can be mitigated by asking respondents to complete the web survey on a smartphone and to record their answers via the built-in microphone. We conducted an experiment in a smartphone survey (N= 1,001), randomizing respondents to different communication modes (written or oral) for answering two comprehension probes about two questions on national identity and citizenship. The results indicate that probes with requests for oral answers produce four to five times more nonresponse than their written counterparts. However, oral answers contain about three times as many words, include about 0.3 more themes (first probing question only), and the proportion of clearly interpretable answers is about 6 percentage points higher (for the first probing question only). Nonetheless, both communication modes result in similar themes mentioned by respondents.
Keywords
- Cognitive pretesting, Experiment, Smartphone survey, Survey question design, Voice recording, Web probing
ASJC Scopus subject areas
- Mathematics(all)
- Statistics and Probability
- Social Sciences(all)
- Social Sciences (miscellaneous)
- Decision Sciences(all)
- Statistics, Probability and Uncertainty
- Mathematics(all)
- Applied Mathematics
Cite this
- Standard
- Harvard
- Apa
- Vancouver
- BibTeX
- RIS
In: Journal of Survey Statistics and Methodology, Vol. 12, No. 5, 11.2024, p. 1295-1317.
Research output: Contribution to journal › Article › Research › peer review
}
TY - JOUR
T1 - Innovating Web Probing
T2 - Comparing Written and Oral Answers to Open-Ended Probing Questions in a Smartphone Survey
AU - Lenzner, Timo
AU - Höhne, Jan Karem
AU - Gavras, Konstantin
N1 - Publisher Copyright: © The Author(s) 2024.
PY - 2024/11
Y1 - 2024/11
N2 - Cognitive interviewing in the form of probing is key for developing methodologically sound survey questions. For a long time, probing was tied to the laboratory setting, making it difficult to achieve large sample sizes and creating a time-intensive undertaking for both researchers and participants. Web surveys paved the way for administering probing questions over the Internet in a time- and cost-efficient manner. In so-called web probing studies, respondents first answer a question and then they receive one or more open-ended questions about their response process, with requests for written answers. However, participants frequently provide very short or no answers at all to open-ended questions, in part because answering questions in writing is tedious. This is especially the case when the web survey is completed via a smartphone with a virtual on-screen keypad that shrinks the viewing space. In this study, we examine whether the problem of short and uninterpretable answers in web probing studies can be mitigated by asking respondents to complete the web survey on a smartphone and to record their answers via the built-in microphone. We conducted an experiment in a smartphone survey (N= 1,001), randomizing respondents to different communication modes (written or oral) for answering two comprehension probes about two questions on national identity and citizenship. The results indicate that probes with requests for oral answers produce four to five times more nonresponse than their written counterparts. However, oral answers contain about three times as many words, include about 0.3 more themes (first probing question only), and the proportion of clearly interpretable answers is about 6 percentage points higher (for the first probing question only). Nonetheless, both communication modes result in similar themes mentioned by respondents.
AB - Cognitive interviewing in the form of probing is key for developing methodologically sound survey questions. For a long time, probing was tied to the laboratory setting, making it difficult to achieve large sample sizes and creating a time-intensive undertaking for both researchers and participants. Web surveys paved the way for administering probing questions over the Internet in a time- and cost-efficient manner. In so-called web probing studies, respondents first answer a question and then they receive one or more open-ended questions about their response process, with requests for written answers. However, participants frequently provide very short or no answers at all to open-ended questions, in part because answering questions in writing is tedious. This is especially the case when the web survey is completed via a smartphone with a virtual on-screen keypad that shrinks the viewing space. In this study, we examine whether the problem of short and uninterpretable answers in web probing studies can be mitigated by asking respondents to complete the web survey on a smartphone and to record their answers via the built-in microphone. We conducted an experiment in a smartphone survey (N= 1,001), randomizing respondents to different communication modes (written or oral) for answering two comprehension probes about two questions on national identity and citizenship. The results indicate that probes with requests for oral answers produce four to five times more nonresponse than their written counterparts. However, oral answers contain about three times as many words, include about 0.3 more themes (first probing question only), and the proportion of clearly interpretable answers is about 6 percentage points higher (for the first probing question only). Nonetheless, both communication modes result in similar themes mentioned by respondents.
KW - Cognitive pretesting
KW - Experiment
KW - Smartphone survey
KW - Survey question design
KW - Voice recording
KW - Web probing
UR - http://www.scopus.com/inward/record.url?scp=85209259012&partnerID=8YFLogxK
U2 - 10.1093/jssam/smae031
DO - 10.1093/jssam/smae031
M3 - Article
AN - SCOPUS:85209259012
VL - 12
SP - 1295
EP - 1317
JO - Journal of Survey Statistics and Methodology
JF - Journal of Survey Statistics and Methodology
SN - 2325-0984
IS - 5
ER -