TY - GEN

T1 - What Is Unclear?

T2 - 32nd ACM Conference on Hypertext and Social Media, HT 2021

AU - Nouri, Zahra

AU - Gadiraju, Ujwal

AU - Engels, Gregor

AU - Wachsmuth, Henning

PY - 2021/8/29

Y1 - 2021/8/29

N2 - Designing tasks clearly to facilitate accurate task completion is a challenging endeavor for requesters on crowdsourcing platforms. Prior research shows that inexperienced requesters fail to write clear and complete task descriptions which directly leads to low quality submissions from workers. By complementing existing works that have aimed to address this challenge, in this paper we study whether clarity flaws in task descriptions can be identified automatically using natural language processing methods. We identify and synthesize seven clarity flaws in task descriptions that are grounded in relevant literature. We build both BERT-based and feature-based binary classifiers, in order to study the extent to which clarity flaws in task descriptions can be computationally assessed, and understand textual properties of descriptions that affect task clarity. Through a crowdsourced study, we collect annotations of clarity flaws in 1332 real task descriptions. Using this dataset, we evaluate several configurations of the classifiers. Our results indicate that nearly all the clarity flaws in task descriptions can be assessed reasonably by the classifiers. We found that the content, style, and readability of tasks descriptions are particularly important in shaping their clarity. This work has important implications on the design of tools to help requesters in improving task clarity on crowdsourcing platforms. Flaw-specific properties can provide for valuable guidance in improving task descriptions.

AB - Designing tasks clearly to facilitate accurate task completion is a challenging endeavor for requesters on crowdsourcing platforms. Prior research shows that inexperienced requesters fail to write clear and complete task descriptions which directly leads to low quality submissions from workers. By complementing existing works that have aimed to address this challenge, in this paper we study whether clarity flaws in task descriptions can be identified automatically using natural language processing methods. We identify and synthesize seven clarity flaws in task descriptions that are grounded in relevant literature. We build both BERT-based and feature-based binary classifiers, in order to study the extent to which clarity flaws in task descriptions can be computationally assessed, and understand textual properties of descriptions that affect task clarity. Through a crowdsourced study, we collect annotations of clarity flaws in 1332 real task descriptions. Using this dataset, we evaluate several configurations of the classifiers. Our results indicate that nearly all the clarity flaws in task descriptions can be assessed reasonably by the classifiers. We found that the content, style, and readability of tasks descriptions are particularly important in shaping their clarity. This work has important implications on the design of tools to help requesters in improving task clarity on crowdsourcing platforms. Flaw-specific properties can provide for valuable guidance in improving task descriptions.

KW - BERT-based binary classification

KW - crowdsourcing

KW - feature-based binary classification

KW - task clarity assessment

KW - task design

KW - unclear task descriptions

UR - http://www.scopus.com/inward/record.url?scp=85114834391&partnerID=8YFLogxK

U2 - 10.1145/3465336.3475109

DO - 10.1145/3465336.3475109

M3 - Conference contribution

AN - SCOPUS:85114834391

SN - 9781450385510

SP - 165

EP - 175

BT - HT 2021 - Proceedings of the 32nd ACM Conference on Hypertext and Social Media

Y2 - 30 August 2021 through 2 September 2021

ER -