Details
Original language | English |
---|---|
Pages (from-to) | 1479-1487 |
Number of pages | 9 |
Journal | Procedia Computer Science |
Volume | 246 |
Early online date | 28 Nov 2024 |
Publication status | Published - 2024 |
Event | 28th International Conference on Knowledge Based and Intelligent information and Engineering Systems, KES 2024 - Seville, Spain Duration: 11 Nov 2022 → 12 Nov 2022 |
Abstract
Understanding the significance of entities in the context of a particular search topic necessitates a special focus on linguistic, cultural, and national aspects. This paper presents an innovative study on obtaining entity importance scores within a cross-national context. While setting up an extensive crowdsourcing task pool, we asked the crowd-workers to rank given entities within a particular general-domain search topic following the best-worst scaling method. Considering two different platforms (Amazon Mechanical Turk and Toloka AI), we focus on crowdworkers from the USA and Russia, speaking English and Russian, respectively. As a result of this, we reveal a strong impact of the national factor of crowdworkers on the entity importance annotations. By highlighting differences and commonalities in the importance scores across nations, our work provides advanced insights and a dataset for future research in the field of cross-national entity ranking. Our findings and insights can be applied beyond the general domain search topics, in particular, when working with recommendations (e.g., for eCommerce items) the consideration of the cross-national factor plays a vital role on the quality.
Keywords
- Best-worst Scaling, Cross-National Context, Crowdsourcing, Entity Importance, Entity Ranking, Multilinguality
ASJC Scopus subject areas
- Computer Science(all)
- General Computer Science
Cite this
- Standard
- Harvard
- Apa
- Vancouver
- BibTeX
- RIS
In: Procedia Computer Science, Vol. 246, 2024, p. 1479-1487.
Research output: Contribution to journal › Conference article › Research › peer review
}
TY - JOUR
T1 - Evaluating Entity Importance in a Cross-National Context using Crowdsourcing and Best-Worst Scaling
AU - Perevalov, Aleksandr
AU - Abdollahi, Sara
AU - Gottschalk, Simon
AU - Both, Andreas
N1 - Publisher Copyright: © 2024 The Authors.
PY - 2024
Y1 - 2024
N2 - Understanding the significance of entities in the context of a particular search topic necessitates a special focus on linguistic, cultural, and national aspects. This paper presents an innovative study on obtaining entity importance scores within a cross-national context. While setting up an extensive crowdsourcing task pool, we asked the crowd-workers to rank given entities within a particular general-domain search topic following the best-worst scaling method. Considering two different platforms (Amazon Mechanical Turk and Toloka AI), we focus on crowdworkers from the USA and Russia, speaking English and Russian, respectively. As a result of this, we reveal a strong impact of the national factor of crowdworkers on the entity importance annotations. By highlighting differences and commonalities in the importance scores across nations, our work provides advanced insights and a dataset for future research in the field of cross-national entity ranking. Our findings and insights can be applied beyond the general domain search topics, in particular, when working with recommendations (e.g., for eCommerce items) the consideration of the cross-national factor plays a vital role on the quality.
AB - Understanding the significance of entities in the context of a particular search topic necessitates a special focus on linguistic, cultural, and national aspects. This paper presents an innovative study on obtaining entity importance scores within a cross-national context. While setting up an extensive crowdsourcing task pool, we asked the crowd-workers to rank given entities within a particular general-domain search topic following the best-worst scaling method. Considering two different platforms (Amazon Mechanical Turk and Toloka AI), we focus on crowdworkers from the USA and Russia, speaking English and Russian, respectively. As a result of this, we reveal a strong impact of the national factor of crowdworkers on the entity importance annotations. By highlighting differences and commonalities in the importance scores across nations, our work provides advanced insights and a dataset for future research in the field of cross-national entity ranking. Our findings and insights can be applied beyond the general domain search topics, in particular, when working with recommendations (e.g., for eCommerce items) the consideration of the cross-national factor plays a vital role on the quality.
KW - Best-worst Scaling
KW - Cross-National Context
KW - Crowdsourcing
KW - Entity Importance
KW - Entity Ranking
KW - Multilinguality
UR - http://www.scopus.com/inward/record.url?scp=85213374323&partnerID=8YFLogxK
U2 - 10.1016/j.procs.2024.09.595
DO - 10.1016/j.procs.2024.09.595
M3 - Conference article
AN - SCOPUS:85213374323
VL - 246
SP - 1479
EP - 1487
JO - Procedia Computer Science
JF - Procedia Computer Science
T2 - 28th International Conference on Knowledge Based and Intelligent information and Engineering Systems, KES 2024
Y2 - 11 November 2022 through 12 November 2022
ER -