Evaluating Entity Importance in a Cross-National Context using Crowdsourcing and Best-Worst Scaling

Research output: Contribution to journalConference articleResearchpeer review

Authors

Research Organisations

External Research Organisations

  • Leipzig University of Applied Sciences
  • DATEV eG
View graph of relations

Details

Original languageEnglish
Pages (from-to)1479-1487
Number of pages9
JournalProcedia Computer Science
Volume246
Early online date28 Nov 2024
Publication statusPublished - 2024
Event28th International Conference on Knowledge Based and Intelligent information and Engineering Systems, KES 2024 - Seville, Spain
Duration: 11 Nov 202212 Nov 2022

Abstract

Understanding the significance of entities in the context of a particular search topic necessitates a special focus on linguistic, cultural, and national aspects. This paper presents an innovative study on obtaining entity importance scores within a cross-national context. While setting up an extensive crowdsourcing task pool, we asked the crowd-workers to rank given entities within a particular general-domain search topic following the best-worst scaling method. Considering two different platforms (Amazon Mechanical Turk and Toloka AI), we focus on crowdworkers from the USA and Russia, speaking English and Russian, respectively. As a result of this, we reveal a strong impact of the national factor of crowdworkers on the entity importance annotations. By highlighting differences and commonalities in the importance scores across nations, our work provides advanced insights and a dataset for future research in the field of cross-national entity ranking. Our findings and insights can be applied beyond the general domain search topics, in particular, when working with recommendations (e.g., for eCommerce items) the consideration of the cross-national factor plays a vital role on the quality.

Keywords

    Best-worst Scaling, Cross-National Context, Crowdsourcing, Entity Importance, Entity Ranking, Multilinguality

ASJC Scopus subject areas

Cite this

Evaluating Entity Importance in a Cross-National Context using Crowdsourcing and Best-Worst Scaling. / Perevalov, Aleksandr; Abdollahi, Sara; Gottschalk, Simon et al.
In: Procedia Computer Science, Vol. 246, 2024, p. 1479-1487.

Research output: Contribution to journalConference articleResearchpeer review

Perevalov A, Abdollahi S, Gottschalk S, Both A. Evaluating Entity Importance in a Cross-National Context using Crowdsourcing and Best-Worst Scaling. Procedia Computer Science. 2024;246:1479-1487. Epub 2024 Nov 28. doi: 10.1016/j.procs.2024.09.595
Perevalov, Aleksandr ; Abdollahi, Sara ; Gottschalk, Simon et al. / Evaluating Entity Importance in a Cross-National Context using Crowdsourcing and Best-Worst Scaling. In: Procedia Computer Science. 2024 ; Vol. 246. pp. 1479-1487.
Download
@article{2cf0aa5b7b424a8fb7bffc1f51fc61d5,
title = "Evaluating Entity Importance in a Cross-National Context using Crowdsourcing and Best-Worst Scaling",
abstract = "Understanding the significance of entities in the context of a particular search topic necessitates a special focus on linguistic, cultural, and national aspects. This paper presents an innovative study on obtaining entity importance scores within a cross-national context. While setting up an extensive crowdsourcing task pool, we asked the crowd-workers to rank given entities within a particular general-domain search topic following the best-worst scaling method. Considering two different platforms (Amazon Mechanical Turk and Toloka AI), we focus on crowdworkers from the USA and Russia, speaking English and Russian, respectively. As a result of this, we reveal a strong impact of the national factor of crowdworkers on the entity importance annotations. By highlighting differences and commonalities in the importance scores across nations, our work provides advanced insights and a dataset for future research in the field of cross-national entity ranking. Our findings and insights can be applied beyond the general domain search topics, in particular, when working with recommendations (e.g., for eCommerce items) the consideration of the cross-national factor plays a vital role on the quality.",
keywords = "Best-worst Scaling, Cross-National Context, Crowdsourcing, Entity Importance, Entity Ranking, Multilinguality",
author = "Aleksandr Perevalov and Sara Abdollahi and Simon Gottschalk and Andreas Both",
note = "Publisher Copyright: {\textcopyright} 2024 The Authors.; 28th International Conference on Knowledge Based and Intelligent information and Engineering Systems, KES 2024 ; Conference date: 11-11-2022 Through 12-11-2022",
year = "2024",
doi = "10.1016/j.procs.2024.09.595",
language = "English",
volume = "246",
pages = "1479--1487",

}

Download

TY - JOUR

T1 - Evaluating Entity Importance in a Cross-National Context using Crowdsourcing and Best-Worst Scaling

AU - Perevalov, Aleksandr

AU - Abdollahi, Sara

AU - Gottschalk, Simon

AU - Both, Andreas

N1 - Publisher Copyright: © 2024 The Authors.

PY - 2024

Y1 - 2024

N2 - Understanding the significance of entities in the context of a particular search topic necessitates a special focus on linguistic, cultural, and national aspects. This paper presents an innovative study on obtaining entity importance scores within a cross-national context. While setting up an extensive crowdsourcing task pool, we asked the crowd-workers to rank given entities within a particular general-domain search topic following the best-worst scaling method. Considering two different platforms (Amazon Mechanical Turk and Toloka AI), we focus on crowdworkers from the USA and Russia, speaking English and Russian, respectively. As a result of this, we reveal a strong impact of the national factor of crowdworkers on the entity importance annotations. By highlighting differences and commonalities in the importance scores across nations, our work provides advanced insights and a dataset for future research in the field of cross-national entity ranking. Our findings and insights can be applied beyond the general domain search topics, in particular, when working with recommendations (e.g., for eCommerce items) the consideration of the cross-national factor plays a vital role on the quality.

AB - Understanding the significance of entities in the context of a particular search topic necessitates a special focus on linguistic, cultural, and national aspects. This paper presents an innovative study on obtaining entity importance scores within a cross-national context. While setting up an extensive crowdsourcing task pool, we asked the crowd-workers to rank given entities within a particular general-domain search topic following the best-worst scaling method. Considering two different platforms (Amazon Mechanical Turk and Toloka AI), we focus on crowdworkers from the USA and Russia, speaking English and Russian, respectively. As a result of this, we reveal a strong impact of the national factor of crowdworkers on the entity importance annotations. By highlighting differences and commonalities in the importance scores across nations, our work provides advanced insights and a dataset for future research in the field of cross-national entity ranking. Our findings and insights can be applied beyond the general domain search topics, in particular, when working with recommendations (e.g., for eCommerce items) the consideration of the cross-national factor plays a vital role on the quality.

KW - Best-worst Scaling

KW - Cross-National Context

KW - Crowdsourcing

KW - Entity Importance

KW - Entity Ranking

KW - Multilinguality

UR - http://www.scopus.com/inward/record.url?scp=85213374323&partnerID=8YFLogxK

U2 - 10.1016/j.procs.2024.09.595

DO - 10.1016/j.procs.2024.09.595

M3 - Conference article

AN - SCOPUS:85213374323

VL - 246

SP - 1479

EP - 1487

JO - Procedia Computer Science

JF - Procedia Computer Science

T2 - 28th International Conference on Knowledge Based and Intelligent information and Engineering Systems, KES 2024

Y2 - 11 November 2022 through 12 November 2022

ER -

By the same author(s)