Details
| Original language | English |
|---|---|
| Title of host publication | JCDL 2024 - Proceedings of the 24th ACM/IEEE Joint Conference on Digital Libraries |
| Editors | Jian Wu, Xiao Hu, Terhi Nurmikko-Fuller, Sam Chu, Ruixian Yang, J. Stephen Downie |
| Publisher | Institute of Electrical and Electronics Engineers Inc. |
| ISBN (electronic) | 9798400710933 |
| Publication status | Published - 13 Mar 2025 |
| Event | 24th ACM/IEEE Joint Conference on Digital Libraries, JCDL 2024 - Hong Kong, Hong Kong Duration: 16 Dec 2024 → 20 Dec 2024 |
Abstract
Peer review is a cornerstone of the academic editorial decisionmaking process, yet it faces significant challenges. Artificial intelligence can help address these challenges, but its use raises concerns about reliability and the potential for reproducing existing biases. In this research, we employ a formal argumentation-Theoretic framework that allows for explicit analysis of arguments and their interrelations, combined with argument mining techniques to streamline the formalization of peer reviews, and resulting in a neuro-symbolic approach to dispute resolution. Our method involves identifying parties arguments in peer reviews and representing them as abstract argumentation frameworks, which facilitate dispute resolution through logical inference. We annotate these frameworks within a corpus of scientific peer reviews, achieving a high Krippendorff s alpha of 0.81. Having the annotated corpus, we implement an argument mining pipeline that integrates BERT sentence embeddings with an LSTM model, classifying sentences into three categories: Authors arguments, reviewers arguments, and nonarguments. We achieved an accuracy of 0.634 and an F1 score of 0.631, which are comparable to models trained on other datasets. However, our approach stands out by enabling the processing of the extracted argumentation with logical inference.
Keywords
- Abstract Argumentation Frameworks, Argumentation Mining, Dispute Resolution, Neuro-Symbolic AI, Scientific Peer Review, Text Annotation
ASJC Scopus subject areas
- Engineering(all)
- General Engineering
Cite this
- Standard
- Harvard
- Apa
- Vancouver
- BibTeX
- RIS
JCDL 2024 - Proceedings of the 24th ACM/IEEE Joint Conference on Digital Libraries. ed. / Jian Wu; Xiao Hu; Terhi Nurmikko-Fuller; Sam Chu; Ruixian Yang; J. Stephen Downie. Institute of Electrical and Electronics Engineers Inc., 2025. 6.
Research output: Chapter in book/report/conference proceeding › Conference contribution › Research › peer review
}
TY - GEN
T1 - Argument identification for neuro-symbolic dispute resolution in scientific peer review
AU - Baimuratov, Ildar
AU - Karpovich, Alexandr
AU - Lisanyuk, Elena
AU - Prokudin, Dmitry
N1 - Publisher Copyright: © 2024 Copyright held by the owner/author(s).
PY - 2025/3/13
Y1 - 2025/3/13
N2 - Peer review is a cornerstone of the academic editorial decisionmaking process, yet it faces significant challenges. Artificial intelligence can help address these challenges, but its use raises concerns about reliability and the potential for reproducing existing biases. In this research, we employ a formal argumentation-Theoretic framework that allows for explicit analysis of arguments and their interrelations, combined with argument mining techniques to streamline the formalization of peer reviews, and resulting in a neuro-symbolic approach to dispute resolution. Our method involves identifying parties arguments in peer reviews and representing them as abstract argumentation frameworks, which facilitate dispute resolution through logical inference. We annotate these frameworks within a corpus of scientific peer reviews, achieving a high Krippendorff s alpha of 0.81. Having the annotated corpus, we implement an argument mining pipeline that integrates BERT sentence embeddings with an LSTM model, classifying sentences into three categories: Authors arguments, reviewers arguments, and nonarguments. We achieved an accuracy of 0.634 and an F1 score of 0.631, which are comparable to models trained on other datasets. However, our approach stands out by enabling the processing of the extracted argumentation with logical inference.
AB - Peer review is a cornerstone of the academic editorial decisionmaking process, yet it faces significant challenges. Artificial intelligence can help address these challenges, but its use raises concerns about reliability and the potential for reproducing existing biases. In this research, we employ a formal argumentation-Theoretic framework that allows for explicit analysis of arguments and their interrelations, combined with argument mining techniques to streamline the formalization of peer reviews, and resulting in a neuro-symbolic approach to dispute resolution. Our method involves identifying parties arguments in peer reviews and representing them as abstract argumentation frameworks, which facilitate dispute resolution through logical inference. We annotate these frameworks within a corpus of scientific peer reviews, achieving a high Krippendorff s alpha of 0.81. Having the annotated corpus, we implement an argument mining pipeline that integrates BERT sentence embeddings with an LSTM model, classifying sentences into three categories: Authors arguments, reviewers arguments, and nonarguments. We achieved an accuracy of 0.634 and an F1 score of 0.631, which are comparable to models trained on other datasets. However, our approach stands out by enabling the processing of the extracted argumentation with logical inference.
KW - Abstract Argumentation Frameworks
KW - Argumentation Mining
KW - Dispute Resolution
KW - Neuro-Symbolic AI
KW - Scientific Peer Review
KW - Text Annotation
U2 - 10.1145/3677389.3702506
DO - 10.1145/3677389.3702506
M3 - Conference contribution
AN - SCOPUS:105001114654
BT - JCDL 2024 - Proceedings of the 24th ACM/IEEE Joint Conference on Digital Libraries
A2 - Wu, Jian
A2 - Hu, Xiao
A2 - Nurmikko-Fuller, Terhi
A2 - Chu, Sam
A2 - Yang, Ruixian
A2 - Downie, J. Stephen
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 24th ACM/IEEE Joint Conference on Digital Libraries, JCDL 2024
Y2 - 16 December 2024 through 20 December 2024
ER -