Details
Originalsprache | Englisch |
---|---|
Titel des Sammelwerks | Proceedings of the 2024 International Conference on Information Technology for Social Good |
Erscheinungsort | New York, NY, USA |
Seiten | 225–230 |
Publikationsstatus | Veröffentlicht - 4 Sept. 2024 |
Abstract
Zitieren
- Standard
- Harvard
- Apa
- Vancouver
- BibTex
- RIS
Proceedings of the 2024 International Conference on Information Technology for Social Good. New York, NY, USA, 2024. S. 225–230.
Publikation: Beitrag in Buch/Bericht/Sammelwerk/Konferenzband › Aufsatz in Konferenzband › Forschung › Peer-Review
}
TY - GEN
T1 - Towards Modeling and Evaluating Instructional Explanations in Teacher-Student Dialogues
AU - Feldhus, Nils
AU - Anagnostopoulou, Aliki
AU - Wang, Qianli
AU - Alshomary, Milad
AU - Wachsmuth, Henning
AU - Sonntag, Daniel
AU - Möller, Sebastian
PY - 2024/9/4
Y1 - 2024/9/4
N2 - For dialogues in which teachers explain difficult concepts to students, didactics research often debates which teaching strategies lead to the best learning outcome. In this paper, we test if LLMs can reliably annotate such explanation dialogues, s.t. they could assist in lesson planning and tutoring systems. We first create a new annotation scheme of teaching acts aligned with contemporary teaching models and re-annotate a dataset of conversational explanations about communicating scientific understanding in teacher-student settings on five levels of the explainee’s expertise: ReWIRED contains three layers of acts (Teaching, Explanation, Dialogue) with increased granularity (span-level). We then evaluate language models on the labeling of such acts and find that the broad range and structure of the proposed labels is hard to model for LLMs such as GPT-3.5/-4 via prompting, but a fine-tuned BERT can perform both act classification and span labeling well. Finally, we operationalize a series of quality metrics for instructional explanations in the form of a test suite, finding that they match the five expertise levels well.1
AB - For dialogues in which teachers explain difficult concepts to students, didactics research often debates which teaching strategies lead to the best learning outcome. In this paper, we test if LLMs can reliably annotate such explanation dialogues, s.t. they could assist in lesson planning and tutoring systems. We first create a new annotation scheme of teaching acts aligned with contemporary teaching models and re-annotate a dataset of conversational explanations about communicating scientific understanding in teacher-student settings on five levels of the explainee’s expertise: ReWIRED contains three layers of acts (Teaching, Explanation, Dialogue) with increased granularity (span-level). We then evaluate language models on the labeling of such acts and find that the broad range and structure of the proposed labels is hard to model for LLMs such as GPT-3.5/-4 via prompting, but a fine-tuned BERT can perform both act classification and span labeling well. Finally, we operationalize a series of quality metrics for instructional explanations in the form of a test suite, finding that they match the five expertise levels well.1
KW - Dialogue
KW - Discourse Analysis
KW - Evaluation
KW - Explanations
U2 - 10.1145/3677525.3678665
DO - 10.1145/3677525.3678665
M3 - Conference contribution
SN - 9798400710940
SP - 225
EP - 230
BT - Proceedings of the 2024 International Conference on Information Technology for Social Good
CY - New York, NY, USA
ER -