Details
| Original language | English |
|---|---|
| Pages (from-to) | 175-184 |
| Number of pages | 10 |
| Journal | ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| Volume | 10 |
| Issue number | 1 |
| Publication status | Published - 5 Dec 2023 |
| Event | 5th Geospatial Week 2023, GSW 2023 - Cairo, Egypt Duration: 2 Sept 2023 → 7 Sept 2023 |
Abstract
Collecting knowledge in the form of databases consisting of images and descriptive texts that represent objects from past centuries is a fundamental part of preserving cultural heritage. In this context, images with known information about depicted artifacts can serve as a source of information for automated methods to complete existing collections. For instance, image classifiers can provide predictions for different object properties (tasks) to semantically enrich collections. A challenge in this context is to train such classifiers given the nature of existing data: Many images do not come along with a class label for all tasks (incomplete samples) and class distributions are commonly imbalanced. In this paper, these challenges are addressed by a multi-task training strategy for a classifier based on a convolutional neural network (SilkNet) that requires images with class labels for the tasks to be learned. The proposed approach can deal with incomplete training examples, while implicitly taking interdependencies between tasks into account. Extensions of the training approach with a focus on hard examples during training as well as the use of an auxiliary feature clustering are developed to counteract problems with class imbalance. Evaluation is conducted based on a dataset consisting of images of historical silk fabrics with labels for five tasks, i.e. silk properties. A comparison of different variants of the classifier shows that the extensions of the training approach significantly improve the classifier's performance; the average F1-score is up to 5.0% larger, where the largest improvements occur with underrepresented classes of a task (up to +14.3%).
Keywords
- class imbalance, Deep learning, image classification, incomplete labelling, multi-task learning, silk heritage
ASJC Scopus subject areas
- Physics and Astronomy(all)
- Instrumentation
- Environmental Science(all)
- Environmental Science (miscellaneous)
- Earth and Planetary Sciences(all)
- Earth and Planetary Sciences (miscellaneous)
Cite this
- Standard
- Harvard
- Apa
- Vancouver
- BibTeX
- RIS
In: ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. 10, No. 1, 05.12.2023, p. 175-184.
Research output: Contribution to journal › Conference article › Research › peer review
}
TY - JOUR
T1 - Addressing Class Imbalance for Training a Multi-Task Classifier in the Context of Silk Heritage
AU - Dorozynski, M.
N1 - Funding Information: The research leading to these results is in the context of the”SILKNOW. Silk heritage in the Knowledge Society: from punched cards to big data, deep learning and visual/tangible simulations” project, which has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement No. 769504.
PY - 2023/12/5
Y1 - 2023/12/5
N2 - Collecting knowledge in the form of databases consisting of images and descriptive texts that represent objects from past centuries is a fundamental part of preserving cultural heritage. In this context, images with known information about depicted artifacts can serve as a source of information for automated methods to complete existing collections. For instance, image classifiers can provide predictions for different object properties (tasks) to semantically enrich collections. A challenge in this context is to train such classifiers given the nature of existing data: Many images do not come along with a class label for all tasks (incomplete samples) and class distributions are commonly imbalanced. In this paper, these challenges are addressed by a multi-task training strategy for a classifier based on a convolutional neural network (SilkNet) that requires images with class labels for the tasks to be learned. The proposed approach can deal with incomplete training examples, while implicitly taking interdependencies between tasks into account. Extensions of the training approach with a focus on hard examples during training as well as the use of an auxiliary feature clustering are developed to counteract problems with class imbalance. Evaluation is conducted based on a dataset consisting of images of historical silk fabrics with labels for five tasks, i.e. silk properties. A comparison of different variants of the classifier shows that the extensions of the training approach significantly improve the classifier's performance; the average F1-score is up to 5.0% larger, where the largest improvements occur with underrepresented classes of a task (up to +14.3%).
AB - Collecting knowledge in the form of databases consisting of images and descriptive texts that represent objects from past centuries is a fundamental part of preserving cultural heritage. In this context, images with known information about depicted artifacts can serve as a source of information for automated methods to complete existing collections. For instance, image classifiers can provide predictions for different object properties (tasks) to semantically enrich collections. A challenge in this context is to train such classifiers given the nature of existing data: Many images do not come along with a class label for all tasks (incomplete samples) and class distributions are commonly imbalanced. In this paper, these challenges are addressed by a multi-task training strategy for a classifier based on a convolutional neural network (SilkNet) that requires images with class labels for the tasks to be learned. The proposed approach can deal with incomplete training examples, while implicitly taking interdependencies between tasks into account. Extensions of the training approach with a focus on hard examples during training as well as the use of an auxiliary feature clustering are developed to counteract problems with class imbalance. Evaluation is conducted based on a dataset consisting of images of historical silk fabrics with labels for five tasks, i.e. silk properties. A comparison of different variants of the classifier shows that the extensions of the training approach significantly improve the classifier's performance; the average F1-score is up to 5.0% larger, where the largest improvements occur with underrepresented classes of a task (up to +14.3%).
KW - class imbalance
KW - Deep learning
KW - image classification
KW - incomplete labelling
KW - multi-task learning
KW - silk heritage
UR - http://www.scopus.com/inward/record.url?scp=85183007835&partnerID=8YFLogxK
U2 - 10.5194/isprs-annals-X-1-W1-2023-175-2023
DO - 10.5194/isprs-annals-X-1-W1-2023-175-2023
M3 - Conference article
AN - SCOPUS:85183007835
VL - 10
SP - 175
EP - 184
JO - ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences
JF - ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences
SN - 2194-9042
IS - 1
T2 - 5th Geospatial Week 2023, GSW 2023
Y2 - 2 September 2023 through 7 September 2023
ER -