Details
| Originalsprache | Englisch |
|---|---|
| Seiten (von - bis) | 153-161 |
| Seitenumfang | 9 |
| Fachzeitschrift | ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| Ausgabenummer | X-2-2024 |
| Publikationsstatus | Veröffentlicht - 10 Juni 2024 |
| Veranstaltung | 2024 ISPRS TC II Mid-term Symposium on The Role of Photogrammetry for a Sustainable World - Las Vegas, USA / Vereinigte Staaten Dauer: 11 Juni 2024 → 14 Juni 2024 |
Abstract
Panoptic segmentation unifies semantic and instance segmentation and thus delivers a semantic class label and, for so-called thing classes, also an instance label per pixel. The differentiation of distinct objects of the same class with a similar appearance is particularly challenging and frequently causes such objects to be incorrectly assigned to a single instance. In the present work, we demonstrate that information on the 3D geometry of the observed scene can be used to mitigate this issue: We present a novel CNN-based method for panoptic segmentation which processes RGB images and depth maps given as input in separate network branches and fuses the resulting feature maps in a late fusion manner. Moreover, we propose a new depth-aware dice loss term which penalises the assignment of pixels to the same thing instance based on the difference between their associated distances to the camera. Experiments carried out on the Cityscapes dataset show that the proposed method reduces the number of objects that are erroneously merged into one thing instance and outperforms the method used as basis by +2.2% in terms of panoptic quality.
ASJC Scopus Sachgebiete
- Physik und Astronomie (insg.)
- Instrumentierung
- Umweltwissenschaften (insg.)
- Umweltwissenschaften (sonstige)
- Erdkunde und Planetologie (insg.)
- Erdkunde und Planetologie (sonstige)
Zitieren
- Standard
- Harvard
- Apa
- Vancouver
- BibTex
- RIS
in: ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Nr. X-2-2024, 10.06.2024, S. 153-161.
Publikation: Beitrag in Fachzeitschrift › Konferenzaufsatz in Fachzeitschrift › Forschung › Peer-Review
}
TY - JOUR
T1 - Depth-Aware Panoptic Segmentation
AU - Nguyen, Tuan
AU - Mehltretter, Max
AU - Rottensteiner, Franz
N1 - Publisher Copyright: © Author(s) 2024.
PY - 2024/6/10
Y1 - 2024/6/10
N2 - Panoptic segmentation unifies semantic and instance segmentation and thus delivers a semantic class label and, for so-called thing classes, also an instance label per pixel. The differentiation of distinct objects of the same class with a similar appearance is particularly challenging and frequently causes such objects to be incorrectly assigned to a single instance. In the present work, we demonstrate that information on the 3D geometry of the observed scene can be used to mitigate this issue: We present a novel CNN-based method for panoptic segmentation which processes RGB images and depth maps given as input in separate network branches and fuses the resulting feature maps in a late fusion manner. Moreover, we propose a new depth-aware dice loss term which penalises the assignment of pixels to the same thing instance based on the difference between their associated distances to the camera. Experiments carried out on the Cityscapes dataset show that the proposed method reduces the number of objects that are erroneously merged into one thing instance and outperforms the method used as basis by +2.2% in terms of panoptic quality.
AB - Panoptic segmentation unifies semantic and instance segmentation and thus delivers a semantic class label and, for so-called thing classes, also an instance label per pixel. The differentiation of distinct objects of the same class with a similar appearance is particularly challenging and frequently causes such objects to be incorrectly assigned to a single instance. In the present work, we demonstrate that information on the 3D geometry of the observed scene can be used to mitigate this issue: We present a novel CNN-based method for panoptic segmentation which processes RGB images and depth maps given as input in separate network branches and fuses the resulting feature maps in a late fusion manner. Moreover, we propose a new depth-aware dice loss term which penalises the assignment of pixels to the same thing instance based on the difference between their associated distances to the camera. Experiments carried out on the Cityscapes dataset show that the proposed method reduces the number of objects that are erroneously merged into one thing instance and outperforms the method used as basis by +2.2% in terms of panoptic quality.
KW - Dice Loss
KW - Panoptic Segmentation
KW - RGB Depth Fusion
UR - http://www.scopus.com/inward/record.url?scp=85199892117&partnerID=8YFLogxK
U2 - 10.48550/arXiv.2405.10947
DO - 10.48550/arXiv.2405.10947
M3 - Conference article
AN - SCOPUS:85199892117
SP - 153
EP - 161
JO - ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences
JF - ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences
SN - 2194-9042
IS - X-2-2024
T2 - 2024 ISPRS TC II Mid-term Symposium on The Role of Photogrammetry for a Sustainable World
Y2 - 11 June 2024 through 14 June 2024
ER -