Details
| Original language | English |
|---|---|
| Title of host publication | ECAI 2024 |
| Editors | Ulle Endriss, Francisco S. Melo, Kerstin Bach, Alberto Bugarin-Diz, Jose M. Alonso-Moral, Senen Barro, Fredrik Heintz |
| Pages | 2781-2789 |
| Number of pages | 9 |
| ISBN (electronic) | 9781643685489 |
| Publication status | Published - 19 Oct 2024 |
| Event | 27th European Conference on Artificial Intelligence, ECAI 2024 - Santiago de Compostela, Spain Duration: 19 Oct 2024 → 24 Oct 2024 |
Publication series
| Name | Frontiers in Artificial Intelligence and Applications |
|---|---|
| Volume | 392 |
| ISSN (Print) | 0922-6389 |
| ISSN (electronic) | 1879-8314 |
Abstract
Accurate spatio-temporal prediction is crucial for the sustainable development of smart cities. However, current approaches often struggle to capture important spatio-temporal relationships, particularly overlooking global relations among distant city regions. Most existing techniques predominantly rely on Convolutional Neural Networks (CNNs) to capture global relations. However, CNNs exhibit neighbourhood bias, making them insufficient for capturing distant relations. To address this limitation, we propose ST-SAMPLENET, a novel transformer-based architecture that combines CNNs with self-attention mechanisms to capture both local and global relations effectively. Moreover, as the number of regions increases, the quadratic complexity of self-attention becomes a challenge. To tackle this issue, we introduce a lightweight region sampling strategy that prunes non-essential regions and enhances the efficiency of our approach. Furthermore, we introduce a spatially constrained position embedding that incorporates spatial neighbourhood information into the self-attention mechanism, aiding in semantic interpretation and improving the performance of ST-SAMPLENET. Our experimental evaluation on three real-world datasets demonstrates the effectiveness of ST-SAMPLENET. Additionally, our efficient variant achieves a 40% reduction in computational costs with only a marginal compromise in performance, approximately 1%.
Cite this
- Standard
- Harvard
- Apa
- Vancouver
- BibTeX
- RIS
ECAI 2024. ed. / Ulle Endriss; Francisco S. Melo; Kerstin Bach; Alberto Bugarin-Diz; Jose M. Alonso-Moral; Senen Barro; Fredrik Heintz. 2024. p. 2781-2789 (Frontiers in Artificial Intelligence and Applications; Vol. 392).
Research output: Chapter in book/report/conference proceeding › Conference contribution › Research › peer review
}
TY - GEN
T1 - Spatially Constrained Transformer with Efficient Global Relation Modelling for Spatio-Temporal Prediction
AU - Sao, Ashutosh
AU - Gottschalk, Simon
N1 - Publisher Copyright: © 2024 The Authors.
PY - 2024/10/19
Y1 - 2024/10/19
N2 - Accurate spatio-temporal prediction is crucial for the sustainable development of smart cities. However, current approaches often struggle to capture important spatio-temporal relationships, particularly overlooking global relations among distant city regions. Most existing techniques predominantly rely on Convolutional Neural Networks (CNNs) to capture global relations. However, CNNs exhibit neighbourhood bias, making them insufficient for capturing distant relations. To address this limitation, we propose ST-SAMPLENET, a novel transformer-based architecture that combines CNNs with self-attention mechanisms to capture both local and global relations effectively. Moreover, as the number of regions increases, the quadratic complexity of self-attention becomes a challenge. To tackle this issue, we introduce a lightweight region sampling strategy that prunes non-essential regions and enhances the efficiency of our approach. Furthermore, we introduce a spatially constrained position embedding that incorporates spatial neighbourhood information into the self-attention mechanism, aiding in semantic interpretation and improving the performance of ST-SAMPLENET. Our experimental evaluation on three real-world datasets demonstrates the effectiveness of ST-SAMPLENET. Additionally, our efficient variant achieves a 40% reduction in computational costs with only a marginal compromise in performance, approximately 1%.
AB - Accurate spatio-temporal prediction is crucial for the sustainable development of smart cities. However, current approaches often struggle to capture important spatio-temporal relationships, particularly overlooking global relations among distant city regions. Most existing techniques predominantly rely on Convolutional Neural Networks (CNNs) to capture global relations. However, CNNs exhibit neighbourhood bias, making them insufficient for capturing distant relations. To address this limitation, we propose ST-SAMPLENET, a novel transformer-based architecture that combines CNNs with self-attention mechanisms to capture both local and global relations effectively. Moreover, as the number of regions increases, the quadratic complexity of self-attention becomes a challenge. To tackle this issue, we introduce a lightweight region sampling strategy that prunes non-essential regions and enhances the efficiency of our approach. Furthermore, we introduce a spatially constrained position embedding that incorporates spatial neighbourhood information into the self-attention mechanism, aiding in semantic interpretation and improving the performance of ST-SAMPLENET. Our experimental evaluation on three real-world datasets demonstrates the effectiveness of ST-SAMPLENET. Additionally, our efficient variant achieves a 40% reduction in computational costs with only a marginal compromise in performance, approximately 1%.
UR - http://www.scopus.com/inward/record.url?scp=85216640333&partnerID=8YFLogxK
U2 - 10.48550/arXiv.2411.06836
DO - 10.48550/arXiv.2411.06836
M3 - Conference contribution
AN - SCOPUS:85216640333
T3 - Frontiers in Artificial Intelligence and Applications
SP - 2781
EP - 2789
BT - ECAI 2024
A2 - Endriss, Ulle
A2 - Melo, Francisco S.
A2 - Bach, Kerstin
A2 - Bugarin-Diz, Alberto
A2 - Alonso-Moral, Jose M.
A2 - Barro, Senen
A2 - Heintz, Fredrik
T2 - 27th European Conference on Artificial Intelligence, ECAI 2024
Y2 - 19 October 2024 through 24 October 2024
ER -