Details
Original language | English |
---|---|
Title of host publication | 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) |
Publisher | Institute of Electrical and Electronics Engineers Inc. |
Pages | 1226-1235 |
Number of pages | 10 |
ISBN (electronic) | 9798350307443 |
ISBN (print) | 9798350307450 |
Publication status | Published - 2023 |
Event | 2023 IEEE/CVF International Conference on Computer Vision Workshops, ICCVW 2023 - Paris, France Duration: 2 Oct 2023 → 6 Oct 2023 |
Abstract
Sparse neural networks are a key factor in developing resource-efficient machine learning applications. We propose the novel and powerful sparse learning method Adaptive Regularized Training (ART) to compress dense into sparse networks. Instead of the commonly used binary mask during training to reduce the number of model weights, we inherently shrink weights close to zero in an iterative manner with increasing weight regularization. Our method compresses the pre-trained model "knowledge"into the weights of highest magnitude. Therefore, we introduce a novel regularization loss named HyperSparse that exploits the highest weights while conserving the ability of weight exploration. Extensive experiments on CIFAR and TinyImageNet show that our method leads to notable performance gains compared to other sparsification methods, especially in extremely high sparsity regimes up to 99.8% model sparsity. Additional investigations provide new insights into the patterns that are encoded in weights with high magnitudes.1
Keywords
- Neural Networks, Pruning, Sparsity, Unstructured Pruning
ASJC Scopus subject areas
- Computer Science(all)
- Artificial Intelligence
- Computer Science(all)
- Computer Science Applications
- Computer Science(all)
- Computer Vision and Pattern Recognition
Cite this
- Standard
- Harvard
- Apa
- Vancouver
- BibTeX
- RIS
2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). Institute of Electrical and Electronics Engineers Inc., 2023. p. 1226-1235.
Research output: Chapter in book/report/conference proceeding › Conference contribution › Research › peer review
}
TY - GEN
T1 - HyperSparse Neural Networks
T2 - 2023 IEEE/CVF International Conference on Computer Vision Workshops, ICCVW 2023
AU - Glandorf, Patrick
AU - Kaiser, Timo
AU - Rosenhahn, Bodo
N1 - Funding Information: This work was supported by the Federal Ministry of Education and Research (BMBF), Germany under the project AI service center KISSKI (grant no. 01IS22093C), the Deutsche Forschungsgemeinschaft (DFG) under Germany's Excellence Strategy within the Cluster of Excellence PhoenixD (EXC 2122), and by the Federal Ministry of the Environment, Nature Conservation, Nuclear Safety and Consumer Protection, Germany under the project GreenAutoML4FAS (grant no. 67KI32007A).
PY - 2023
Y1 - 2023
N2 - Sparse neural networks are a key factor in developing resource-efficient machine learning applications. We propose the novel and powerful sparse learning method Adaptive Regularized Training (ART) to compress dense into sparse networks. Instead of the commonly used binary mask during training to reduce the number of model weights, we inherently shrink weights close to zero in an iterative manner with increasing weight regularization. Our method compresses the pre-trained model "knowledge"into the weights of highest magnitude. Therefore, we introduce a novel regularization loss named HyperSparse that exploits the highest weights while conserving the ability of weight exploration. Extensive experiments on CIFAR and TinyImageNet show that our method leads to notable performance gains compared to other sparsification methods, especially in extremely high sparsity regimes up to 99.8% model sparsity. Additional investigations provide new insights into the patterns that are encoded in weights with high magnitudes.1
AB - Sparse neural networks are a key factor in developing resource-efficient machine learning applications. We propose the novel and powerful sparse learning method Adaptive Regularized Training (ART) to compress dense into sparse networks. Instead of the commonly used binary mask during training to reduce the number of model weights, we inherently shrink weights close to zero in an iterative manner with increasing weight regularization. Our method compresses the pre-trained model "knowledge"into the weights of highest magnitude. Therefore, we introduce a novel regularization loss named HyperSparse that exploits the highest weights while conserving the ability of weight exploration. Extensive experiments on CIFAR and TinyImageNet show that our method leads to notable performance gains compared to other sparsification methods, especially in extremely high sparsity regimes up to 99.8% model sparsity. Additional investigations provide new insights into the patterns that are encoded in weights with high magnitudes.1
KW - Neural Networks
KW - Pruning
KW - Sparsity
KW - Unstructured Pruning
UR - http://www.scopus.com/inward/record.url?scp=85180564637&partnerID=8YFLogxK
U2 - 10.48550/arXiv.2308.07163
DO - 10.48550/arXiv.2308.07163
M3 - Conference contribution
AN - SCOPUS:85180564637
SN - 9798350307450
SP - 1226
EP - 1235
BT - 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 2 October 2023 through 6 October 2023
ER -