Multi-Label Degradation Classification Of Weather-Affected Images Using Hybrid CNN-Transformer Architecture For Advanced Driving Assistance Systems

Anmol Jain; Veerendra Yadav; Harsh Khatter

doi:10.29284/xzgvhp35

Authors

Anmol Jain Department of Computer Science and Engineering, Noida International University, Gautam Budh Nagar, Greater Noida, (U.P.), India.
Veerendra Yadav Department of Computer Science and Engineering, Noida International University, Gautam Budh Nagar, Greater Noida, (U.P.), India.
Harsh Khatter Department of Information Technology, Krishna Institute of Engineering & Technology, Ghaziabad, (U.P.), India.

DOI:

https://doi.org/10.29284/xzgvhp35

Keywords:

ADAS; multi-label classification; adverse weather; CNN–transformer; dehazing; deraining; calibration; attention; RESIDE; Snow100K; DENSE.

Abstract

In the proposed research, the objective of the task is to ensure robust ADAS performance with cameras in adverse weather conditions by being able to determine, for each image, which types of degradation, possibly simultaneously (fog, haze, rain, snow, or their combination), exist. This problem is formalized as multi-label classification with a proposed CNN-transformer network that combines local CNN information with global self-attention capabilities through an image-dependent gate function α. Each individual degradation is represented as an independent Bernoulli distribution with a composite loss function combining binary cross-entropy loss, focal reweighting loss, L2 regularization, and sparse attention, with per-class threshold calibration tailored for ADAS safety preferences. The fundamental contributions of this study are as follows: The weather understanding task for ADAS is modeled as calibrated multi-label recognition instead of single-label scene annotation or simple image restoration. We propose a hybrid ResNet+ViT model featuring an adaptive fusion gate α, which chooses between local and global contexts at the per-image level. We developed a training and calibration pipeline (composite loss function plus class-wise thresholding) to enhance the precision-recall performance, especially for the majority and combined degradations. The study offers a production-ready solution for the front-end, providing degraded-aware indications for routing images to the modules for dehazing, deraining, or desnowing. On the Results On RESIDE, Snow100K, and Dense, testing cross-datasets, we observe that our method outperforms the CNN-only and ViT-only baselines for macro F1 and mAP scores while achieving lower Hamming Loss values and comparable Subset Accuracy/LRAP metric values. The ablation study confirms the effectiveness of α fusion and regularizers, while the FP16/INT8 models stay within the ADAS latency/energy constraints while suffering little accuracy loss. We plan to make our splits and calibrate the thresholds publicly available.

References

[1] Wu, Haiyan, Yanyun Qu, Shaohui Lin, Jian Zhou, Ruizhi Qiao, Zhizhong Zhang, Yuan Xie, and Lizhuang Ma. "Contrastive learning for compact single image dehazing." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10551-10560. 2021.

[2] Liang, Jingyun, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. "Swinir: Image restoration using swin transformer." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1833-1844. 2021.

[3] Ancuti, Codruta O., Cosmin Ancuti, Florin-Alexandru Vasluianu, and Radu Timofte. "NTIRE 2021 nonhomogeneous dehazing challenge report." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 627-646. 2021.

[4] Dhananjaya, Mahesh M., Varun Ravi Kumar, and Senthil Yogamani. "Weather and light level classification for autonomous driving: Dataset, baseline and active learning." In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 2816-2821. IEEE, 2021.

[5] Mușat, Valentina, Ivan Fursa, Paul Newman, Fabio Cuzzolin, and Andrew Bradley. "Multi-weather city: Adverse weather stacking for autonomous driving." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2906-2915. 2021.

[6] Zamir, Syed Waqas, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. "Restormer: Efficient transformer for high-resolution image restoration." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5728-5739. 2022.

[7] Guo, Chun-Le, Qixin Yan, Saeed Anwar, Runmin Cong, Wenqi Ren, and Chongyi Li. "Image dehazing transformer with transmission-aware 3d position embedding." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5812-5820. 2022.

[8] Song, Yuda, Zhuqing He, Hui Qian, and Xin Du. "Vision transformers for single image dehazing." IEEE Transactions on Image Processing 32 (2023): 1927-1941.

[9] Liang, Yuanchu, Saeed Anwar, and Yang Liu. "Drt: A lightweight single image deraining recursive transformer." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 589-598. 2022.

[10] Jiang, Kui, Zhongyuan Wang, Chen Chen, Zheng Wang, Laizhong Cui, and Chia-Wen Lin. "Magic ELF: Image deraining meets association learning and transformer." arXiv preprint arXiv:2207.10455 (2022).

[11] Xiao, Jie, Xueyang Fu, Aiping Liu, Feng Wu, and Zheng-Jun Zha. "Image de-raining transformer." IEEE transactions on pattern analysis and machine intelligence 45, no. 11 (2022): 12978-12995.

[12] Namba, Yuto, and Xian-Hua Han. "Multi-Scale Channel Transformer Network for Single Image Deraining." In Proceedings of the 4th ACM International Conference on Multimedia in Asia, pp. 1-7. 2022.

[13] Yang, Hao, Dongming Zhou, Jinde Cao, Qian Zhao, and Miao Li. "RainFormer: a pyramid transformer for single image deraining." Journal of Supercomputing 79, no. 6 (2023).

[14] Ba, Yunhao, Howard Zhang, Ethan Yang, Akira Suzuki, Arnold Pfahnl, Chethan Chinder Chandrappa, Celso M. De Melo et al. "Not just streaks: Towards ground truth for single image deraining." In European Conference on Computer Vision, pp. 723-740. Cham: Springer Nature Switzerland, 2022.

[15] Valanarasu, Jeya Maria Jose, Rajeev Yasarla, and Vishal M. Patel. "Transweather: Transformer-based restoration of images degraded by adverse weather conditions." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2353-2363. 2022.

[16] Chen, Xiang, Hao Li, Mingqiang Li, and Jinshan Pan. "Learning a sparse transformer network for effective image deraining." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5896-5905. 2023.

[17] Chen, Sixiang, Tian Ye, Jinbin Bai, Erkang Chen, Jun Shi, and Lei Zhu. "Sparse sampling transformer with uncertainty-driven ranking for unified removal of raindrops and rain streaks." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 13106-13117. 2023.

[18] Qiu, Yuwei, Kaihao Zhang, Chenxi Wang, Wenhan Luo, Hongdong Li, and Zhi Jin. "Mb-taylorformer: Multi-branch efficient transformer expanded by taylor formula for image dehazing." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 12802-12813. 2023.

[19] Li, Yufeng, Jiyang Lu, Hongming Chen, Xianhao Wu, and Xiang Chen. "Dilated convolutional transformer for high-quality image deraining." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4199-4207. 2023.

Yang, Dawei, Xin He, and Ruiheng Zhang. "Alternating attention transformer for single image deraining." Digital Signal Processing 141 (2023): 104144.

[20] Guo, Yun, Xueyao Xiao, Yi Chang, Shumin Deng, and Luxin Yan. "From sky to the ground: A large-scale benchmark and simple baseline towards real rain removal." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 12097-12107. 2023.

[21] J. Li and X. Luo, “A Study of Weather-Image Classification Combining ViT and a Dual Enhanced-Attention Module,” Electronics, vol. 12, no. 5, p. 1213, 2023.

[22] Wang, Tao, Kaihao Zhang, Ziqian Shao, Wenhan Luo, Bjorn Stenger, Tong Lu, Tae-Kyun Kim, Wei Liu, and Hongdong Li. "Gridformer: Residual dense transformer with grid structure for image restoration in adverse weather conditions." International journal of computer vision 132, no. 10 (2024): 4541-4563.

[23] Wang, Tao, Kaihao Zhang, Ziqian Shao, Wenhan Luo, Bjorn Stenger, Tong Lu, Tae-Kyun Kim, Wei Liu, and Hongdong Li. "Gridformer: Residual dense transformer with grid structure for image restoration in adverse weather conditions." International journal of computer vision 132, no. 10 (2024): 4541-4563.

[24] Chen, Sixiang, Tian Ye, Yun Liu, Jinbin Bai, Haoyu Chen, Yunlong Lin, Jun Shi, and Erkang Chen. "Cplformer: Cross-scale prototype learning transformer for image snow removal." In Proceedings of the 31st ACM international conference on multimedia, pp. 4228-4239. 2023.

[25] Fan, Junkai, Jiangwei Weng, Kun Wang, Yijun Yang, Jianjun Qian, Jun Li, and Jian Yang. "Driving-video dehazing with non-aligned regularization for safety assistance." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26109-26119. 2024.

[26] Feng, Yuxin, Long Ma, Xiaozhe Meng, Fan Zhou, Risheng Liu, and Zhuo Su. "Advancing real-world image dehazing: Perspective, modules, and training." IEEE Transactions on Pattern Analysis and Machine Intelligence 46, no. 12 (2024): 9303-9320.

[27] Fang, Chengyu, Chunming He, Fengyang Xiao, Yulun Zhang, Longxiang Tang, Yuelin Zhang, Kai Li, and Xiu Li. "Real-world image dehazing with coherence-based label generator and cooperative unfolding network." arXiv e-prints (2024): arXiv-2406

[28] Ancuti, Codruta O., Cosmin Ancuti, Florin-Alexandru Vasluianu, Radu Timofte, Yidi Liu, Xingbo Wang, Yurui Zhu et al. "NTIRE 2024 dense and non-homogeneous dehazing challenge report." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6453-6468. 2024.

[29] Aloufi, Nasser, Abdulaziz Alnori, and Abdullah Basuhail. "Enhancing autonomous vehicle perception in adverse weather: A multi objectives model for integrated weather classification and object detection." Electronics 13, no. 15 (2024): 3063.

[30] Chen, Yanfei, Tong Yue, Pei An, Hanyu Hong, Tao Liu, Yangkai Liu, and Yihui Zhou. "ICAFormer: An Image Dehazing Transformer Based on Interactive Channel Attention." Sensors 25, no. 12 (2025): 3750.

[31] Wu, Xianhao, Hongming Chen, Xiang Chen, and Guili Xu. "Multi-scale transformer with conditioned prompt for image deraining." Digital Signal Processing 156 (2025): 104847.

[32] Hao, Yu, and Xiaoyan Liu. "A Semi-Supervised Single-Image Deraining Algorithm Based on the Integration of Wavelet Transform and Swin Transformer." Applied Sciences 15, no. 8 (2025): 4325.

[33] Chai, Guoqiang, Xiangshi Bo, Yulei Chen, Dawei Wang, Rui Yang, and Yuyou Nie. "Hybrid CNN-transformer attentive network for single image deraining." In International Symposium on Artificial Intelligence Innovations (IS-AII 2025), vol. 13681, pp. 89-98. SPIE, 2025.

[34] Li, Shuhong, Xiaotao Shao, and Yan Shen. "Physics-based model and transformer for single image dehazing." In Fifth International Conference on Image Processing and Intelligent Control (IPIC 2025), vol. 13782, pp. 60-65. SPIE, 2025.

[35] Zou, Shun, Yi Zou, Juncheng Li, Guangwei Gao, and Guojun Qi. "Cross Paradigm Representation and Alignment Transformer for Image Deraining." arXiv preprint arXiv:2504.16455 (2025).

[36] Vasilescu, Vlad, Ana Neacsu, and Daniela Faur. "Fine-Tuning Adversarially-Robust Transformers for Single-Image Dehazing." arXiv preprint arXiv:2504.17829 (2025).

[37] Nawaz, Mehmood, Shehryar Khan, Muhammad Daud, Muhammad Asim, Ghazanfar Ali Anwar, Ali Raza Shahid, Ho-Pui Ho, Tom Chan, Daniel Pak Kong, and Wu Yuan. "Improving Autonomous Vehicle Cognitive Robustness in Extreme Weather with Deep Learning and Thermal Camera Fusion." IEEE Open Journal of Vehicular Technology (2025).

[38] Huang, Liqin, Hanyu Zheng, Lin Pan, Zhipeng Su, and Qiang Wu. "Codebook prior-guided hybrid attention dehazing network." Image and Vision Computing (2025): 105700.

[39] Yang, Junjie, Haibo Wan, and Zhihai Shang. "Enhanced hybrid CNN and transformer network for remote sensing image change detection." Scientific Reports 15, no. 1 (2025): 10161.