Multimodal Fusion of Hyperspectral and LiDAR Data Using Attention Mechanisms for Land Cover Mapping
DOI:
https://doi.org/10.29284/y374x371Keywords:
.Abstract
The proper classification of land-cover in the remote sensing needs to be associated with the effective fusion of complementary spectral and structural data. In this paper, the idea of Attention-Guided Fusion Network (GAFNet) is presented to integrate hyperspectral image (HSI) with LiDAR data by reweighting modality-specific features based on a spatial attention mechanism to accentuate the informative part of the image and reduce the background noise. The framework was tested on two benchmark datasets namely Trento and Houston which depict semi-urban and complex urban settings, respectively. Experimental evidence shows that GAFNet is always superior to state-of-the-art fusion methods, being able to obtain the Overall Accuracy of 96.75% and Kappa of 0.9564 on Trento, and the Average accuracy of 89.50%,Overall Accuracy of 91.08% and Kappa of 0.8861 on Houston. Quantitative measures and qualitative analyses have shown that the underlying nature of attention-guided multimodal feature fusion is robust, has the ability to generalize, and still performs well in remote sensing classification.
References
[1] G. Camps-Valls, D. Tuia, L. Bruzzone, and J. A. Benediktsson, “Advances in hyperspectral image classification: Earth monitoring with statistical learning methods,” IEEE Signal Processing Magazine, vol. 31, no. 1, pp. 45–54, 2014.
[2] P. Ghamisi, J. A. Benediktsson, and S. Phinn, “Land-cover classification using both hyperspectral and LiDAR data,” IEEE Geoscience and Remote Sensing Letters, vol. 12, no. 7, pp. 1521–1525, 2015.
[3] P. Ghamisi, N. Yokoya, J. Li, et al.,“Advances in hyperspectral image and signal processing: A comprehensive overview,” IEEE Geoscience and Remote Sensing Magazine, vol. 9, no. 2, pp. 37–78, 2021.
[4] S. Roy, B. Rasti, P. Ghamisi, “Deep multimodal fusion for land cover mapping: A review,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 188, pp. 21–42, 2022.
[5] J. A. Benediktsson, P. H. Swain, and O. K. Ersoy, “Neural network approaches versus statistical methods in classification of multisource remote sensing data,”
IEEE Transactions on Geoscience and Remote Sensing, vol. 28, no. 4, pp. 540–552, 1990.
[6] J. Li, J. M. Bioucas-Dias, and A. Plaza, “Hyperspectral image segmentation using a new Bayesian approach with active learning,” IEEE Transactions on Geoscience and Remote Sensing, vol. 48, no. 11, pp. 4089–4101, 2010.
[7] Y. Xu, Z. Wu, J. Chanussot, and Z. Wei, “Nonlocal patch-based weighted sparse representation for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 56, no. 2, pp. 741–754, 2018.
[8] A. L. Fassnacht, H. Latifi, K. Stereńczak, A. Modzelewska, M. Lefsky, and P. Waser, “Review of studies on tree species classification from remotely sensed data,” Remote Sensing of Environment, vol. 186, pp. 64–87, 2016.
[9] Y. Chen, H. Jiang, C. Li, X. Jia, and P. Ghamisi, “Deep feature extraction and classification of hyperspectral images based on convolutional neural networks,” IEEE Transactions on Geoscience and Remote Sensing, vol. 54, no. 10, pp. 6232–6251, 2016.
[10] B. Rasti, P. Ghamisi, J. Plaza, and A. Plaza, “Fusion of hyperspectral and LiDAR data using sparse and low-rank component analysis,” IEEE Transactions on Geoscience and Remote Sensing, vol. 55, no. 11, pp. 6354–6365, 2017.
[11] D. Tuia, F. Pacifici, M. Kanevski, and W. J. Emery, “Classification of very high spatial resolution imagery using mathematical morphology and support vector machines,”
IEEE Transactions on Geoscience and Remote Sensing, vol. 47, no. 11, pp. 3866–3879, 2009.
[12] L. Mou, P. Ghamisi, and X. X. Zhu, “Deep recurrent neural networks for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 55, no. 7, pp. 3639–3655, 2017.
[13] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7132–7141, 2018.
[14] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “CBAM: Convolutional block attention module,” European Conference on Computer Vision (ECCV), pp. 3–19, 2018.
[15] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” CVPR, pp. 7794–7803, 2018.
[16] P. Ghamisi, N. Yokoya, J. Li, et al.,“Advances in hyperspectral image and signal processing: A comprehensive overview,” IEEE Geoscience and Remote Sensing Magazine, vol. 9, no. 2, pp. 37–78, 2021.
[17] L. Mou, Y. Zhu, X. X. Zhu,“Vision transformers for remote sensing image classification,”
IEEE Transactions on Geoscience and Remote Sensing, vol. 60, 2022.
[18] Y. Li, H. Zhang, Q. Shen,“Multimodal remote sensing data fusion using attention-based deep networks,” Remote Sensing, vol. 13, no. 18, 2021.
[19] Z. Hong, J. Chanussot, X. Zhu, “Learning a unified attention model for hyperspectral–LiDAR data fusion,” IEEE Transactions on Neural Networks and Learning Systems, 2022.
[20] M. He, B. Li, H. Chen, “Attention-guided feature fusion for hyperspectral and LiDAR classification,” Pattern Recognition, vol. 136, 2023.
[21] Y. Zhang, L. Wang, J. Li, “Cross-modal attention networks for hyperspectral–LiDAR urban classification,” IEEE Transactions on Geoscience and Remote Sensing, 2024.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
