3D change detection; 3D change detection metrics; data augmentation; remote sensing; zero-shot learning; Data augmentation; Deep learning; Remote-sensing
Abstract :
[en] Detecting and quantifying changes is crucial for monitoring transformations of the Earth’s surface. It is thus essential to employ methods that can effectively retrieve both 2D and 3D changes over time. The MultiTask Bitemporal Images Transformer (MTBIT) was recently introduced to tackle 2D and 3D Change Detection (CD) tasks using bi-temporal optical images. Despite strong performances on existing benchmarks, MTBIT shows some limitations, i.e. a pronounced tendency to overfit the training distribution and difficulty in inferring extreme values, which motivates the need for improvements. We hence propose a new set of custom augmentations, applied individually or in specific combinations, to discern intricate geometries small structures and subtle terrain changes. Furthermore, to address conventional evaluation metrics' limitations we introduce the true positive RMSE (tpRMSE) metric which provides a more comprehensive understanding of MTBIT efficacy. The most successful augmentation combination reduces cRMSE to 5.88 m and tpRMSE to 5.34 m, from 6.33 m and 5.60 m of the baseline, respectively. Finally, a first zero-shot learning experiment is carried out on a new small dataset, achieving promising improvements towards domain generalization. In summary, the proposed contributions enhance the practical utility and reliability of MTBIT in real-world applications, addressing critical challenges in the field of Remote Sensing CD.
Disciplines :
Engineering, computing & technology: Multidisciplinary, general & others
Author, co-author :
Contu, Riccardo; Department of Civil, Edil and Environmental Engineering, Sapienza University of Rome, Rome, Italy
Marsocci, Valerio; Geomatics Research Group, KU Leuven, Gent, Belgium
Coletta, Virginia; Department of Civil, Edil and Environmental Engineering, Sapienza University of Rome, Rome, Italy
Ravanelli, Roberta ; Université de Liège - ULiège > Sphères ; Department of Civil, Edil and Environmental Engineering, Sapienza University of Rome, Rome, Italy
Scardapane, Simone; Department of Information, Electronics, and Telecommunications Engineering, Sapienza University of Rome, Rome, Italy
Language :
English
Title :
Towards zero-shot learning in 3D change detection: improving generalization with custom augmentations and evaluation
Asadi Shamsabadi, E., Xu, C., Rao, A. S., Nguyen, T., Ngo, T., & da Costa, D. D., (2022). Vision transformer-based autonomous crack detection on asphalt and concrete surfaces. Automation in Construction, 140, 104316. https://doi.org/10.1016/j.autcon.2022.104316
Audebert, N., Le Saux, B., & Lefèvre, S., (2017). Joint learning from earth observation and OpenStreetMap data to get faster better semantic maps. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (pp. 1552–19). https://doi.org/10.1109/CVPRW.2017.199
Bai, T., Wang, L., Yin, D., Sun, K., Chen, Y., Li, W., & Li, D., (2022). Deep learning for change detection in remote sensing: A review. Geo-Spatial Information Science, 8(9), 1–27. https://doi.org/10.1080/10095020.2022.2085633
Baldi, P., Fabris, M., Marsella, M., & Monticelli, R., (2005). Monitoring the morphological evolution of the sciara del fuoco during the 2002–2003 stromboli eruption using multi-temporal photogrammetry. Isprs Journal of Photogrammetry & Remote Sensing, 59(4), 199–211. https://doi.org/10.1016/j.isprsjprs.2005.02.004
Bandara, W. G. C., Nair, N. G., & Patel, V. M., (2022). Ddpm-cd: Remote sensing change detection using denoising diffusion probabilistic models. arXiv preprint arXiv: 2206.11892.v2. https://doi.org/10.48550/arXiv.2206.11892
Bandara, W. G. C., & Patel, V. M., (2022). A transformer-based siamese network for change detection. IGARSS, 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium (pp. 207–210). https://doi.org/10.1109/IGARSS46834.2022.9883686
Bendig, J., Bolten, A., & Bareth, G., (2013, 12). Uav-based imaging for multi-temporal, very high resolution crop surface models to monitor crop growth variability. Photogrammetrie - Fernerkundung - Geoinformation, 2013(6), 551–562. https://doi.org/10.1127/1432-8364/2013/0200
Bing Liu, B., Yu, A., Zuo, X., Wang, R., Qiu, C., & Yu, X., (2023). Deep hierarchical transformer for change detection in high-resolution remote sensing images. European Journal of Remote Sensing, 56(1), 2196641. https://doi.org/10.1080/22797254.2023.2196641
Buslaev, A., Iglovikov, V. I., Khvedchenya, E., Parinov, A., Druzhinin, M., & Kalinin, A. A., (2020). Albumentations: Fast and flexible image augmentations. Information, 11(2), 125. https://doi.org/10.3390/info11020125
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., & Joulin, A., (2021). Emerging properties in self-supervised vision transformers. arXiv preprint arXiv: 2104.14294. https://doi.org/10.48550/arXiv.2104.14294
Chen, C., Ma, H., Yao, G., Lv, N., Yang, H., Li, C., & Wan, S., (2021). Remote sensing image augmentation based on text description for waterside change detection. Remote Sensing, 13(10), 1894. https://doi.org/10.3390/rs13101894
Chen, H., Song, J., Wu, C., Du, B., & Yokoya, N., (2023). Exchange means change: An unsupervised single-temporal change detection framework based on intra- and inter-image patch exchange. Isprs Journal of Photogrammetry & Remote Sensing, 206, 87–105. https://doi.org/10.1016/j.isprsjprs.2023.11.004
Chen, H., Yokoya, N., & Chini, M., (2023). Fourier domain structural relationship analysis for unsupervised multimodal change detection. Isprs Journal of Photogrammetry & Remote Sensing, 198, 99–114. https://doi.org/10.1016/j.isprsjprs.2023.03.004
Chen, H., Yokoya, N., Wu, C., & Du, B., (2022). Unsupervised multimodal change detection based on structural relationship graph representation learning. IEEE Transactions on Geoscience & Remote Sensing, 60, 1–18. https://doi.org/10.1109/TGRS.2022.3229027
Chen, K., Liu, C., Li, W., Liu, Z., Chen, H., Zhang, H., & Shi, Z., (2023). Time travelling pixels: Bitemporal features integration with foundation model for remote sensing image change detection. https://doi.org/10.48550/arXiv.2312.16202
Chen, K., Zou, Z., & Shi, Z., (2021). Building extraction from remote sensing images with sparse token transformers. Remote Sensing, 13(21), 4441. https://doi.org/10.3390/rs13214441
Coletta, V., Marsocci, V., & Ravanelli, R., (2022). 3DCD: A new dataset for 2D and 3D change detection using deep learning techniques. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLIII- B3-2022, 1349–1354. https://doi.org/10.5194/isprs-archives-XLIII-B3-2022-1349-2022
Daudt, R. C., Le Saux, B., & Boulch, A., (2018). Fully convolutional siamese networks for change detection. 2018 25th IEEE International Conference on Image Processing (ICIP) (pp. 4063–4067). https://doi.org/10.1109/ICIP.2018.8451652
Deuser, F., Habel, K., & Oswald, N., (2023). Sample4geo: Hard negative sampling for cross-view geo-localisation. arXiv preprint arXiv: 2303.11851. https://doi.org/10.48550/arXiv.2303.11851
Ding, L., Zhu, K., Peng, D., Tang, H., Yang, K., & Bruzzone, L., (2024). Adapting segment anything model for change detection in vhr remote sensing images. IEEE Transactions on Geoscience & Remote Sensing, 62, 1–11. https://doi.org/10.1109/tgrs.2024.3368168
Dong, S., Wang, L., Du, B., & Meng, X., (2024). Changeclip: Remote sensing change detection with multimodal vision-language representation learning. Isprs Journal of Photogrammetry & Remote Sensing, 208, 53–69. https://doi.org/10.1016/j.isprsjprs.2024.01.004
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., & Houlsby, N., (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv. https://doi.org/10.48550/ARXIV.2010.11929
Emmanuel Fundisi, E., Tesfamichael, S. G., & Ahmed, F., (2022). A combination of sentinel-1 RADAR and Sentinel-2 multispectral data improves classification of morphologically similar savanna woody plants. European Journal of Remote Sensing, 55(1), 372–387. https://doi.org/10.1080/22797254.2022.2083984
Fuentes Reyes, M., Xie, Y., Yuan, X., d’Angelo, P., Kurz, F., Cerra, D., & Tian, J., (2023). A 2D/3D multimodal data simulation approach with applications on urban semantic segmentation, building extraction and change detection. Isprs Journal of Photogrammetry & Remote Sensing, 205, 74–97. https://doi.org/10.1016/j.isprsjprs.2023.09.013
Ghasem Abdi, G., Samadzadegan, F., & Reinartz, P., (2017). A decision-based multi-sensor classification system using thermal hyperspectral and visible data in urban area. European Journal of Remote Sensing, 50(1), 414–427. https://doi.org/10.1080/22797254.2017.1348914
Ghuffar, S., Székely, B., Roncat, A., & Pfeifer, N., (2013). Landslide displacement monitoring using 3D range flow on airborne and terrestrial LiDAR data. Remote Sensing, 5(6), 2720–2745. https://doi.org/10.3390/rs5062720
Hafner, S., Ban, Y., & Nascetti, A., (2022). Unsupervised domain adaptation for global urban extraction using sentinel-1 SAR and Sentinel-2 MSI data. Remote Sensing of Environment, 280, 113192. https://doi.org/10.1016/j.rse.2022.113192
Hafner, S., Nascetti, A., Azizpour, H., & Ban, Y., (2022). Sentinel-1 and sentinel-2 data fusion for urban change detection using a dual stream U-Net. IEEE Geoscience & Remote Sensing Letters, 19, 1–5. https://doi.org/10.1109/LGRS.2021.3119856
Hao, X., Liu, L., Yang, R., Yin, L., Zhang, L., & Li, X., (2023). A review of data augmentation methods of remote sensing image target recognition. Remote Sensing, 15(3), 827. https://doi.org/10.3390/rs15030827
Haut, J. M., Paoletti, M. E., Plaza, J., Plaza, A., & Plaza, L., (2019). Hyperspectral image classification using random occlusion data augmentation. IEEE Geoscience & Remote Sensing Letters, 16(11), 1751–1755. https://doi.org/10.1109/LGRS.2019.2909495
Hayet, S. S., Sally, E. G., Abdelmounaam, R., Samy, A.-A., & Nour, E. I. B., (2020). What is a remote sensing change detection technique? Towards a conceptual framework. International Journal of Remote Sensing, 41(5), 1788–1812. https://doi.org/10.1080/01431161.2019.1674463
Hong, D., Zhang, B., Li, H., Li, Y., Yao, J., Li, C., Werner, M., Chanussot, J., Zipf, A., & Zhu, X. X., (2023). Cross-city matters: A multimodal remote sensing benchmark dataset for cross-city semantic segmentation using high-resolution domain adaptation networks. Remote Sensing of Environment, 299, 113856. https://doi.org/10.1016/j.rse.2023.113856
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., & Girshick, R., (2023). Segment anything. https://doi.org/10.48550/arXiv.2304.02643
Lalitha, V., & Latha, B., (2022). A review on remote sensing imagery augmentation using deep learning. Materials Today: Proceedings (Vol. 62. pp. 4772–4778). (International Conference on Innovative Technology for Sustainable Development). https://doi.org/10.1016/j.matpr.2022.03.341
Leenstra, M., Marcos, D., Bovolo, F., & Tuia, D., (2021). Self-supervised pre-training enhances change detection in sentinel-2 imagery. International conference on pattern recognition (pp. 578–590). https://doi.org/10.1007/978-3-030-68787-8˙426
Liu, M., Chai, Z., Deng, H., & Liu, R., (2022). A cnn-transformer network with multiscale context aggregation for fine-grained cropland change detection. IEEE Journal of Selected Topics in Applied Earth Observations & Remote Sensing, 15, 4297–4306. https://doi.org/10.1109/JSTARS.2022.3177235
Loshchilov, I., & Hutter, F., (2019). 7th International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=Bkg6RiCqY7.
Lu, D., Mausel, P., Brondízio, E., & Moran, E., (2004). Change detection techniques. International Journal of Remote Sensing, 25(12), 2365–2401. https://doi.org/10.1080/0143116031000139863
Lv, N., Ma, H., Chen, C., Pei, Q., Zhou, Y., Xiao, F., & Li, J., (2021). Remote sensing data augmentation through adversarial training. IEEE Journal of Selected Topics in Applied Earth Observations & Remote Sensing, 14, 9318–9333. https://doi.org/10.1109/JSTARS.2021.3110842
Malpica, J. A., Alonso, M. C., Papí, F., Arozarena, A., & Agirre, A. M. D., (2013). Change detection of buildings from satellite imagery and lidar data. International Journal of Remote Sensing, 34(5), 1652–1675. https://doi.org/10.1080/01431161.2012.725483
Marsocci, V., Coletta, V., Ravanelli, R., Scardapane, S., & Crespi, M., (2023). Inferring 3D change detection from bitemporal optical images. Isprs Journal of Photogrammetry & Remote Sensing, 196(23), 325–339. https://doi.org/10.1016/j.isprsjprs.2022.12.009
Marsocci, V., Gonthier, N., Garioud, A., Scardapane, S., & Mallet, C., (2023). Geomultitasknet: Remote sensing unsupervised domain adaptation using geo-graphical coordinates. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (Vol. 5. pp. 2075–2085). https://doi.org/10.1109/CVPRW59228.2023.00201
Menderes, A., Erener, A., & Sarp, G., (2015). Automatic detection of damaged buildings after earthquake hazard by using remote sensing and information technologies. Procedia Earth and Planetary Science, 15(6), 257–262. (World Multidisciplinary Earth Sciences Symposium, WMESS 2015). https://doi.org/10.1016/j.proeps.2015.08.063
Moieez, H., Marsocci, V., & Scardapane, S., (2023). Continual self-supervised learning in earth observation with embedding regularization. Igarss 2023 - 2023 ieee international geoscience and remote sensing symposium (Vol. 5. pp. 5029–5032). https://doi.org/10.1109/IGARSS52108.2023.10283121
Oubara, A., Wu, F., Amamra, A., & Yang, G., (2022). Survey on remote sensing data augmentation: Advances, challenges, and future perspectives. In M.R., Senouci, S.Y., Boulahia, & M.A., Benatia (Eds.), Advances in computing systems and applications (Vol. 6, pp. 95–104). Springer International Publishing. https://doi.org/10.1007/978-3-031-12097-8˙9
Qin, R., Tian, J., & Reinartz, P., (2016). 3D change detection–approaches and applications. Isprs Journal of Photogrammetry & Remote Sensing, 122(3), 41–56. https://doi.org/10.1016/j.isprsjprs.2016.09.013
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., & Sutskever, I., (2021, July18–24). Learning transferable visual models from natural language supervision. In M., Meila & T., Zhang (Eds.), Proceedings of the 38th international conference on machine learning (Vol. 139. pp. 8748–8763). PMLR. https://doi.org/10.48550/arXiv.2103.00020
Rottensteiner, F., Sohn, G., Jung, J., Gerke, M., Baillard, C., Benitez, S., & Breitkopf, U., (2012). The isprs benchmark on urban object classification and 3d building reconstruction. ISPRS Annals of the Photogrammetry, Remote Sensing & Spatial Information Sciences, I-3(5), 293–298. https://doi.org/10.5194/isprsannals-I-3-293-2012
Shafique, A., Cao, G., Khan, Z., Asad, M., & Aslam, M., (2022). Deep learning-based change detection in remote sensing images: A review. Remote Sensing, 14(4), 871. https://doi.org/10.3390/rs14040871
Shawky, O. A., Hagag, A., El-Dahshan, E.-S. A., & Ismail, M. A., (2020). Remote sensing image scene classification using cnn-mlp with data augmentation. Optik, 221(6), 165356. https://doi.org/10.1016/j.ijleo.2020.165356
Shi, Q., Liu, M., Li, S., Liu, X., Wang, F., & Zhang, L., (2022). A deeply supervised attention metric-based network and an open aerial image dataset for remote sensing change detection. IEEE Transactions on Geoscience & Remote Sensing, 60(6), 1–16. https://doi.org/10.1109/TGRS.2021.3085870
Shirowzhan, S., Sepasgozar, S. M., Li, H., Trinder, J., & Tang, P., (2019). Comparative analysis of machine learning and point-based algorithms for detecting 3D changes in buildings over time using bi-temporal lidar data. Automation in Construction, 105(6), 102841. https://doi.org/10.1016/j.autcon.2019.102841
Shorten, C., & Khoshgoftaar, T. M., (2019). A survey on image data augmentation for deep learning. Journal of Big Data, 6(1), 60. https://doi.org/10.1186/s40537-019-0197-0
Shuwen Xu, S., Liao, Y., Yan, X., & Zhang, G., (2020). Change detection in sar images based on iterative otsu. European Journal of Remote Sensing, 53(1), 331–339. https://doi.org/10.1080/22797254.2020.1852606
Song, J., Chen, H., & Yokoya, N., (2023). Syntheworld: A large-scale synthetic dataset for land cover mapping and building change detection. https://doi.org/10.48550/arXiv.2309.01907
Stilla, U., & Xu, Y., (2023). Change detection of urban objects using 3D point clouds: A review. Isprs Journal of Photogrammetry & Remote Sensing, 197(6), 228–255. https://doi.org/10.1016/j.isprsjprs.2023.01.010
Sun, X., Wang, B., Wang, Z., Li, H., Li, H., & Fu, K., (2021). Research progress on few-shot learning for remote sensing image interpretation. IEEE Journal of Selected Topics in Applied Earth Observations & Remote Sensing, 14(7), 2387–2402. https://doi.org/10.1109/JSTARS.2021.3052869
Sun, Y., Lei, L., Li, X., Tan, X., Kuang, G., Li, X., Zhang, B., & Plaza, A., (2022). Multisource data reconstruction-based deep unsupervised hashing for unisource remote sensing image retrieval. IEEE Transactions on Geoscience & Remote Sensing, 60(5), 1–16. https://doi.org/10.1109/TGRS.2022.3231215
Taneja, A., Ballan, L., & Pollefeys, M., (2011). Image based detection of geometric changes in urban environments. 2011 International Conference on Computer Vision (Vol. 6. pp. 2336–2343). https://doi.org/10.1109/ICCV.2011.6126515
Wang, Z., Liu, D., Wang, Z., Liao, X., & Zhang, Q., (2023). A new remote sensing change detection data augmentation method based on mosaic simulation and haze image simulation. IEEE Journal of Selected Topics in Applied Earth Observations & Remote Sensing, 16(7), 4579–4590. https://doi.org/10.1109/JSTARS.2023.3269784
Xiao, Q., Liu, B., Li, Z., Ni, W., Yang, Z., & Li, L., (2021). Progressive data augmentation method for remote sensing ship image classification based on imaging simulation system and neural style transfer. IEEE Journal of Selected Topics in Applied Earth Observations & Remote Sensing, 14(6), 9176–9186. https://doi.org/10.1109/JSTARS.2021.3109600
Xingrui Yu, X., Wu, X., Luo, C., & Ren, P., (2017). Deep learning in remote sensing scene classification: A data augmentation enhanced convolutional neural network framework. GIScience & Remote Sensing, 54(5), 741–758. https://doi.org/10.1080/15481603.2017.1323377
Xu, B., Zeng, Z., Lian, C., & Ding, Z., (2022). Generative mixup networks for zero-shot learning. IEEE Transactions on Nesural Networks and Learning Systems, 7. https://doi.org/10.1109/TNNLS.2022.3142181
Xu, M., Yoon, S., Fuentes, A., & Park, D. S., (2023). A comprehensive survey of image augmentation techniques for deep learning. Pattern Recognition, 137(3), 109347. https://doi.org/10.1016/j.patcog.2023.109347
Yan, Y., Zhang, Y., & Su, N., (2019). A novel data augmentation method for detection of specific aircraft in remote sensing rgb images. Institute of Electrical and Electronics Engineers Access, 7(6), 56051–56061. https://doi.org/10.1109/ACCESS.2019.2913191
Yang, L., Zhang, Z., Song, Y., Hong, S., Xu, R., Zhao, Y., & Yang, M.-H., (2024). Diffusion models: A comprehensive survey of methods and applications. https://doi.org/10.48550/arXiv.2209.00796
Zhang, L., & Zhang, L., (2022). Artificial intelligence for remote sensing data analysis: A review of challenges and opportunities. IEEE Geoscience and Remote Sensing Magazine, 10(2), 270–294. https://doi.org/10.1109/MGRS.2022.3145854
