data fusion; deep learning; enriched semantic segmentation; images; LiDAR point clouds; prior knowledge; prior-level fusion; urban environment; Deep learning; Enriched semantic segmentation; Image; Level fusion; LiDAR point cloud; Point-clouds; Prior-knowledge; Prior-level fusion; Semantic segmentation; Urban environments
Abstract :
[en] Three-dimensional semantic segmentation is the foundation for automatically creating enriched Digital Twin Cities (DTCs) and their updates. For this task, prior-level fusion approaches show more promising results than other fusion levels. This article proposes a new approach by developing and benchmarking three prior-level fusion scenarios to enhance the outcomes of point cloud-enriched semantic segmentation. The latter were compared with a baseline approach that used the point cloud only. In each scenario, specific prior knowledge (geometric features, classified images, or classified geometric information) and aerial images were fused into the neural network’s learning pipeline with the point cloud data. The goal was to identify the one that most profoundly enhanced the neural network’s knowledge. Two deep learning techniques, “RandLaNet” and “KPConv”, were adopted, and their parameters were modified for different scenarios. Efficient feature engineering and selection for the fusion step facilitated the learning process and improved the semantic segmentation results. Our contribution provides a good solution for addressing some challenges, particularly for more accurate extraction of semantically rich objects from the urban environment. The experimental results have demonstrated that Scenario 1 has higher precision (88%) on the SensatUrban dataset compared to the baseline approach (71%), the Scenario 2 approach (85%), and the Scenario 3 approach (84%). Furthermore, the qualitative results obtained by the first scenario are close to the ground truth. Therefore, it was identified as the efficient fusion approach for point cloud-enriched semantic segmentation, which we have named the efficient prior-level fusion (Efficient-PLF) approach.
Disciplines :
Earth sciences & physical geography
Author, co-author :
Ballouch, Zouhair ; Université de Liège - ULiège > Sphères ; College of Geomatic Sciences and Surveying Engineering, IAV Hassan II, Rabat, Morocco
Hajji, Rafika ; Université de Liège - ULiège > Département de géographie > Geospatial Data Science and City Information Modelling (GeoScITY) ; College of Geomatic Sciences and Surveying Engineering, IAV Hassan II, Rabat, Morocco
Shahat E. Hyun C.T. Yeom C. City Digital Twin Potentials: A Review and Research Agenda Sustainability 2021 13 3386 10.3390/su13063386
Ruohomäki T. Airaksinen E. Huuska P. Kesäniemi O. Martikka M. Suomisto J. Smart City Platform Enabling Digital Twin Proceedings of the 2018 International Conference on Intelligent Systems (IS) Funchal, Portugal 25–27 September 2018 155 161
White G. Zink A. Codecá L. Clarke S. A Digital Twin Smart City for Citizen Feedback Cities 2021 110 103064 10.1016/j.cities.2020.103064
Zhang J. Zhao X. Chen Z. Lu Z. A Review of Deep Learning-Based Semantic Segmentation for Point Cloud IEEE Access 2019 7 179118 179133 10.1109/ACCESS.2019.2958671
Hu Q. Yang B. Xie L. Rosa S. Guo Y. Wang Z. Trigoni N. Markham A. RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Seattle, WA, USA 13–19 June 2020 11105 11114
Landrieu L. Simonovsky M. Large-Scale Point Cloud Semantic Segmentation with Superpoint Graphs Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Salt Lake City, UT, USA 18–23 June 2018 4558 4567
Zhang R. Wu Y. Jin W. Meng X. Deep-Learning-Based Point Cloud Semantic Segmentation: A Survey Electronics 2023 12 3642 10.3390/electronics12173642
Thomas H. Qi C.R. Deschaud J.-E. Marcotegui B. Goulette F. Guibas L.J. KPConv: Flexible and Deformable Convolution for Point Clouds Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV) Seoul, Republic of Korea 27 October–2 November 2019 6411 6420
Ballouch Z. Hajji R. Ettarid M. Toward a Deep Learning Approach for Automatic Semantic Segmentation of 3D Lidar Point Clouds in Urban Areas Geospatial Intelligence: Applications and Future Trends Barramou F. El Brirchi E.H. Mansouri K. Dehbi Y. Springer International Publishing Cham, Switzerland 2022 67 77 978-3-030-80458-9
Weinmann M. Weinmann M. Fusion of hyperspectral, multispectral, color and 3D point cloud information for the semantic interpretation of urban environments Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2019 XLII-2/W13 1899 1906 10.5194/isprs-archives-XLII-2-W13-1899-2019
Hu Q. Yang B. Khalid S. Xiao W. Trigoni N. Markham A. Towards Semantic Segmentation of Urban-Scale 3D Point Clouds: A Dataset, Benchmarks and Challenges Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Nashville, TN, USA 20–25 June 2021 4977 4987
Gao W. Nan L. Boom B. Ledoux H. SUM: A Benchmark Dataset of Semantic Urban Meshes ISPRS J. Photogramm. Remote Sens. 2021 179 108 120 10.1016/j.isprsjprs.2021.07.008
Hackel T. Savinov N. Ladicky L. Wegner J.D. Schindler K. Pollefeys M. Semantic3D.Net: A New Large-Scale Point Cloud Classification Benchmark arXiv 2017 1704.03847 10.5194/isprs-annals-IV-1-W1-91-2017
Chen X. Jia D. Zhang W. Integrating UAV Photogrammetry and Terrestrial Laser Scanning for Three-Dimensional Geometrical Modeling of Post-Earthquake County of Beichuan Proceedings of the 18th International Conference on Computing in Civil and Building Engineering Toledo Santos E. Scheer S. Springer International Publishing Cham, Switzerland 2021 1086 1098
Oh S.-I. Kang H.-B. Object Detection and Classification by Decision-Level Fusion for Intelligent Vehicle Systems Sensors 2017 17 207 10.3390/s17010207
Chen Y. Liu X. Xiao Y. Zhao Q. Wan S. Three-Dimensional Urban Land Cover Classification by Prior-Level Fusion of LiDAR Point Cloud and Optical Imagery Remote Sens. 2021 13 4928 10.3390/rs13234928
Zhang R. Li G. Li M. Wang L. Fusion of Images and Point Clouds for the Semantic Segmentation of Large-Scale 3D Scenes Based on Deep Learning ISPRS J. Photogramm. Remote Sens. 2018 143 85 96 10.1016/j.isprsjprs.2018.04.022
Ballouch Z. Hajji R. Poux F. Kharroubi A. Billen R. A Prior Level Fusion Approach for the Semantic Segmentation of 3D Point Clouds Using Deep Learning Remote Sens. 2022 14 3415 10.3390/rs14143415
Poliyapram V. Wang W. Nakamura R. A Point-Wise LiDAR and Image Multimodal Fusion Network (PMNet) for Aerial Point Cloud 3D Semantic Segmentation Remote Sens. 2019 11 2961 10.3390/rs11242961
Ye C. Pan H. Yu X. Gao H. A Spatially Enhanced Network with Camera-Lidar Fusion for 3D Semantic Segmentation Neurocomputing 2022 484 59 66 10.1016/j.neucom.2020.12.135
Luo S. Wang C. Xi X. Zeng H. Li D. Xia S. Wang P. Fusion of Airborne Discrete-Return LiDAR and Hyperspectral Data for Land Cover Classification Remote Sens. 2016 8 3 10.3390/rs8010003
Mirzapour F. Ghassemian H. Improving Hyperspectral Image Classification by Combining Spectral, Texture, and Shape Features Int. J. Remote Sens. 2015 36 1070 1096 10.1080/01431161.2015.1007251
Bai X. Liu C. Ren P. Zhou J. Zhao H. Su Y. Object Classification via Feature Fusion Based Marginalized Kernels IEEE Geosci. Remote Sens. Lett. 2015 12 8 12 10.1109/LGRS.2014.2322953
Zhang Y. Chi M. Mask-R-FCN: A Deep Fusion Network for Semantic Segmentation IEEE Access 2020 8 155753 155765 10.1109/ACCESS.2020.3012701
Tabib Mahmoudi F. Samadzadegan F. Reinartz P. Object Recognition Based on the Context Aware Decision-Level Fusion in Multiviews Imagery IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2015 8 12 22 10.1109/JSTARS.2014.2362103
Zhang R. Candra S.A. Vetter K. Zakhor A. Sensor Fusion for Semantic Segmentation of Urban Scenes Proceedings of the 2015 IEEE International Conference on Robotics and Automation (ICRA) Seattle, WA, USA 26–30 May 2015 1850 1857
Guo Y. Wang H. Hu Q. Liu H. Liu L. Bennamoun M. Deep Learning for 3D Point Clouds: A Survey IEEE Trans. Pattern Anal. Mach. Intell. 2021 43 4338 4364 10.1109/TPAMI.2020.3005434
Grzeczkowicz G. Vallet B. Semantic Segmentation of Urban Textured Meshes through Point Sampling arXiv 2023 2302.10635 10.5194/isprs-annals-V-2-2022-177-2022
Li W. Zhan L. Min W. Zou Y. Huang Z. Wen C. Semantic Segmentation of Point Cloud With Novel Neural Radiation Field Convolution IEEE Geosci. Remote Sens. Lett. 2023 20 1 5 10.1109/LGRS.2023.3307421
Lin Y. Vosselman G. Cao Y. Yang M.Y. Local and Global Encoder Network for Semantic Segmentation of Airborne Laser Scanning Point Clouds ISPRS J. Photogramm. Remote Sens. 2021 176 151 168 10.1016/j.isprsjprs.2021.04.016
Song H. Jo K. Cho J. Son Y. Kim C. Han K. A Training Dataset for Semantic Segmentation of Urban Point Cloud Map for Intelligent Vehicles ISPRS J. Photogramm. Remote Sens. 2022 187 159 170 10.1016/j.isprsjprs.2022.02.007
Atik M.E. Duran Z. Seker D.Z. Machine Learning-Based Supervised Classification of Point Clouds Using Multiscale Geometric Features ISPRS Int. J. Geo-Inf. 2021 10 187 10.3390/ijgi10030187
Özdemir E. Remondino F. Classification of aerial point clouds with deep learning Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2019 XLII-2/W13 103 110 10.5194/isprs-archives-XLII-2-W13-103-2019
Kingma D.P. Ba J. Adam: A Method for Stochastic Optimization arXiv 2014 1412.6980
Qi C.R. Su H. Mo K. Guibas L.J. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Honolulu, HI, USA 21–26 July 2017 652 660
Qi C.R. Yi L. Su H. Guibas L.J. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space Proceedings of the Advances in Neural Information Processing Systems Long Beach, CA, USA 4–9 December 2017 Curran Associates, Inc. Red Hook, NY, USA 2017 Volume 30
Tatarchenko M. Park J. Koltun V. Zhou Q.-Y. Tangent Convolutions for Dense Prediction in 3D Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Salt Lake City, UT, USA 18–23 June 2018 3887 3896
