3D point cloud; aerial images; semantic segmentation; data fusion; deep learning
Abstract :
[en] Three-dimensional digital models play a pivotal role in city planning, monitoring, and sustainable management of smart and Digital Twin Cities (DTCs). In this context, semantic segmentation of airborne 3D point clouds is crucial for modeling, simulating, and understanding large-scale urban environments. Previous research studies have demonstrated that the performance of 3D semantic segmentation can be improved by fusing 3D point clouds and other data sources. In this paper, a new prior-level fusion approach is proposed for semantic segmentation of large-scale urban areas using optical images and point clouds. The proposed approach uses image classification obtained by the Maximum Likelihood Classifier as the prior knowledge for 3D semantic segmentation. Afterwards, the raster values from classified images are assigned to Lidar point clouds at the data preparation step. Finally, an advanced Deep Learning model (RandLaNet) is adopted to perform the 3D semantic segmentation. The results show that the proposed approach provides good results in terms of both evaluation metrics and visual examination with a higher Intersection over Union (96%) on the created dataset, compared with (92%) for the non-fusion approach.
Hajji, Rafika ; Université de Liège - ULiège > Département de géographie > Unité de Géomatique - Topographie et géométrologie
Poux, Florent ; Université de Liège - ULiège > Département de géographie > Unité de Géomatique - Topographie et géométrologie
Kharroubi, Abderrazzaq ; Université de Liège - ULiège > Département de géographie > Unité de Géomatique - Topographie et géométrologie
Billen, Roland ; Université de Liège - ULiège > Département de géographie > Département de Géographie : Plateforme "GITAN" (Geographic Information Technological Aid Network"
Language :
English
Title :
A Prior Level Fusion Approach for the Semantic Segmentation of 3D Point Clouds Using Deep Learning
Yan J. Zlatanova S. Aleksandrov M. Diakite A. Pettit C.J. Integration of 3D Objects and Terrain for 3D Modelling Supporting the Digital Twin Proceedings of the 14th 3D GeoInfo Conference Singapore 24–27 September 2019
Wang R. Peethambaran J. Chen D. LiDAR Point Clouds to 3-D Urban Models: A Review IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2018 11 606 627 10.1109/JSTARS.2017.2781132
Macher H. Landes T. Grussenmeyer P. From Point Clouds to Building Information Models: 3D Semi-Automatic Reconstruction of Indoors of Existing Buildings Appl. Sci. 2017 7 1030 10.3390/app7101030
Guo Y. Wang H. Hu Q. Liu H. Liu L. Bennamoun M. Deep Learning for 3D Point Clouds: A Survey IEEE Trans. Pattern Anal. Mach. Intell. 2021 43 4338 4364 10.1109/TPAMI.2020.3005434 32750799
Beil C. Kutzner T. Schwab B. Willenborg B. Gawronski A. Kolbe T.H. Integration of 3D Point Clouds with Semantic 3D City Models—Providing Semantic Information Beyond Classification ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 2021 VIII-4/W2-2021 105 112 10.5194/isprs-annals-VIII-4-W2-2021-105-2021
Martinovic A. Knopp J. Riemenschneider H. Van Gool L. 3D All The Way: Semantic Segmentation of Urban Scenes From Start to End in 3D Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Boston, MA, USA 7–12 June 2015 4456 4465
Zhang L. Zhang L. Deep Learning-Based Classification and Reconstruction of Residential Scenes From Large-Scale Point Clouds IEEE Trans. Geosci. Remote Sens. 2018 56 1887 1897 10.1109/TGRS.2017.2769120
Murtiyoso A. Veriandi M. Suwardhi D. Soeksmantono B. Harto A.B. Automatic Workflow for Roof Extraction and Generation of 3D CityGML Models from Low-Cost UAV Image-Derived Point Clouds ISPRS Int. J. Geo-Inf. 2020 9 743 10.3390/ijgi9120743
Gobeawan L. Lin E.S. Tandon A. Yee A.T.K. Khoo V.H.S. Teo S.N. Yi S. Lim C.W. Wong S.T. Wise D.J. et al. Modeling Trees for Virtual Singapore: From Data Acquisition to CityGML Models Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2018 XLII-4/W10 55 62 10.5194/isprs-archives-XLII-4-W10-55-2018
Loutfia E. Mahmoud H. Amr A. Mahmoud S. 3D Model Reconstruction from Aerial Ortho-Imagery and LiDAR Data J. Geomat. 2017 11 9
Kwak E. Automatic 3D Building Model Generation by Integrating LiDAR and Aerial Images Using a Hybrid Approach Ph.D. Thesis University of Calgary Calgary, AB, Canada 2013 10.11575/PRISM/25078
Chen X. Jia D. Zhang W. Integrating UAV Photogrammetry and Terrestrial Laser Scanning for Three-Dimensional Geometrical Modeling of Post-Earthquake County of Beichuan Proceedings of the 18th International Conference on Computing in Civil and Building Engineering São Paulo, Brazil 18–20 August 2020 Toledo Santos E. Scheer S. Springer International Publishing Cham, Switzerland 2021 1086 1098
Luo H. Khoshelham K. Fang L. Chen C. Unsupervised Scene Adaptation for Semantic Segmentation of Urban Mobile Laser Scanning Point Clouds ISPRS J. Photogramm. Remote Sens. 2020 169 253 267 10.1016/j.isprsjprs.2020.10.002
Marmanis D. Wegner J.D. Galliani S. Schindler K. Datcu M. Stilla U. Semantic Segmentation of Aerial Images with an Ensemble of CNSS Proceedings of the ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences Prague, Czech Republic 12–19 July 2016 Halounova L. Schindler K. Limpouch A. Šafář V. Pajdla T. Mayer H. Oude Elberink S. Mallet C. Rottensteiner F. Skaloud J. et al. Copernicus Publications Göttingen, Germany 2016 Volume III–3 473 480
Castillo-Navarro J. Le Saux B. Boulch A. Lefèvre S. Réseaux de Neurones Semi-Supervisés Pour La Segmentation Sémantique En Télédétection Proceedings of the Colloque GRETSI sur le Traitement du Signal et des Images Lille, France 26–29 August 2019
Garcia-Garcia A. Orts-Escolano S. Oprea S. Villena-Martinez V. Garcia-Rodriguez J. A Review on Deep Learning Techniques Applied to Semantic Segmentation arXiv 2017 170406857
Poliyapram V. Wang W. Nakamura R. A Point-Wise LiDAR and Image Multimodal Fusion Network (PMNet) for Aerial Point Cloud 3D Semantic Segmentation Remote Sens. 2019 11 2961 10.3390/rs11242961
Zhao L. Zhou H. Zhu X. Song X. Li H. Tao W. LIF-Seg: LiDAR and Camera Image Fusion for 3D LiDAR Semantic Segmentation arXiv 2021 210807511
Meyer G.P. Charland J. Hegde D. Laddha A. Vallespi-Gonzalez C. Sensor Fusion for Joint 3D Object Detection and Semantic Segmentation Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) Long Beach, CA, USA 16–17 June 2019 1230 1237
Zhang R. Li G. Li M. Wang L. Fusion of Images and Point Clouds for the Semantic Segmentation of Large-Scale 3D Scenes Based on Deep Learning ISPRS J. Photogramm. Remote Sens. 2018 143 85 96 10.1016/j.isprsjprs.2018.04.022
Ballouch Z. Hajji R. Ettarid M. The Contribution of Deep Learning to the Semantic Segmentation of 3D Point-Clouds in Urban Areas Proceedings of the 2020 IEEE International Conference of Moroccan Geomatics (Morgeo) Casablanca, Morocco 11–13 May 2020 1 6
Khodadadzadeh M. Li J. Prasad S. Plaza A. Fusion of Hyperspectral and LiDAR Remote Sensing Data Using Multiple Feature Learning IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2015 8 2971 2983 10.1109/JSTARS.2015.2432037
Zhang J. Lin X. Advances in Fusion of Optical Imagery and LiDAR Point Cloud Applied to Photogrammetry and Remote Sensing Int. J. Image Data Fusion 2017 8 1 31 10.1080/19479832.2016.1160960
Ghamisi P. Rasti B. Yokoya N. Wang Q. Hofle B. Bruzzone L. Bovolo F. Chi M. Anders K. Gloaguen R. et al. Multisource and Multitemporal Data Fusion in Remote Sensing: A Comprehensive Review of the State of the Art IEEE Geosci. Remote Sens. Mag. 2019 7 6 39 10.1109/MGRS.2018.2890023
Luo S. Wang C. Xi X. Zeng H. Li D. Xia S. Wang P. Fusion of Airborne Discrete-Return LiDAR and Hyperspectral Data for Land Cover Classification Remote Sens. 2015 8 3 10.3390/rs8010003
Armeni I. Sener O. Zamir A.R. Jiang H. Brilakis I. Fischer M. Savarese S. 3D Semantic Parsing of Large-Scale Indoor Spaces Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Las Vegas, NV, USA 27–30 June 2016 1534 1543
Hackel T. Savinov N. Ladicky L. Wegner J.D. Schindler K. Pollefeys M. Semantic3D.Net: A New Large-Scale Point Cloud Classification Benchmark arXiv 2017 170403847 10.5194/isprs-annals-IV-1-W1-91-2017
Hu Q. Yang B. Khalid S. Xiao W. Trigoni N. Markham A. Towards Semantic Segmentation of Urban-Scale 3D Point Clouds: A Dataset, Benchmarks and Challenges Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Nashville, TN, USA 20–25 June 2021 4977 4987
Xu Y. Hoegner L. Tuttas S. Stilla U. Voxel- and Graph-Based Point Cloud Segmentation of 3D Scenes Using Perceptual Grouping Laws ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 2017 IV-1/W1 43 50 10.5194/isprs-annals-IV-1-W1-43-2017
Boulch A. Saux B.L. Audebert N. Unstructured Point Cloud Semantic Labeling Using Deep Segmentation Networks Proceedings of the Eurographics Workshop 3D Object Retrieval Lyon, France 23–24 April 2017 8p 10.2312/3DOR.20171047
Tchapmi L. Choy C. Armeni I. Gwak J. Savarese S. SEGCloud: Semantic Segmentation of 3D Point Clouds Proceedings of the 2017 International Conference on 3D Vision (3DV) Qingdao, China 10–12 October 2017 537 547
Landrieu L. Simonovsky M. Large-Scale Point Cloud Semantic Segmentation with Superpoint Graphs Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City UT, USA 18–23 June 2018 4558 4567
Hu Q. Yang B. Xie L. Rosa S. Guo Y. Wang Z. Trigoni N. Markham A. RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Seattle, WA, USA 13–19 June 2020 11105 11114
Megahed Y. Shaker A. Yan W.Y. Fusion of Airborne LiDAR Point Clouds and Aerial Images for Heterogeneous Land-Use Urban Mapping Remote Sens. 2021 13 814 10.3390/rs13040814
Ghassemian H. A Review of Remote Sensing Image Fusion Methods Inf. Fusion 2016 32 75 89 10.1016/j.inffus.2016.03.003
Chen Y. Liu X. Xiao Y. Zhao Q. Wan S. Three-Dimensional Urban Land Cover Classification by Prior-Level Fusion of LiDAR Point Cloud and Optical Imagery Remote Sens. 2021 13 4928 10.3390/rs13234928
Ballouch Z. Hajji R. Ettarid M. Toward a Deep Learning Approach for Automatic Semantic Segmentation of 3D Lidar Point Clouds in Urban Areas Geospatial Intelligence: Applications and Future Trends Barramou F. El Brirchi E.H. Mansouri K. Dehbi Y. Springer International Publishing Cham, Switzerland 2022 67 77 978-3-030-80458-9
Cao Z. Fu K. Lu X. Diao W. Sun H. Yan M. Yu H. Sun X. End-to-End DSM Fusion Networks for Semantic Segmentation in High-Resolution Aerial Images IEEE Geosci. Remote Sens. Lett. 2019 16 1766 1770 10.1109/LGRS.2019.2907009
Pan X. Gao L. Marinoni A. Zhang B. Yang F. Gamba P. Semantic Labeling of High Resolution Aerial Imagery and LiDAR Data with Fine Segmentation Network Remote Sens. 2018 10 743 10.3390/rs10050743
Zhang W. Huang H. Schmitz M. Sun X. Wang H. Mayer H. Effective Fusion of Multi-Modal Remote Sensing Data in a Fully Convolutional Network for Semantic Labeling Remote Sens. 2017 10 52 10.3390/rs10010052
Lodha S.K. Fitzpatrick D.M. Helmbold D.P. Aerial Lidar Data Classification Using AdaBoost Proceedings of the Sixth International Conference on 3-D Digital Imaging and Modeling (3DIM 2007) Montreal, QC, Canada 21–23 August 2007 435 442
Weinmann M. Weinmann M. Fusion of Hyperspectral, Multispectral, Color and 3D Point Cloud Information for the Semantic Interpretation of Urban Environments Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2019 XLII-2/W13 1899 1906 10.5194/isprs-archives-XLII-2-W13-1899-2019
Onojeghuo A.O. Onojeghuo A.R. Object-Based Habitat Mapping Using Very High Spatial Resolution Multispectral and Hyperspectral Imagery with LiDAR Data Int. J. Appl. Earth Obs. Geoinf. 2017 59 79 91 10.1016/j.jag.2017.03.007
Yousefhussien M. Kelbe D.J. Ientilucci E.J. Salvaggio C. A Multi-Scale Fully Convolutional Network for Semantic Labeling of 3D Point Clouds ISPRS J. Photogramm. Remote Sens. 2018 143 191 204 10.1016/j.isprsjprs.2018.03.018
Siljander M. Adero N.J. Gitau F. Nyambu E. Land Use/Land Cover Classification for the Iron Mining Site of Kishushe, Kenya: A Feasibility Study of Traditional and Machine Learning Algorithms Afr. J. Min. Entrep. Nat. Resour. Manag. 2020 2 115 124
Asad M.H. Bais A. Weed Detection in Canola Fields Using Maximum Likelihood Classification and Deep Convolutional Neural Network Inf. Process. Agric. 2020 7 535 545 10.1016/j.inpa.2019.12.002
Gevana D. Camacho L. Carandang A. Camacho S. Im S. Land Use Characterization and Change Detection of a Small Mangrove Area in Banacon Island, Bohol, Philippines Using a Maximum Likelihood Classification Method For. Sci. Technol. 2015 11 197 205 10.1080/21580103.2014.996611
Berila A. Isufi F. Two Decades (2000–2020) Measuring Urban Sprawl Using GIS, RS and Landscape Metrics: A Case Study of Municipality of Prishtina (Kosovo) J. Ecol. Eng. 2021 22 114 125 10.12911/22998993/137078
Cortinhal T. Tzelepis G. Erdal Aksoy E. SalsaNext: Fast, Uncertainty-Aware Semantic Segmentation of LiDAR Point Clouds Advances in Visual Computing, Proceedings of the 15th International Symposium on Visual Computing, San Diego, CA, USA, 5–7 October 2020 Bebis G. Yin Z. Kim E. Bender J. Subr K. Kwon B.C. Zhao J. Kalkofen D. Baciu G. Springer International Publishing Cham, Switzerland 2020 207 222
Xu C. Wu B. Wang Z. Zhan W. Vajda P. Keutzer K. Tomizuka M. SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation Computer Vision—ECCV 2020, Proceedings of the 16th European Conference on Computer Vision, Glasgow, UK, 23–28 August 2020 Vedaldi A. Bischof H. Brox T. Frahm J.-M. Springer International Publishing Cham, Switzerland 2020 1 19
Li Y. Tong G. Du X. Yang X. Zhang J. Yang L. A Single Point-Based Multilevel Features Fusion and Pyramid Neighborhood Optimization Method for ALS Point Cloud Classification Appl. Sci. 2019 9 951 10.3390/app9050951
