Enhancing Semantic Segmentation of Large-Scale 3D Point Clouds with Deep Learning Techniques for Urban Digital Twin Creation

Classified point clouds often serve as the primary data source for decision-making scenarios. For example, these data can be used as the main layer for creating Digital Twins, as a basis for urban simulation studies (such as flood simulations, vegetation inventories, rooftop solar potential, etc.), as a reference for detecting object changes, or as a foundation for automatic 3D modeling of the urban environment… The applications are numerous and potentially growing, especially when considering classified point clouds as digital assets. However, the automatic and precise extraction of maximum semantic information from an urban environment (such as parking lots, street furniture, pedestrian pathways, etc.) remains a challenge. The growing development of LiDAR technology, in terms of precision and spatial resolution, provides a good opportunity to offer reliable semantic segmentation in large-scale urban environments. Additionally, the advancement of Deep Learning techniques has revolutionized the field of computer vision and demonstrated high performance in semantic segmentation. This thesis aims to address the challenges of precisely extracting urban details from airborne LiDAR point clouds using Deep Learning techniques, in order to meet the various needs of Urban Digital Twins. Several challenges related to object extraction from airborne point clouds are explored, particularly the adaptation of Deep Learning techniques, fusion of point clouds with corresponding images, efficient feature engineering and selection, semantic segmentation, automatic 3D modeling from semantic segmentation, as well as visualization and interaction with cognitive decision-making systems. Several fusion scenarios of point clouds and images were developed and evaluated, leading to a 3D semantic segmentation fusion approach that is less data-intensive, and one that effectively extracted the maximum semantic information from the urban environment, demonstrating good results in terms of both quality and quantity. Another fusion approach was recommended due to its performance in specific semantic classes. Furthermore, a new approach was developed to exploit enriched semantic 3D point clouds as an alternative to 3D models in urban simulations. This approach was designed to meet the needs of Digital Twins. Modeling procedures were implemented for each extracted object, enabling the automatic production of 3D urban models. Finally, a case study was conducted to create the foundational elements of a Digital Twin for the city of Liège, Belgium. Several concepts, algorithms, codes, and resources are provided to reproduce the results and expand current applications.