Augmented Reality; Web-AR; Marker-Less AR; 3D Point cloud; Semantics; Classification; Virtual environment
Abstract :
[en] Mobile Augmented Reality (MAR) attracts significant research and development efforts from both the industry and academia, but rarely integrate massive 3D dataset’s interactions. The emergence of dedicated AR devices and powerful Software Development Kit (ARCore for android and ARKit for iOS) improves performance on mobile devices (Smartphones and tablets). This is aided by new sensor integration and advances in computer vision that fuels the development of MAR. In this paper, we propose a direct integration of massive 3D point clouds with semantics in a web-based marker-less mobile Augmented Reality (AR) application for real-time visualization. We specifically investigate challenges linked to point cloud data structure and semantic injection. Our solution consolidates some of the overarching principles of AR, of which pose estimation, registration and 3D tracking. The developed AR system is tested on mobile phones web-browsers providing clear insights on the performance of the system. Promising results highlight a number of frame per second varying between 27 and 60 for a real-time point budget of 4.3 million points. The point cloud tested is composed of 29 million points and shows how our indexation strategy permits the integration of massive point clouds aiming at the point budget. The results also gives research directions concerning the dependence and delay related to the quality of the network connection, and the battery consumption since portable sensors are used all the time.
CloudCompare, 2019. CloudCompare 3D point cloud and mesh processing software Open Source Project www. cloudcompare. org (25 december 19).
Codina, Castells-Rufas, Carrabina, Salmon, Ayuso, Guerendiain, Alvarez, 2019. Augmented Reality for Emergency Situations in Buildings with the Support of Indoor Localization. Proceedings 31, 76. https://doi. org/10. 3390/proceedings2019031076
de Ruyter, K., Heller, J., Hilken, T., Chylinski, M., Keeling, D. I., Mahr, D., 2020. Seeing with the Customer's Eye: Exploring the Challenges and Opportunities of AR Advertising. J. Advert. 0, 1-16. https://doi. org/10. 1080/00913367. 2020. 1740123
Fraga-Lamas, P., Fernandez-Carames, T. M., Blanco-Novoa, O., Vilar-Montesinos, M. A., 2018. A Review on Industrial Augmented Reality Systems for the Industry 4. 0 Shipyard. IEEE Access 6, 13358-13375. https://doi. org/10. 1109/ACCESS. 2018. 2808326
Fraiss, S. M., 2017. Rendering Large Point Clouds in Unity. https://www. cg. tuwien. ac. at/research/publications/2017/FRAISS-2017-PCU/Hübner, P., Clintworth, K., Liu, Q., Weinmann, M., Wursthorn, S., 2020. Evaluation of hololens tracking and depth sensing for indoor mapping applications. Sensors (Switzerland) 20. https://doi. org/10. 3390/s20041021
Karakus, M., Ersozlu, A., Clark, A. C., 2019. Augmented reality research in education: A bibliometric study. Eurasia J. Math. Sci. Technol. Educ. 15. https://doi. org/10. 29333/ejmste/103904
Khairnar, Khushal, Khairnar, Kamleshwar, Mane, S., Chaudhari, R., 2015. Furniture Layout Application Based on Marker Detection and Using Augmented Reality. Int. Res. J. Eng. Technol. 2, 540-544. https://www. irjet. net/archives/V2/i7/IRJET-V2I780. pdf
Kharroubi, A., Hajji, R., Billen, R., Poux, F., 2019. Classification and Integration of Massive 3D Points Clouds in a Virtual Reality (Vr) Environment. ISPRS-Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. XLII-2/W17, 165-171. https://doi. org/10. 5194/isprs-archives-XLII-2-W17-165-2019
Khoshelham, K., Tran, H., Acharya, D., 2019. Indoor mapping eyewear: Geometric evaluation of spatial mapping capability of hololens. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.-ISPRS Arch. 42, 805-810. https://doi. org/10. 5194/isprs-archives-XLII-2-W13-805-2019
Martinez-rubi, O., Verhoeven, S., Meersbergen, M. Van, Schuetz, M., van Oosterom, P., Gonclves, R., Tijssen, T., 2015. Taming the beast: Free and open-source massive point cloud web visualization. Capturing Real. Forum 2015 23-25. https://doi. org/10. 13140/RG. 2. 1. 1731. 4326
Mures, O. A., Jaspe, A., Padrón, E. J., Rabuñal, J. R., 2018. Virtual reality and point-Based rendering in architecture and heritage. Virtual Augment. Real. Concepts, Methodol. Tools, Appl. 1, 549-565. https://doi. org/10. 4018/978-1-5225-5469-1. ch026
Paavilainen, J., Korhonen, H., Alha, K., Stenros, J., Koskinen, E., Mäyrä, F., 2017. The Pokémon go experience: A location-based augmented reality mobile game goes mainstream. Conf. Hum. Factors Comput. Syst.-Proc. 2017-May, 2493-2498. https://doi. org/10. 1145/3025453. 3025871
Parra, C., Cebollada, S., Paya, L., Holloway, M., Reinoso, O., 2020. A novel method to estimate the position of a mobile robot in underfloor environments using RGB-D point clouds. IEEE Access 8, 9084-9101. https://doi. org/10. 1109/ACCESS. 2020. 2964317
Pierdicca, R., Paolanti, M., Matrone, F., Martini, M., Morbidoni, C., Malinverni, E. S., Frontoni, E., Lingua, A. M., 2020. Point cloud semantic segmentation using a deep learning framework for cultural heritage. Remote Sens. 12, 1-23. https://doi. org/10. 3390/rs12061005
Potree, 2019. PotreeConverter. https://github. com/potree/PotreeConverter (22 December 19).
Poux, F., Billen, R., 2019. Voxel-based 3D Point Cloud Semantic Segmentation: Unsupervised Geometric and Relationship Featuring vs Deep Learning Methods. ISPRS Int. J. Geo-Information 8, 213. https://doi. org/10. 3390/ijgi8050213
Poux, F., Neuville, R., Hallot, P., Billen, R., 2017. MODEL FOR SEMANTICALLY RICH POINT CLOUD DATA. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. IV-4/W5, 107-115. https://doi. org/10. 5194/isprs-annals-IV-4-W5-107-2017
Poux, F., Neuville, R., Nys, G.-A., Billen, R., 2018. 3D Point Cloud Semantic Modelling: Integrated Framework for Indoor Spaces and Furniture. Remote Sens. 10, 1412. https://doi. org/10. 3390/rs10091412
Poux, Florent, Neuville, R., Van Wersch, L., Nys, G.-A., Billen, R., 2017. 3D Point Clouds in Archaeology: Advances in Acquisition, Processing and Knowledge Integration Applied to Quasi-Planar Objects. Geosciences 7, 96. https://doi. org/10. 3390/geosciences7040096
Qiao, X., Ren, P., Dustdar, S., Liu, L., Ma, H., Chen, J., 2019. Web AR: A Promising Future for Mobile Augmented Reality-State of the Art, Challenges, and Insights. Proc. IEEE 107, 651-666. https://doi. org/10. 1109/JPROC. 2019. 2895105
Raimbaud, P., Lou, R., Merienne, F., Danglade, F., Figueroa, P., Hernandez, J. T., 2019. BIM-based mixed reality application for supervision of construction. 26th IEEE Conf. Virtual Real. 3D User Interfaces, VR 2019-Proc. 1903-1907. https://doi. org/10. 1109/VR. 2019. 8797784
ahin, D., Togay, A., 2016. Augmented reality applications in product design process. Glob. J. Humanit. Soc. Sci. https://doi. org/10. 18844/gjhss. v0i0. 288
Shih, N. J., Diao, P. H., Chen, Y., 2019. ARTS, an AR tourism system, for the integration of 3D scanning and smartphone AR in cultural heritage tourism and pedagogy. Sensors (Switzerland) 19. https://doi. org/10. 3390/s19173725
Stojanovic, V., Trapp, M., Richter, R., Döllner, J., 2019. Generation of approximate 2D and 3D floor plans from 3D point clouds. VISIGRAPP 2019-Proc. 14th Int. Jt. Conf. Comput. Vision, Imaging Comput. Graph. Theory Appl. 1, 177-184. https://doi. org/10. 5220/0007247601770184
Wahl, R., Klein, R., 2007. Efficient RANSAC for Point-Cloud Shape Detection 0, 1-12. https://doi. org/10. 1111/j. 1467-8659. 2007. 01016. x Zhang, W., Qi, J., Wan, P., Wang, H., Xie, D., Wang, X., Yan, G., 2016. An easy-to-use airborne LiDAR data filtering method based on cloth simulation. Remote Sens. 8, 1-22. https://doi. org/10. 3390/rs8060501
