3D modeling; 3D reconstruction; cultural heritage; heritage information system; laser scanning; photogrammetry; point clouds; reality capture; virtual reality
Abstract :
[en] Reality capture allows for the reconstruction, with a high accuracy, of the physical reality of cultural heritage sites. Obtained 3D models are often used for various applications such as promotional content creation, virtual tours, and immersive experiences. In this paper, we study new ways to interact with these high-quality 3D reconstructions in a real-world scenario. We propose a user-centric product design to create a virtual reality (VR) application specifically intended for multi-modal purposes. It is applied to the castle of Jehay (Belgium), which is under renovation, to permit multi-user digital immersive experiences. The article proposes a high-level view of multi-disciplinary processes, from a needs analysis to the 3D reality capture workflow and the creation of a VR environment incorporated into an immersive application. We provide several relevant VR parameters for the scene optimization, the locomotion system, and the multi-user environment definition that were tested in a heritage tourism context [fr] La capture de la réalité permet de reconstituer, avec une grande précision, la réalité physique des sites du patrimoine culturel. Les modèles 3D obtenus sont souvent utilisés pour diverses applications telles que la création de contenu promotionnel, les visites virtuelles et les expériences immersives. Dans cet article, nous étudions de nouvelles façons d'interagir avec ces reconstructions 3D de haute qualité dans un scénario réel. Nous proposons une conception de produit centrée sur l'utilisateur pour créer une application de réalité virtuelle (RV) spécifiquement destinée à des fins multimodales. Elle est appliquée au château de Jehay (Belgique), qui est en cours de rénovation, pour permettre des expériences numériques immersives multi-utilisateurs. L'article propose une vue de haut niveau des processus multidisciplinaires, de l'analyse des besoins au flux de travail de capture de la réalité 3D et à la création d'un environnement de RV intégré à une application immersive. Nous fournissons plusieurs paramètres de RV pertinents pour l'optimisation de la scène, le système de locomotion et la définition de l'environnement multi-utilisateurs qui ont été testés dans un contexte de tourisme patrimonial
Traduit avec www.DeepL.com/Translator (version gratuite)
Ortiz, P.; Sánchez, H.; Pires, H.; Pérez, J.A. Experiences about fusioning 3D digitalization techniques for cultural heritage documentation. In Proceedings of the ISPRS Commission V Symposium 'Image Engineering and Vision Metrology', Dresden, Germany, 25-27 September 2006; pp. 224-229.
Mayer, H.; Mosch, M.; Peipe, J. Comparison of photogrammetric and computer vision techniques: 3D reconstruction and visualization of Wartburg Castle. In Proceedings of the XIXth International Symposium, CIPA 2003: New Perspectives to Save Cultural Heritage, Antalya, Turkey, 30 September-4 October 2003; pp. 103-108.
Koch, M.; Kaehler, M. Combining 3D laser-Scanning and close-range Photogrammetry-An approach to Exploit the Strength of Both methods. In Proceedings of the Computer Applications to Archaeology, Williamsburg, VA, USA, 22-26 March 2009; pp. 22-26.
Bekele, M.K.; Pierdicca, R.; Frontoni, E.; Malinverni, E.S.; Gain, J. A survey of augmented, virtual, and mixed reality for cultural heritage. J. Comput. Cult. Herit. 2018, 11, 1-36.
Kersten, T.P.; Tschirschwitz, F.; Deggim, S.; Lindstaedt, M. Virtual reality for cultural heritage monuments-from 3D data recording to immersive visualisation. In Proceedings of the Euro-Mediterranean Conference, Nicosia, Cyprus, 29 October 2018; Springer: Cham, Switzerland, 2018; pp. 74-83.
Santana, M. Heritage recording, documentation and information systems in preventive maintenance. Reflect. Prev. Conserv. Maint. Monit. Monum. Sites 2013, 6, 10.
Chiabrando, F.; Donato, V.; Lo Turco, M.; Santagati, C. Cultural Heritage Documentation, Analysis and Management Using Building Information Modelling: State of the Art and Perspectives BT. In Mechatronics for Cultural Heritage and Civil Engineering; Ottaviano, E., Pelliccio, A., Gattulli, V., Eds.; Springer International Publishing: Cham, Switzerland, 2018; pp. 181-202. ISBN 978-3-319-68646-2.
Liu, X.; Wang, X.; Wright, G.; Cheng, J.; Li, X.; Liu, R. A State-of-the-Art Review on the Integration of Building Information Modeling (BIM) and Geographic Information System (GIS). Isprs Int. J. Geo-Inf. 2017, 6, 53.
Poux, F.; Billen, R. A Smart Point Cloud Infrastructure for intelligent environments. In Laser Scanning: An Emerging Technology in Structural Engineering; Lindenbergh, R., Belen, R., Eds.; ISPRS Book Series; Taylor & Francis Group/CRC Press: Boca Raton, FL, USA, 2019.
Poux, F.; Hallot, P.; Neuville, R.; Billen, R. Smart point cloud: Definition and remaining challenges. Isprs Ann. Photogramm. Remote. Sens. Spat. Inf. Sci. 2016, IV-2/W1, 119-127.
Palomar, I.J.; García Valldecabres, J.L.; Tzortzopoulos, P.; Pellicer, E. An online platform to unify and synchronise heritage architecture information. Autom. Constr. 2020, 110, 103008.
Dutailly, B.; Feruglio, V.; Ferrier, C.; Chapoulie, R.; Bousquet, B.; Bassel, L.; Mora, P.; Lacanette, D. Accéder en 3D aux données de terrain pluridisciplinaires-un outil pour l'étude des grottes ornées. In Proceedings of the Le réel et le virtuel, Marseille, France, 9-11 May 2019.
Potenziani, M.; Callieri, M.; Scopigno, R. Developing and Maintaining a Web 3D Viewer for the CH Community: An Evaluation of the 3DHOP Framework. In Proceedings of the GCH, Vienna, Austria, 12-15 November 2018; pp. 169-178.
Cheong, R. The virtual threat to travel and tourism. Tour. Manag. 1995, 16, 417-422.
Perry Hobson, J.S.; Williams, A.P. Virtual reality: A new horizon for the tourism industry. J. Vacat. Market. 1995, 1, 124-135.
Musil, S.; Pigel, G. Can Tourism be Replaced by Virtual Reality Technology? In Information and Communications Technologies in Tourism; Springer: Vienna, Austria, 1994; pp. 87-94.
Williams, P.; Hobson, J.P. Virtual reality and tourism: Fact or fantasy? Tour. Manag. 1995, 16, 423-427.
Perry Hobson, J.S.; Williams, P. Virtual Reality: The Future of Leisure and Tourism? World Leis. Recreat. 1997, 39, 34-40.
Google's latest VR app lets you gaze at prehistoric paintings Engadget. Available online: www.engadget.com (accessed on 28 March 2020).
Tsingos, N.; Gallo, E.; Drettakis, G. Perceptual audio rendering of complex virtual environments. Acm Trans. Graph. (Tog) 2004, 23, 249-258.
Dinh, H.Q.; Walker, N.; Hodges, L.F.; Song, C.; Kobayashi, A. Evaluating the importance of multi-sensory input on memory and the sense of presence in virtual environments. In Proceedings of the IEEE Virtual Reality (Cat. No. 99CB36316), Houston, TX, USA, 13-17 March 1999; pp. 222-228.
Tussyadiah, I.P.; Wang, D.; Jung, T.H.; tom Dieck, M.C. Virtual reality, presence, and attitude change: Empirical evidence from tourism. Tour. Manag. 2018, 66, 140-154.
Top Trends in the Gartner Hype Cycle for Emerging Technologies, 2017. Smarter with Gartner. Available online: www.gartner.com (accessed on 28 March 2020).
Han, D.-I.D.; Weber, J.; Bastiaansen, M.; Mitas, O.; Lub, X. Virtual and augmented reality technologies to enhance the visitor experience in cultural tourism. In Augmented Reality and Virtual Reality; Springer: Nature, Switzerland, 2019; pp. 113-128.
Guttentag, D.A. Virtual reality: Applications and implications for tourism. Tour. Manag. 2010, 31, 637-651.
Jacobson, J.; Ellis, M.; Ellis, S.; Seethaler, L. Immersive Displays for Education Using CaveUT. In Proceedings of the EdMedia + Innovate Learning, Montreal, QC, Canada, 27 June-2 July 2005; Kommers, P., Richards, G., Eds.; Association for the Advancement of Computing in Education (AACE): Montreal, QC, Canada, 2005; pp. 4525-4530.
Mikropoulos, T.A. Presence: A unique characteristic in educational virtual environments. Virtual Real. 2006, 10, 197-206.
Roussou, M. Learning by doing and learning through play: An exploration of interactivity in virtual environments for children. Comput. Entertain. (Cie) 2004, 2, 10.
Gimblett, H.R.; Richards, M.T.; Itami, R.M. RBSim: Geographic Simulation of Wilderness Recreation Behavior. J. For. 2001, 99, 36-42.
Marasco, A.; Buonincontri, P.; van Niekerk, M.; Orlowski, M.; Okumus, F. Exploring the role of next-generation virtual technologies in destination marketing. J. Destin. Market. Manag. 2018, 9, 138-148.
Tussyadiah, I.P.; Wang, D.; Jia, C.H. Virtual reality and attitudes toward tourism destinations. In Information and Communication Technologies in Tourism 2017; Springer: Cham, Switzerland, 2017; pp. 229-239.
Buhalis, D.; Law, R. Progress in information technology and tourism management: 20 years on and 10 years after the Internet-The state of eTourism research. Tour. Manag. 2008, 29, 609-623.
Wei, W.; Qi, R.; Zhang, L. Effects of virtual reality on theme park visitors' experience and behaviors: A presence perspective. Tour. Manag. 2019, 71, 282-293.
Wilkinson, C. Childhood, mobile technologies and everyday experiences: Changing technologies = changing childhoods? Child. Geogr. 2016, 14, 372-373.
Kharroubi, A.; Hajji, R.; Billen, R.; Poux, F. Classification and integration of massive 3d points clouds in a virtual reality (VR) environment. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2019, 42, 165-171.
Virtanen, J.P.; Daniel, S.; Turppa, T.; Zhu, L.; Julin, A.; Hyyppä, H.; Hyyppä, J. Interactive dense point clouds in a game engine. Isprs J. Photogramm. Remote Sen. 2020, 163, 375-389.
Poux, F. The Smart Point Cloud: Structuring 3D Intelligent Point Data. Ph.D. Thesis, Université de Liège, Liège, Belgique, 2019.
Berger, M.; Tagliasacchi, A.; Seversky, L.M.; Alliez, P.; Guennebaud, G.; Levine, J.A.; Sharf, A.; Silva, C.T. A Survey of Surface Reconstruction from Point Clouds. Comput. Graph. Forum 2017, 36, 301-329.
Poux, F.; Neuville, R.; Hallot, P.; Billen, R. Model for Semantically Rich Point Cloud Data. Isprs Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 2017, IV-4/W5, 107-115.
Schütz, M.; Krösl, K.; Wimmer, M. Real-Time Continuous Level of Detail Rendering of Point Clouds. In Proceedings of the IEEE VR 2019, the 26th IEEE Conference on Virtual Reality and 3D User Interfaces, Osaka, Japan, 23-27 March 2019; pp. 1-8.
Remondino, F. From point cloud to surface: The modeling and visualization problem. Isprs-Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2003, XXXIV, 24-28.
Koenderink, J.J.; van Doorn, A.J. Affine structure from motion. J. Opt. Soc. Am. A 1991, 8, 377.
Haala, N. The Landscape of Dense Image Matching Algorithms. J. Opt. Soc. Am. A 2013, 8, 377-385.
Quintero, M.S.; Blake, B.; Eppich, R. Conservation of Architectural Heritage: The Role of Digital Documentation Tools: The Need for Appropriate Teaching Material. Int. J. Arch. Comput. 2007, 5, 239-253.
Remondino, F. Heritage Recording and 3D Modeling with Photogrammetry and 3D Scanning. Remote Sens. 2011, 3, 1104-1138.
Noya, N.C.; Garcia, Á.L.; Ramirez, F.C.; García, Á.L.; Ramírez, F.C. Combining photogrammetry and photographic enhancement techniques for the recording of megalithic art in north-west Iberia. Digit. Appl. Archaeol. Cult. Herit. 2015, 2, 89-101.
Yastikli, N. Documentation of cultural heritage using digital photogrammetry and laser scanning. J. Cult. Herit. 2007, 8, 423-427.
Poux, F.; Neuville, R.; Van Wersch, L.; Nys, G.-A.; Billen, R. 3D Point Clouds in Archaeology: Advances in Acquisition, Processing and Knowledge Integration Applied to Quasi-Planar Objects. Geosciences 2017, 7, 96.
Tabkha, A.; Hajji, R.; Billen, R.; Poux, F. Semantic enrichment of point cloud by automatic extraction and enhancement of 360° panoramas. Isprs-Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2019, XLII-2/W17, 355-362.
Napolitano, R.K.; Scherer, G.; Glisic, B. Virtual tours and informational modeling for conservation of cultural heritage sites. J. Cult. Herit. 2018, 29, 123-129.
Rua, H.; Alvito, P. Living the past: 3D models, virtual reality and game engines as tools for supporting archaeology and the reconstruction of cultural heritage-the case-study of the Roman villa of Casal de Freiria. J. Archaeol. Sci. 2011, 38, 3296-3308.
Poux, F.; Billen, R. Voxel-based 3D point cloud semantic segmentation: Unsupervised geometric and relationship featuring vs. deep learning methods. Isprs Int. J. Geo-Inf. 2019, 8, 213.
Poux, F.; Neuville, R.; Billen, R. Point Cloud Classification Of Tesserae From Terrestrial Laser Data Combined With Dense Image Matching For Archaeological Information Extraction. Isprs Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 2017, IV-2/W2, 203-211.
Koutsoudis, A.; Vidmar, B.; Ioannakis, G.; Arnaoutoglou, F.; Pavlidis, G.; Chamzas, C. Multi-image 3D reconstruction data evaluation. J. Cult. Herit. 2014, 15, 73-79.
Mozas-Calvache, A.T.; Pérez-García, J.L.; Cardenal-Escarcena, F.J.; Mata-Castro, E.; Delgado-García, J. Method for photogrammetric surveying of archaeological sites with light aerial platforms. J. Archaeol. Sci. 2012, 39, 521-530.
Burens, A.; Grussenmeyer, P.; Guillemin, S.; Carozza, L.; Lévêque, F.; Mathé, V. Methodological Developments in 3D Scanning and Modelling of Archaeological French Heritage Site: The Bronze Age Painted Cave of "Les Fraux", Dordogne (France). Isprs-Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2013, XL-5/W2, 131-135.
Kai-Browne, A.; Kohlmeyer, K.; Gonnella, J.; Bremer, T.; Brandhorst, S.; Balda, F.; Plesch, S.; Lehmann, D. 3D Acquisition, Processing and Visualization of Archaeological Artifacts. In Proceedings of the EuroMed 2016: Digital Heritage. Progress in Cultural Heritage: Documentation, Preservation, and Protection, LImassol, Cyprus, 31 October-5 November 2016; Springer: Cham, Switzerland, 2016; pp. 397-408.
Armesto-González, J.; Riveiro-Rodríguez, B.; González-Aguilera, D.; Rivas-Brea, M.T. Terrestrial laser scanning intensity data applied to damage detection for historical buildings. J. Archaeol. Sci. 2010, 37, 3037-3047.
James, M.R.; Quinton, J.N. Ultra-rapid topographic surveying for complex environments: The hand-held mobile laser scanner (HMLS). Earth Surf. Process. Landf. 2014, 39, 138-142.
Lauterbach, H.; Borrmann, D.; Heß, R.; Eck, D.; Schilling, K.; Nüchter, A. Evaluation of a Backpack-Mounted 3D Mobile Scanning System. Remote Sens. 2015, 7, 13753-13781.
Slama, C.; Theurer, C.; Henriksen, S. Manual of Photogrammetry; Slama, C., Theurer, C., Henriksen, S.W., Eds.; Taylor & Francis Group/CRC Press: Boca Raton, FL, USA, 1980; ISBN 0-937294-01-2.
Weinmann, M. Reconstruction and Analysis of 3D Scenes; Springer International Publishing: Cham, Switzerland, 2016; ISBN 978-3-319-29244-1.
Remondino, F.; Spera, M.G.; Nocerino, E.; Menna, F.; Nex, F. State of the art in high density image matching. Photogramm. Rec. 2014, 29, 144-166.
Pix4D Terrestrial 3D Mapping Using Fisheye and Perspective Sensors-Support. Available online: https://support.pix4d.com/hc/en-us/articles/204220385-Scientific-White-Paper-Terrestrial-3DMapping-Using-Fisheye-and-Perspective-Sensors#gsc.tab=0 (accessed on 18 April 2016).
Bentley 3D City Geographic Information System-A Major Step Towards Sustainable Infrastructure. Available online: https://www.bentley.com/en/products/product-line/reality-modeling-software/contextcapture (accessed on 12 February 2020).
Agisoft LLC Agisoft Photoscan. Available online: http://www.agisoft.com/pdf/photoscan_presentation.pdf (accessed on 12 February 2020).
nFrames. Available online: http://nframes.com/ (accessed on 12 February 2020).
Changchang Wu VisualSFM. Available online: http://ccwu.me/vsfm/ (accessed on 12 February 2020).
Moulon, P. openMVG. Available online: http://imagine.enpc.fr/~moulonp/openMVG/ (accessed on 12 February 2020).
Autodesk. Autodesk: 123D catch. Available online: http://www.123dapp.com/catch (accessed on 12 February 2020).
PROFACTOR GmbH ReconstructMe. Available online: http://reconstructme.net/ (accessed on 12 February 2020).
Eos Systems Inc Photomodeler. Available online: http://www.photomodeler.com/index.html (accessed on 12 February 2020).
Capturing Reality s.r.o RealityCapture. Available online: https://www.capturingreality.com (accessed on 12 February 2020).
Bassier, M.; Vergauwen, M.; Poux, F. Point Cloud vs. Mesh Features for Building Interior Classification. Remote Sens. 2020, 12, 2224.
Gareth, J. The elements of user experience: User-centered design for the Web and beyond. Choice Rev. Online 2011, 49.
Gasson, S. Human-Centered Vs. User-Centered Approaches to Information System Design. J. Inf. Technol. Theory Appl. 2003, 5, 29-46.
Reunanen, M.; Díaz, L.; Horttana, T. A holistic user-centered approach to immersive digital cultural heritage installations: Case Vrouw Maria. J. Comput. Cult. Herit. 2015, 7, 1-16.
Barbieri, L.; Bruno, F.; Muzzupappa, M. User-centered design of a virtual reality exhibit for archaeological museums. Int. J. Interact. Des. Manuf. 2018, 12, 561-571.
Ibrahim, N.; Ali, N.M. A conceptual framework for designing virtual heritage environment for cultural learning. J. Comput. Cult. Herit. 2018, 11, 1-27.
Cipolla Ficarra, F.V.; Cipolla Ficarra, M. Multimedia, user-centered design and tourism: Simplicity, originality and universality. Stud. Comput. Intell. 2008, 142, 461-470.
Loizides, F.; El Kater, A.; Terlikas, C.; Lanitis, A.; Michael, D. Presenting Cypriot Cultural Heritage in Virtual Reality: A User Evaluation. In LNCS; Springer: Cham, Switzerland, 2014; Volume 8740, pp. 572-579.
Sukaviriya, N.; Sinha, V.; Ramachandra, T.; Mani, S.; Stolze, M. User-Centered Design and Business Process Modeling: Cross Road in Rapid Prototyping Tools. In Human-Computer Interaction-INTERACT 2007. INTERACT 2007. Lecture Notes in Computer Science; Baranauskas, C., Palanque, P., Abascal, J., Barbosa, S.D.J., Eds.; Springer: Berlin/Heidelberg, Germany, 2007; Volume 4662 LNCS, pp. 165-178.
Kinzie, M.B.; Cohn, W.F.; Julian, M.F.; Knaus, W.A. A user-centered model for Web site design: Needs assessment, user interface design, and rapid prototyping. J. Am. Med. Inform. Assoc. 2002, 9, 320-330.
Jung, T.; tom Dieck, M.C.; Lee, H.; Chung, N. Effects of virtual reality and augmented reality on visitor experiences in museum. In Information and Communication Technologies in Tourism 2016; Springer: Cham, Switzerland, 2016; pp. 621-635.
Treffer, M. Un Système d' Information Géographique 3D Pour le Château de Jehay; Définition des besoins et modélisation conceptuelle sur base de CityGML. Master's Thesis, Université de Liège, Liège, Belgique, 2017.
Besl, P.J.; McKay, N.D. A Method for Registration of 3-D Shapes. IEEE Trans. Pattern Anal. Mach. Intell. 1992, 14, 239-256.
AliceVision Meshroom-3D Reconstruction Software. Available online: https://alicevision.org/#meshroom (accessed on 30 July 2019).
IGN France MicMac. Available online: https://micmac.ensg.eu/index.php/Accueil (accessed on 30 July 2019).
Nirenstein, S.; Blake, E.; Gain, J. Exact From-Region Visibility Culling. In Proceedings of theThirteenth Eurographics Workshop on Rendering, Pisa, Italy, 26-28 June 2002.
LaViola, J.J., Jr. A discussion of cybersickness in virtual environments. ACM Sigchi Bull. 2000, 32, 47-56.
Templeman, J.N.; Denbrook, P.S.; Sibert, L.E. Virtual locomotion: Walking in place through virtual environments. Presence: Teleoper. Virtual Environ. 1999, 8, 598-617.
Tregillus, S.; Folmer, E. VR-STEP: Walking-in-Place using Inertial Sensing for Hands Free Navigation in Mobile VR Environments. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems-CHI '16, San Jose, CA, USA, 7-12 May 2016; pp. 1250-1255.
Mccauley, M.E.; Sharkey, T.J. Cybersickness: Perception of Self-Motion in Virtual Environment. Presence Teleoper. Virtual Environ. 1992, 1, 311-318.
Parger, M.; Mueller, J.H.; Schmalstieg, D.; Steinberger, M. Human upper-body inverse kinematics for increased embodiment in consumer-grade virtual reality. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology, Tokyo, Japan, 28 November 2018; pp. 1-10.
Aristidou, A.; Lasenby, J. FABRIK: A fast, iterative solver for the Inverse Kinematics problem. Graph. Model. 2011, 73, 243-260.
Maciel, A.; De, S. Extending FABRIK with model constraints. Graph. Model. Wil. Online Libr. 2015, 73, 243-260.
Schyns, O. Locomotion and Interactions in Virtual Reality; Liège University: Liège, Belgique, 2018.
Poux, F.; Neuville, R.; Nys, G.-A.; Billen, R. 3D Point Cloud Semantic Modelling: Integrated Framework for Indoor Spaces and Furniture. Remote Sens. 2018, 10, 1412.
