Paper published in a book (Scientific congresses and symposiums)
Leveraging the Wizard of Oz Method for Stimulating Design: Benefits and Challenges From an Application Case Study
Baudoux, Gaëlle; Gronier, Guillaume
2025 • In Proceedings of ASME 2025 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference - 37th International Conference on Design Theory and Methodology (DTM)
Method; Simulation; Wizard of Oz; Testing and evaluation; Design Stimuli
Abstract :
[en] This paper presents a comprehensive understanding of the use of the Wizard of Oz (WOz) method for studying emerging de- sign tools. This method involves hidden human "wizards" simulating the functionalities of an undeveloped technology without the user’s knowledge, making it a valuable method for assessing technologies that are not yet mature or are too resource-demanding. Although WOz has been widely used in fields like human-robot interaction and autonomous vehicles, its application in design research is relatively novel. Through a case study aimed at stimulating design ideas generation, we applied the WOz method to simulate an AI design assistant based on designer’s sketches during 17 one-and-a-half-hour design sessions. We evaluated the success of this WOz simulation, and the results showed exemplary performance by the wizards in both time management and content creation, along with a strongly positive designer perception of the simulation, and a rich data collection to inform research analysis. The WOz method offers two major advantages: versatility across domains and tasks, and cost-effectiveness. However, it also presents challenges, including the workload on wizards and ethical concerns. We conclude with three key axes of recommendations for implementing WOz in design settings: training wizards for feasibility, clearly defining simulated functionalities and experimental protocols, and ensuring system credibility. These insights will support future design research anticipating techno- logical advancements to study emerging tools.
Baudoux, Gaëlle ; Université de Liège - ULiège > Département ArGEnCo > Lucid - Lab for User Cognition & Innovative Design
Gronier, Guillaume; LIST - Luxembourg Institute of Science and Technology
Language :
English
Title :
Leveraging the Wizard of Oz Method for Stimulating Design: Benefits and Challenges From an Application Case Study
Publication date :
2025
Event name :
ASME 2025 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference - 37th International Conference on Design Theory and Methodology (DTM)
Event date :
17/08/2025
Audience :
International
Main work title :
Proceedings of ASME 2025 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference - 37th International Conference on Design Theory and Methodology (DTM)
Dahlbäck, Nils, Jönsson, Arne and Ahrenberg, Lars. “Wizard of Oz studies: why and how.” Proceedings of the 1st international conference on Intelligent user interfaces: pp. 193–200. 1993.
Browne, Jacob T. “Wizard of oz prototyping for machine learning experiences.” Extended abstracts of the 2019 CHI conference on human factors in computing systems: pp. 1–6. 2019.
Rietz, Finn, Sutherland, Alexander, Bensch, Suna, Wermter, Stefan and Hellström, Thomas. “WoZ4U: an open-source wizard-of-oz interface for easy, efficient and robust HRI experiments.” Frontiers in Robotics and AI Vol. 8 (2021): p. 668057.
Alevêque, Guillaume. “Intelligence et artifice. Le Magicien d’Oz ou la simulation de l’interaction humain-machine.” Techniques & Culture. Revue semestrielle d’anthropologie des techniques (2019).
Dow, Steven, MacIntyre, Blair, Lee, Jaemin, Oezbek, Christopher, Bolter, Jay David and Gandy, Maribeth. “Wizard of Oz support throughout an iterative design process.” IEEE Pervasive Computing Vol. 4 No. 4 (2005): pp. 18–26.
Taib, Ronnie and Ruiz, Natalie. “Wizard of Oz for multimodal interfaces design: Deployment considerations.” Human-Computer Interaction. Interaction Design and Usability: 12th International Conference, HCI International 2007, Beijing, China, July 22-27, 2007, Proceedings, Part I 12: pp. 232–241. 2007. Springer.
Bradley, Jay, Benyon, David, Mival, Oli and Webb, Nick. “Wizard of Oz experiments and companion dialogues.” Proceedings of HCI 2010. 2010. BCS Learning & Development.
Weschke, Jan, Bahamonde-Birke, Francisco J, Gade, Kay and Kazagli, Evanthia. “Asking the Wizard-of-Oz: How experiencing autonomous buses affects preferences towards their use for feeder trips in public transport.” Transportation research part C: emerging technologies Vol. 133 (2021): p. 103454.
Ranjbar, Parivash, Krishnakumari, Pournami Krishnan, Andersson, Jonas and Klingegård, Maria. “Vibrotactile guidance for trips with autonomous vehicles for persons with blindness, deafblindness, and deafness.” Transportation research interdisciplinary perspectives Vol. 15 (2022): p. 100630.
Vithanage, Shashindi, Dey, Arindam, Korte, Jessica, of Electrical, Institute and Electronics Engineers, issuing body., author. ‘Auslan Alexa’: A case study of VR Wizard of Oz prototyping for requirements elicitation with Deaf participants. 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), IEEE, Piscataway, NJ: (2023-3).
Helgert, Andre, Straßmann, Carolin and Eimler, Sabrina C. “Unlocking Potentials of Virtual Reality as a Research Tool in Human-Robot Interaction: A Wizard-of-Oz Approach.” Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction: pp. 535–539. 2024.
Sabeti, Sepehr, Ardecani, Fatemeh Banani and Shoghli, Omidreza. “Augmented reality safety warnings in roadway work zones: Evaluating the effect of modality on worker reaction times.” Transportation Research Part C: Emerging Technologies Vol. 169 (2024): p. 104867.
Rodríguez-Domínguez, María Trinidad, Bazago-Dómine, María Isabel, Jiménez-Palomares, María, Pérez-González, Gerardo, Núñez, Pedro, Santano-Mogena, Esperanza and Garrido-Ardila, Elisa María. “Interaction Assessment of a Social-Care Robot in Day center Patients with Mild to Moderate Cognitive Impairment: A Pilot Study.” International Journal of Social Robotics Vol. 16 No. 3 (2024): pp. 513–528.
Sienkiewicz, Barbara, Sejnova, Gabriela, Gajewski, Paul, Vavrecka, Michal and Indurkhya, Bipin. “How language of interaction affects the user perception of a robot.” International Conference on Social Robotics: pp. 308–321. 2023. Springer.
Jang, Hyeji, Park, Daehee and Bae, Sowoon. “Intelligent Moving Display Robots: Human-Centric Design Guidelines for Human–Robot Interaction Designers.” InternationalJournalofHuman–ComputerInteraction(2024): pp. 1–15.
Vrins, Anita, Pruss, Ethel, Ceccato, Caterina, Prinsen, Jos, De Rooij, Alwin, Alimardani, Maryam and De Wit, Jan. “Wizard-of-Oz vs. GPT-4: a comparative study of perceived social intelligence in HRI brainstorming.” Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction: pp. 1090–1094. 2024.
Aljuneidi, Saja, Heuten, Wilko, Tepe, Markus and Boll, Susanne. “Did that AI just charge me a fine? Citizens’ perceptions of AI-based discretion in public administration.” Proceedings of the 2023 ACM Conference on Information Technology for Social Good: pp. 57–67. 2023.
Page, Matthew J, Moher, David, Bossuyt, Patrick M, Boutron, Isabelle, Hoffmann, Tammy C, Mulrow, Cynthia D, Shamseer, Larissa, Tetzlaff, Jennifer M, Akl, Elie A, Brennan, Sue E et al. “PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews.” bmj Vol. 372 (2021).
Page, Matthew J, McKenzie, Joanne E, Bossuyt, Patrick M, Boutron, Isabelle, Hoffmann, Tammy C, Mulrow, Cynthia D, Shamseer, Larissa, Tetzlaff, Jennifer M, Akl, Elie A, Brennan, Sue E et al. “The PRISMA 2020 statement: an updated guideline for reporting systematic reviews.” bmj Vol. 372 (2021).
Kahn, Peter H, Kanda, Takayuki, Ishiguro, Hiroshi, Gill, Brian T, Shen, Solace, Ruckert, Jolina H and Gary, Heather E. “Human creativity can be facilitated through interacting with a social robot.” 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI): pp. 173–180. 2016. IEEE.
Xu, Jingwen, Chao, Chi-Ju and Fu, Zhiyong. “Research on intelligent design tools to stimulate creative thinking.” International Conference on Human-Computer Interaction: pp. 661–672. 2020. Springer.
Fu, Zhiyong and Zhou, Yuyao. “Research on human–AI co-creation based on reflective design practice.” CCF Transactions on Pervasive Computing and Interaction Vol. 2 (2020): pp. 33–41.
Kim, Taesu, Shunayeva, Aigerim, Lee, Gyunpyo and Suk, Hyeon-Jeong. “Sketching in-vehicle ambient lighting in virtual reality with the Wizard-of-Oz method.” Digital Creativity Vol. 33 No. 1 (2022): pp. 49–63.
Poser, Mathis, Küstermann, Gerrit C, Tavanapour, Navid and Bittner, Eva AC. “Design and evaluation of a conversational agent for facilitating idea generation in organizational innovation processes.” Information Systems Frontiers Vol. 24 No. 3 (2022): pp. 771–796.
Mollo, Vanina and Falzon, Pierre. “Auto-and alloconfrontation as tools for reflective activities.” Applied ergonomics Vol. 35 No. 6 (2004): pp. 531–540.
Lejeune, Christophe. Manuel d’analyse qualitative. De Boeck Supérieur (2019).
Baudoux, Gaëlle and Goucher-Lambert, Kosa. “Understanding Complex Sketch Recognition Strategies for Intelligent Sketch-Based Design Tools.” International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Vol. 88407: p. V006T06A017. 2024. American Society of Mechanical Engineers.
Baudoux, Gaëlle and Goucher-Lambert, Kosa. “Automating Analogical Reasoning: A Wizard of Oz study on the benefits and pitfalls of a sketch-based AI image generator for design.” International Conference on-Design Computing and Cognition: pp. 21–37. 2024. Springer.
Baudoux, Gaëlle and Safin, Stéphane. “Study of computer multi-instrumented reflexive conversation activity in preliminary architectural design.” International Journal of Architectural Computing (2025): p. 14780771241310207.
Baudoux, Gaëlle. “The benefits and challenges of artificial intelligence image generators for architectural ideation: Study of an alternative human-machine co-creation exchange based on sketch recognition.” International Journal of Architectural Computing Vol. 22 No. 2 (2024): pp. 201–215.
Robert, Jean-Marc and Brangier, Eric. “Prospective ergonomics for the design of future things.” Ergonomics (2024): pp. 1–18.
Bernsen, Niels Ole, Dybkjær, Hans and Dybkjær, Laila. “Wizard of oz prototyping: How and when.” Proc. CCI Working Papers Cognit. Sci./HCI, Roskilde, Denmark (1994): p. 67.
Fraser, Norman M and Gilbert, G Nigel. “Simulating speech systems.” Computer Speech & Language Vol. 5 No. 1 (1991): pp. 81–99.
