Multimodal generative AI for conceptual design: Enabling text-based and sketch-based human-AI conversations

Baudoux, Gaëlle; Guo, Chenjun; Goucher-Lambert, Kosa

doi:10.1017/pds.2025.10264

Download

Paper published in a book (Scientific congresses and symposiums)

Multimodal generative AI for conceptual design: Enabling text-based and sketch-based human-AI conversations

Baudoux, Gaëlle; Guo, Chenjun; Goucher-Lambert, Kosa

2025 • In Proceedings of 25th International Conference on Engineering Design

Peer reviewed

Permalink
https://hdl.handle.net/2268/331246

DOI
10.1017/pds.2025.10264

Files (1)Send to Details Statistics Bibliography Similar publications

Files

Full Text

325_MultimodalGenAI_CamReady_withAbstract.docx.pdf

Author postprint (3.93 MB)

Download

All documents in ORBi are protected by a user license.

Send to

RIS BibTex APA Chicago Permalink X Linkedin

Details

Keywords :

Design creativity; Human-AI Collaboration; AI image generation; Ideation; Text-to-image; Sketch-to-image

Abstract :

[en] Creative design could benefit from recent advances in AI, by providing inspirational images generation. While researchers have studied general benefits and limitations of existing text-to-image tools, this work investigates how to overcome these limitations and enable non-disruptive multimodal prompting for project-appropriate human-AI conversation. This paper presents the development of a system for human-AI co-creativity that supports text-based and sketch-based image generation through the elaboration of a complex prompt structure. The developed interface supports dialog interaction and the visualization of the respective contributions. Furthermore, this system constitutes a means for further studying human-AI interaction and how it varies in regard to conversation modality, between text and sketch.

Disciplines :

Architecture
Computer science
Mechanical engineering

Author, co-author :

Baudoux, Gaëlle ; Université de Liège - ULiège > Département ArGEnCo > Lucid - Lab for User Cognition & Innovative Design ; UCB - University of California Berkeley > Mechanical engineering

Guo, Chenjun; UCB - University of California Berkeley

Goucher-Lambert, Kosa; UCB - University of California Berkeley > Mechanical engineering

Language :

English

Title :

Multimodal generative AI for conceptual design: Enabling text-based and sketch-based human-AI conversations

Publication date :

2025

Event name :

25th International Conference on Engineering Design

Event date :

11/08/2025

Audience :

International

Main work title :

Proceedings of 25th International Conference on Engineering Design

Publisher :

Design Society, United States

Peer review/Selection committee :

Peer reviewed

Available on ORBi :

since 29 April 2025

Statistics

Number of views

165 (3 by ULiège)

Number of downloads

38 (0 by ULiège)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenAlex citations

Bibliography

Baudoux, G., & Safin, S., (2025). Study of computer multi-instrumented reflexive conversation activity in preliminary architectural design. International Journal of Architectural Computing, 14780771241310207.
Beyan, E. V., P., & Rossy, A. G. C., (2023). A Review of AI Image Generator: Influences, Challenges, and Future Prospects for Architectural Field. Journal of Artificial Intelligence in Architecture, 2 (1), 53-65.
Casakin H, and Wodehouse A., A systematic review of design creativity in the architectural design studio. Buildings 2021; 11 (1): 31.
Cherry, E.C., Latulipe, C.,: Quantifying the creativity support of digital tools through the creativity support index. ACM Trans. Comput. Hum. Interact. 21, 1-25 (2014).
Enjellina, B. E. V. P., & Rossy, A. G. C., (2023). A review of AI image generator: Influences, challenges, and future prospects for architectural field. Journal of Artificial Intelligence in Architecture, 2 (1), 53-65.
Figoli, F. A., Rampino, L., & Mattioli, F., (2022). AI in design idea development: A workshop on creativity and human-AI collaboration. PROCEEDINGS OF DRS, 1-17.
Gao, J., (2024). From Sketch to Design: A Cross-scale Workflow for Procedural Generative Urban Design. Proceedings of the 29th CAADRIA Conference, Singapore, 20-26 April 2024, Volume 1, Pp. 343-352.
Hu, M., McComb, C., & Goucher-Lambert, K., (2023). Uncovering hidden patterns of design ideation using hidden Markov modeling and neuroimaging. Artificial Intelligence for Engineering Design, Analysis and Manufacturing 1-20.
Huang, J., Jing, L., Tan, Z., & Kwong, S., (2022). Multi-Density Sketch-to-Image Translation Network. IEEE Transactions on Multimedia, 24, 4002-4015.
Karimi, P., Rezwana, J., Siddiqui, S., Maher, M. L., and Dehbozorgi, N., (2020). Creative sketching partner: An analysis of human-AI co-creativity. In Proceedings of the 25th International Conference on Intelligent User Interfaces (pp. 221-230).
Kim, J., Maher, M. L., & Siddiqui, S., (2021). Collaborative Ideation Partner: Design Ideation in Human-AI Co-creativity. In CHIRA (pp. 123-130).
Kwon, E., Huang, F., & Goucher-Lambert, K., (2022). Enabling Multi-Modal Search for Inspirational Design Stimuli Using Deep Learning. Artif. Intell. Eng. Des. Anal. Manuf., 36 (1), p. e22.
Liapis, A.; Yannakakis, G. N.; Alexopoulos, C.; and Lopes, P., (2016). Can computers foster human users' creativity? Theory and praxis of mixed-initiative co-creativity. DCE. Accepted: 2018-04-23T12:31:38Z Publisher: DCE.
McComb, C., Boatwright, P., & Cagan, J., (2023). Focus and Modality: Defining a roadmap to future AI-Human teaming in design. Proceedings of the Design Society, 3, 1905-1914. doi: Https://doi.org/10.1017/pds.2023.191
Nagele, J., (2023). Fantasy on Demand: The Temptation Of Text-to-Image AI. CTBUH Journal, 2023 (3), 46-51.
Paananen, V., Oppenlaender, J., & Visuri, A., (2024). Using text-to-image generation for architectural design ideation. International Journal of Architectural Computing, 22 (3), 458-474.
Rezwana, J., (2023). Towards designing engaging and ethical human-centered AI partners for human-AI co-creativity (Doctoral dissertation, The University of North Carolina at Charlotte).
Seeber, I., Bittner, E., Briggs, R. O., de Vreede, T., de Vreede, G. J., Elkins, A.,⋯ Söllner, M., (2020). Machines as teammates: A research agenda on AI in team collaboration. Information and Management, 57 (2), 103174. https://doi.org/10.1016/j.im.2019.103174
Stoimenova, N., & Price, R., (2020). Exploring the Nuances of Designing (with/for) Artificial Intelligence. DesignIssues, 36 (4), 45-55.
Zhang, C., Wang, W., Pangaro, P., Martelaro, N., & Byrne, D., (2023). Generative Image AI Using Design Sketches as input: Opportunities and Challenges. 254-261.
Zhang, G., Raina, A., Cagan, J., & McComb, C., (2021). A cautionary tale about the impact of AI on human design teams. Design Studies, 72, 100990.
Zhou, H., Zhu, J., Mateas, M., & Wardrip-Fruin, N., (2024). The Eyes, the Hands and the Brain: What can Text-to-Image Models Offer for Game Design and Visual Creativity? ACM International Conference Proceeding Series.