Investigating Complex Sketch Recognition Strategies for Developing Future Design Tools

Baudoux, Gaëlle; Goucher-Lambert, Kosa

doi:10.1115/1.4069831

Download

Article (Scientific journals)

Investigating Complex Sketch Recognition Strategies for Developing Future Design Tools

Baudoux, Gaëlle; Goucher-Lambert, Kosa

2025 • In Journal of Mechanical Design, p. 1-28

Peer Reviewed verified by ORBi

Permalink
https://hdl.handle.net/2268/335975

DOI
10.1115/1.4069831

Files (1)Send to Details Statistics Bibliography Similar publications

Files

Full Text

MD-25-1023_Final_UnderstandingComplexSketch.docx.pdf

Author postprint (18.03 MB)

Download

All documents in ORBi are protected by a user license.

Send to

RIS BibTex APA Chicago Permalink X Linkedin

Details

Keywords :

artificial intelligence; computer-aided design; conceptual design; design representation; system design

Abstract :

[en] Despite recent advances in multi-modal AI technology, there remains a significant gap in their ability to be incorporated into complex design and engineering work. One such challenge relates to contexts where sketch-based inputs are desirable, due to the difficulty in recognizing freehand sketches or interpreting underlying human intent. To elucidate requirements for emerging sketch-based AI systems for complex design context, we consider an architectural design case-study. Using a Wizard of Oz experimental paradigm, we substitute the “tool” with human agents and conduct a lab-based study in which professional architectural designers complete a design brief using this “tool”. Here, participants execute functions such as recognizing freely produced design plans and perspective drawings for downstream applications (e.g., generating inspirational images or high quality renders). Observing the human agents performing the sketch recognition task, results demonstrate that agents not only rely on visible sketch elements (i.e., lines) and architectural drawing codes, but also on their memory of previous lines and their knowledge of the design brief to comprehend perceived lines. Agents gradually develop an understanding of the designed artifact, but also of the designer's intentions. These activities are crucial for the agent to obtain a functional model of the designed object, beyond a purely topological and geometric perception model. Insights about this human workflow bring new potential techniques of sketch recognition for design and engineering tasks, informing the inclusion of new resources within AI tools.

Disciplines :

Architecture
Computer science

Author, co-author :

Baudoux, Gaëlle ; Université de Liège - ULiège > Département ArGEnCo > Lucid - Lab for User Cognition & Innovative Design ; UCB - University of California Berkeley > Mechanical Engineering

Goucher-Lambert, Kosa; Dept of Mechanical Engineering, Etcheverry Hall RM 2120/ETCHEO6179 Berkeley, CA 94720

Language :

English

Title :

Investigating Complex Sketch Recognition Strategies for Developing Future Design Tools

Publication date :

16 September 2025

Journal title :

Journal of Mechanical Design

ISSN :

1050-0472

eISSN :

1528-9001

Publisher :

ASME International

Pages :

1-28

Peer reviewed :

Peer Reviewed verified by ORBi

Additional URL :

https://asmedigitalcollection.asme.org/mechanicaldesign/article-pdf/doi/10.1115/1.4069831/7537814/md-25-1023.pdf

Available on ORBi :

since 19 September 2025

Statistics

Number of views

61 (0 by ULiège)

Number of downloads

129 (0 by ULiège)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

Roberts, M., Allen, S., and Coley, D., 2020, "Life Cycle Assessment in the Building Design Process-A Systematic Literature Review, " Build. Environ., 185, p. 107274.
Leclercq, P., and Heylighen, A., 2002, "5, 8 Analogies Per Hour. A Designers View on Analogical Reasoning, " Paper Presented at the AID-02 Artificial Intelligence in Design, Cambridge, UK, July 15-18.
Goel, A. K., 1997, "Design, Analogy, and Creativity, " IEEE Expert, 12(3), pp. 62-70.
Linsey, J. S., 2007, "Design-By-Analogy and Representation in Innovative Engineering Concept Generation, " Ph.D. dissertation, The University of Texas, Austin, TX.
Sio, U. N., Kotovsky, K., and Cagan, J., 2015, "Fixation or Inspiration? A Meta-Analytic Review of the Role of Examples on Design Processes, " Des. Stud., 39, pp. 70-99.
Jiang, S., Hu, J., Wood, K. L., and Luo, J., 2022, "Data-Driven Design-By-Analogy: State-of-The-Art and Future Directions, " ASME J. Mech. Des., 144(2), p. 020801.
Elara, M. R., 2023, "Do Analogies and Analogical Distance Influence Ideation Outcomes in Engineering Design?" Applied Cognitive Science and Technology: Implications of Interactions Between Human Cognition and Technology, M. Sumitava, D. Varun, and N. Srinivasan, eds., Springer Nature, Singapore, pp. 211-228.
Ball, L. J., and Christensen, B. T., 2019, "Advancing an Understanding of Design Cognition and Design Metacognition: Progress and Prospects, " Des. Stud., 65, pp. 35-59.
Jaruga-Rozdolska, A., 2022, "Artificial Intelligence as Part of Future Practices in the Architects Work: MidJourney Generative Tool as Part of a Process of Creating an Architectural Form, " Architectus, 3(71), pp. 95-104.
Paananen, V., Oppenlaender, J., and Visuri, A., 2024, "Using Text-To-Image Generation for Architectural Design Ideation, " Int. J. Architect. Comput., 22(3), pp. 458-474.
Safin, S., Maitrallin, M., Fruchard, B., and Lecolinet, E., 2021, "Appropriation and Memorisation Processes of Gestural Shortcuts on Trackpad: Longitudinal Study of Users? Strategies and Impact of Visuo-Semantic aid, " Actes de la 32e Conférence Francophone sur LInteraction Humain-Machine, Metz, France, Apr. 13, pp. 1-12.
Baudoux, G., and Goucher-Lambert, K., 2024, "Automating Analogical Reasoning: A Wizard of Oz Study on the Benefits and Pitfalls of a Sketch-Based AI Image Generator for Design, " Proceedings of DCC International Conference, Montreal, Canada, July 7-10, pp. 21-37.
Baudoux, G., 2024, "The Benefits and Challenges of AI Image Generators for Architectural Ideation: Study of an Alternative Human-Machine Co-Creation Exchange Based on Sketch Recognition, " Int. J. Architect. Comput., 22(2), pp. 201-215.
Egan, P., and Cagan, J., 2016, "Human and Computational Approaches for Design Problem-Solving, " Experimental Design Research: Approaches, Perspectives, Applications, Springer Nature, Switzerland, pp. 187-205.
Nguyen, T., Nguyen, N. D., and Nahavandi, S., 2019, "Multi-Agent Deep Reinforcement Learning With Human Strategies, " 2019 IEEE International Conference on Industrial Technology (ICIT), Melbourne, Australia, February, pp. 1357-1362.
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., et al., 2021, "The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews., " Br. Med. J., 372, p. n71.
Sutherland, I. E., 1963, "Sketchpad: A Man-Machine Graphical Communication System, " Proceedings of the May 21-23, 1963, Spring Joint Computer Conference (AFIPS 63 (Spring)), New York, May 21-23, pp. 329-346.
Stahovich, T. F., 1996, "SketchIT: A Sketch Interpretation Tool for Conceptual Mechanical Design, " Ph.D. dissertation, Massachusetts Institute of Technology, Cambridge, MA.
Davis, R., 2002, "Sketch Understanding in Design: Overview of Work at the MIT AI Lab, " Sketch Understanding, Papers From the 2002 AAAI Spring Symposium, Palo Alto, CA, Mar. 25, pp. 24-31.
Notowidigdo, M. J., 2004, "User-Directed Sketch Interpretation, " Doctoral dissertation, Massachusetts Institute of Technology, Cambridge, MA.
Ding, C., and Liu, L., 2016, "A Survey of Sketch Based Modeling Systems, " Front. Comput. Sci., 10(6), pp. 985-999.
Seff, A., Zhou, W., Richardson, N., and Adams, R. P., 2022, "Vitruvion: A Generative Model of Parametric CAD Sketches."
Zhang, Z., and Jin, Y., 2021, "Toward Computer Aided Visual Analogy Support (CAVAS): Augment Designers Through Deep Learning, " Proceedings of the ASME 2021 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Virtual, Online, Aug. 17-19, p. V006T06A057.
Arora, R., Darolia, I., Namboodiri, V. P., Singh, K., and Bousseau, A., 2017, "Sketchsoup: Exploratory Ideation Using Design Sketches, " Comput. Graphics Forum, 36(8), pp. 302-312.
Kazi, R. H., Grossman, T., Cheong, H., Hashemi, A., and Fitzmaurice, G., 2017, "DreamSketch: Early Stage 3D Design Explorations With Sketching and Generative Design, " Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, New York, Oct. 20, pp. 401-414.
Para, W., Bhat, S., Guerrero, P., Kelly, T., Mitra, N., Guibas, L. J., and Wonka, P., 2021, "Sketchgen: Generating Constrained CAD Sketches, " Adv. Neural Inf. Process. Syst., 34, pp. 5077-5088.
Wang, J., Lin, J., Yu, Q., Liu, R., Chen, Y., and Yu, S. X., 2022, "3d Shape Reconstruction From Free-Hand Sketches, " Proceedings of European Conference on Computer Vision, Tel Aviv, Israel, Oct. 23, pp. 184-202.
Zhang, S. H., Guo, Y. C., and Gu, Q. W., 2021, "Sketch2model: View-Aware 3D Modeling From Single Free-Hand Sketches, " Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Montreal, Canada, Oct. 10-17, pp. 6012-6021.
Zhang, Z., and Jin, Y., 2022, "Data-Enabled Sketch Search and Retrieval for Visual Design Stimuli Generation, " Artif. Intell. Eng. Des. Anal. Manuf., 36(25), pp. 1-18.
Kim, J., and Maher, M. L., 2023, "The Effect of AI-Based Inspiration on Human Design Ideation, " Int. J. Des. Creat. Innov., 11(2), pp. 81-98.
Jiang, S., Luo, J., Ruiz-Pava, G., Hu, J., and Magee, C. L., 2021, "Deriving Design Feature Vectors for Patent Images Using Convolutional Neural Networks, " ASME J. Mech. Des., 143(6), p. 061405.
Kwon, E., Pehlken, A., Thoben, K. D., Bazylak, A., and Shu, L. H., 2019, "Visual Similarity to Aid Alternative-Use Concept Generation for Retired Wind-Turbine Blades, " ASME J. Mech. Des., 141(3), p. 031106.
Kwon, E., Huang, F., and Goucher-Lambert, K., 2022, "Enabling Multi-modal Search for Inspirational Design Stimuli Using Deep Learning, " Artif. Intell. Eng. Des. Anal. Manuf., 36(1), p. e22.
Zhang, X., Huang, Y., Zou, Q., Pei, Y., Zhang, R., and Wang, S., 2020, "A Hybrid Convolutional Neural Network for Sketch Recognition, " Pattern Recognit. Lett., 130, pp. 73-82.
Valveny, E., and Martí, E., 2000, "Deformable Template Matching Within a Bayesian Framework for Hand-Written Graphic Symbol Recognition, " International Workshop on Graphics Recognition, A. K. Chhabra, and D. Dori, eds., Springer, Berlin, Heidelberg, pp. 193-208.
Lee, S., Feng, D., Grimm, C., and Gooch, B., 2008, "A Sketch-Based User Interface for Reconstructing Architectural Drawings, " Comput. Graphics Forum, 27(1), pp. 81-90.
Johnson, G., Gross, M., Yi-Luen Do, E., and Hong, J., 2012, "Sketch It, Make It: Sketching Precise Drawings for Laser Cutting, " Proceedings of CHI-12 Extended Abstracts on Human Factors in Computing Systems, Austin, TX, May 5, pp. 1079-1082.
Hsiao, C.-P., 2015, "SolidSketch: "Toward Enactive Interactions for Semantic Model Creation, " Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition, Glasgow, UK, June 22, pp. 329-330.
Juchmes, R., Leclercq, P., and Azar, S., 2004, "A Multi-Agent System for Architectural Sketches Interpretation, " Proceedings of Eurographics Workshop on Sketch-Based Interfaces and Modeling, Grenoble, France, Aug. 30-31.
Demaret, J.-N., and Leclercq, P., 2012, "Génération Automatique Dun Modèle de Bâtiment à Partir Dun Croquis, " Actes du 5ème Séminaire de Conception Architecturale Numérique, Paris, France, June 21-23.
Wang, X., Chen, X., and Zha, Z., 2018, "Sketchpointnet: A Compact Network for Robust Sketch Recognition, " Proceedings of 25th IEEE International Conference on Image Processing, Athens, Greece, Oct. 7, pp. 2994-2998.
Liu, F., Deng, X., Lai, Y. K., Liu, Y. J., Ma, C., and Wang, H., 2019, "Sketchgan: Joint Sketch Completion and Recognition With Generative Adversarial Network, " Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, June 16-20, pp. 5830-5839.
Gross, M., and Yi-Luen Do, E., 1996, "The Electronic Cocktail Napkin, " Des. Stud., 17(1), pp. 53-69.
Davis, N., Hsiao, C.-P., Yashraj Singh, K., and Magerko, B., 2016, "Co-Creative Drawing Agent With Object Recognition, " Proceedings of Twelfth Artificial Intelligence and Interactive Digital Entertainment Conference, Burlingame, CA, Oct. 8-12, pp. 9-15.
Liapis, A., Yannakakis, G., and Togelius, J., 2013, "Sentient Sketchbook: Computer-Aided Game Level Authoring, " Proceedings of FDG, Chania, Greece, May 14-17, pp. 213-220.
Lucas, P., and Martinho, C., 2017, "Stay A While and Listen to 3Buddy, A Co-Creative Level Design Support Tool, " Proceedings of ICCC, Atlanta, GA, June 17, pp. 205-212.
Steinfeld, K., 2020, "Sketch2pix: Machine-Augmented Architectural Sketching, " Proceedings of Computational Design and Robotic Fabrication Conference, Shanghai, China, July 5-6.
Karimi, P., Rezwana, J., Siddiqui, S., Maher, M. L., and Dehbozorgi, N., 2020, "Creative Sketching Partner: An Analysis of Human-AI Co-Creativity, " Proceedings of the 25th International Conference on Intelligent User Interfaces, Cagliari, Italy, Mar. 17.
Brie, P., Burny, N., Sluters, A., and Vanderdonckt, J., 2023, "Evaluating a Large Language Model on Searching for GUI Layouts, " Proc. ACM Hum. Comput. Interact., 7(EICS), pp. 1-37.
Safin, S., Delfosse, V., Elsen, C., and Leclercq, P., 2008, "Distributed Collaborative Design Studio: A Sketch-Based Environment to Support Rich Distant Collaboration, " Design Computing and Cognition Conference: Workshop "IT in Design", Atlanta, GA, June 21-22.
Rietz, F., Sutherland, A., Bensch, S., Wermter, S., and Hellström, T., 2021, "WoZ4U: An Open-Source Wizard-of-Oz Interface for Easy, Efficient and Robust HRI Experiments, " Front. Robot. AI, 8, p. 668057.
Baudoux, G., and Safin, S., 2025, "Study of Computer Multi-instrumented Reflexive Conversation Activity in Preliminary Architectural Design, " Int. J. Architect. Comput., pp. 1-21.
Baudoux, G., and Gronier, G., 2025, "Leveraging the Wizard of Oz Method for Stimulating Design: Benefits and Challenges From an Application Case Study, " Proceedings of IDETC, Anaheim, CA, Aug. 17-20.
Wasson, C., 2000, "Ethnography in the Field of Design, " Hum. Organ., 59(4), pp. 377-388.
Lejeune, C., 2019, Manuel Danalyse Qualitative, De Boeck Supérieur, Belgium.
Cole, T., and Gillies, M., 2022, "More Than a Bit of Coding:(Un-) Grounded (Non-) Theory in HCI, " Proceedings of CHI Conference on Human Factors in Computing Systems Extended Abstracts, New Orleans, LA, Apr. 27.
Ching, F. D., 2023, Architectural Graphics, John Wiley & Sons, Hoboken, NJ.
Hedges, K. E., 2017, Architectural Graphic Standards, John Wiley & Sons, Hoboken, NJ.
Gaudilliere, N., 2022, "Automatiser Larchitecture Savoir-Faire et Calculabilité Dans la Pratique des Courants Computationnels en Architecture (1965-2020), " Ph.D. dissertation, Université de Paris Est, France.
Baudoux, G., and Goucher-Lambert, K., 2024, "Understanding Complex Sketch Recognition Strategies for Intelligent Sketch-Based Design Tools, " Proceedings of International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Washington, DC, Aug. 25-28, pp. 1-10.