[en] The multimodal behaviour of IVAs may convey different socio-affective dimensions, such as emotions, personality, or social capabilities. Several research works show that factors may impact the perception of the IVA's behaviour. This paper proposes a systematic review, based on the PRISMA method, to investigate how the multimodal behaviour of IVAs is perceived with respect to socio-affective dimensions. To compare the results of different research works, a socio-emotional framework is proposed, considering the dimensions commonly employed in the studies. The conducted analysis of a wide array of studies ensures a comprehensive and transparent review, providing guidelines on the design of socio-affective IVAs.
Disciplines :
Quantitative methods in economics & management Computer science Social & behavioral sciences, psychology: Multidisciplinary, general & others
Elisabeth André, Martin Klesen, Patrick Gebhard, Steve Allen, and Thomas Rist. 1999. Integrating Models of Personality and Emotions into Lifelike Characters. In Affective Interactions (IWAI 1999). Springer, 150-165. https://doi.org/10.1007/10720296_11
Christoph Bartneck, Dana Kulić, Elizabeth Croft, and Susana Zoghbi. 2009. Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. International Journal of Social Robotics 1, 1 (2009), 71-81. https://doi.org/10.1007/s12369- 008- 0001- 3
Nikolaus Bee, Colin Pollock, Elisabeth André, and Marilyn Walker. 2010. Bossy or wimpy: Expressing social dominance by combining gaze and linguistic behaviors. In 10th International Conference on Intelligent Virtual Agents (IVA 2010). Springer, 265-271. https://doi.org/10.1007/978-3-642-15892-6_28
Béatrice Biancardi, Chen Wang, Maurizio Mancini, Angelo Cafaro, Guillaume Chanel, and Catherine Pelachaud. 2019. A Computational Model for Managing Impressions of an Embodied Conversational Agent in Real-Time. In 8th International Conference on Affective Computing and Intelligent Interaction (Cambridge, UK) (ACII 2019). Institute of Electrical and Electronics Engineers Inc., 234 - 240. https://doi.org/10.1109/ACII.2019.8925495
Zoraida Callejas, Brian Ravenet, Magalie Ochs, and Catherine Pelachaud. 2014. A Computational model of Social Attitudes for a Virtual Recruiter. In 13th International Conference on Autonomous Agents and Multiagent Systems (Paris, France) (AAMAS 2014). International Foundation for Autonomous Agents and Multiagent Systems, 93 - 100.
Mathieu Chollet and Stefan Scherer. 2017. Perception of virtual audiences. IEEE Computer Graphics and Applications 37, 4 (2017), 50-59. https://doi.org/10.1109/MCG.2017.3271465
Ionut Damian, Brigit Endrass, Peter Huber, Nikolaus Bee, and Elisabeth André. 2011. Individualized agent interactions. In 4th International Conference on Motion in Games (MIG 2011). 15-26. https://doi.org/10.1007/978-3-642-25090-3_2
Nick Degens, Birgit Endrass, Gert Jan Hofstede, Adrie Beulens, and Elisabeth Andre. 2017. ‘What I see is not what you get’: why culture-specific behaviours for virtual characters should be user-tested across cultures. AI and Society 32, 1 (2017), 37-49. https://doi.org/10.1007/s00146-014-0567-2
Guillaume Demary, Jean-Claude Martin, Stéphane Dubourdieu, Stéphane Travers, and Virginie Demulier. 2019. How do Leaders Perceive Stress and Followership from Nonverbal Behaviors Displayed by Virtual Followers?. In 19th ACM International Conference on Intelligent Virtual Agents (Paris, France) (IVA 2019). Association for Computing Machinery, 56-61.
Soumia Dermouche and Catherine Pelachaud. 2018. Attitude modeling for virtual character based on temporal sequence mining: Extraction and evaluation. In 5th International Conference on Movement and Computing (Genoa, Italy) (MOCO 2018). Association for Computing Machinery. https://doi.org/10.1145/3212721.3212806
Soumia Dermouche and Catherine Pelachaud. 2022. Leveraging the Dynamics of Non-Verbal Behaviors for Social Attitude Modeling. IEEE Transactions on Affective Computing 13, 2 (2022), 1072-1085. https://doi.org/10.1109/TAFFC.2020.2989262
Tahirou Djara, Abdoul Matine Ousmane, and Antoine Vianou. 2019. Mood and personality influence on emotion. In 2nd International EAI Conference on Emerging Technologies for Developing Countries (Cotonou, Benin) (AFRICATEK 2018). Springer Verlag, 166-174. https://doi.org/10.1007/978-3-030-05198-3_15
Paul Ekman and Wallace V Friesen. 1978. Facial action coding system: Investigator’s guide. Consulting Psychologists Press.
Elodie Etienne, Anne-Lise Leclercq, Angélique Remacle, Laurence Dessart, and Michaël Schyns. 2023. Perception of avatars nonverbal behaviors in virtual reality. Psychology and Marketing 40, 11 (2023), 2464-2481. https://doi.org/10.1002/mar. 21871
Susan T. Fiske, Amy J.C. Cuddy, and Peter Glick. 2007. Universal dimensions of social cognition: warmth and competence. Trends in Cognitive Sciences 11, 2 (2007), 77-83. https://doi.org/10.1016/j.tics.2006.11.005
Sofia Fountoukidou, Uwe Matzat, Jaap Ham, and Cees Midden. 2019. Effects of a Virtual Model’s Pitch and Speech Rate on Affective and Cognitive Learning. In 14th International Conference on Persuasive Technology (Limassol, Cyprus) (PERSUASIVE 2019). Springer Verlag, 16-27. https://doi.org/10.1007/978-3-030-17287-9_2
Patrick Gebhard, Tanja Schneeberger, Gregor Mehlmann, Tobias Baur, and Elisabeth Andre. 2019. Designing the Impression of Social Agents’ Real-time Interruption Handling. In 19th ACM International Conference on Intelligent Virtual Agents (Paris, France) (IVA 2019). Association for Computing Machinery, 19-21. https://doi.org/10.1145/3308532.3329435
Yann Glemarec, Jean-Luc Lugrin, Anne-Gwenn Bosser, Aryana Collins Jackson, Cedric Buche, and Marc Erich Latoschik. 2021. Indifferent or Enthusiastic? Virtual Audiences Animation and Perception in Virtual Reality. Frontiers in Virtual Reality 2 (2021). https://doi.org/10.3389/frvir.2021.666232
Jonathan Gratch, Anna Okhmatovskaia, Francois Lamothe, Stacy Marsella, Mathieu Morales, Rick J. Van Der Werf, and Louis-Philippe Morency. 2006. Virtual rapport. In 6th International Conference on Intelligent Virtual Agents (IVA 2006). Springer Verlag, 14-27. https://doi.org/10.1007/11821830_2
Peter A. Hancock, Deborah R. Billings, Kristin E. Schaefer, Jessie Y. C. Chen, Ewart J. De Visser, and Raja Parasuraman. 2011. A Meta-Analysis of Factors Affecting Trust in Human-Robot Interaction. Human Factors 53, 5 (2011), 517-527. https://doi.org/10.1177/0018720811417254
Willem K. B. Hofstee, Boele de Raad, and Lewis R. Goldberg. 1992. Integration of the Big Five and Circumplex Approaches to Trait Structure. Journal of Personality and Social Psychology 63, 1 (1992), 146-163. https://doi.org/10.1037//0022- 3514. 63.1.146
Adineh Hosseinpanah and Nicole C. Krämer. 2021. Lost in Interpretation? The Role of Culture on Rating the Emotional Nonverbal Behaviors of a Virtual Agent. In Cross-Cultural Design. Applications in Cultural Heritage, Tourism, Autonomous Vehicles, and Intelligent Agents. Springer International Publishing, 350-368. https://doi.org/10.1007/978- 3- 030- 77080- 8_28
Adineh Hosseinpanah, Nicole C. Krämer, and Carolin Straßmann. 2018. Empathy for Everyone? The Effect of Age When Evaluating a Virtual Agent. In Proceedings of the 6th International Conference on Human-Agent Interaction (Southampton, United Kingdom) (HAI’18). Association for Computing Machinery, 184-190. https://doi.org/10.1145/3284432.3284442
Ni Kang, Willem-Paul Brinkman, M Birna van Riemsdijk, and Mark A Neerincx. 2013. An expressive virtual audiencewith flexible behavioral styles. IEEE Transactions on Affective Computing 4, 4 (2013), 326-340. https://doi.org/10.1109/TAFFC.2013.2297104
Sin-Hwa Kang, Jonathan Gratch, Candy Sidner, Ron Artstein, Lixing Huang, and Louis-Philippe Morency. 2012. Towards building a virtual counselor: Modeling nonverbal behavior during intimate self-disclosure. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems (Valencia, Spain) (AAMAS 2012, Vol. 1). International Foundation for Autonomous Agents and Multiagent Systems, 63-70.
Peter Kullmann, Timo Menzel, Mario Botsch, and Marc Erich Latoschik. 2023. An Evaluation of Other-Avatar Facial Animation Methods for Social VR. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHIEA’23). Association for Computing Machinery, Article 33, 7 pages. https://doi.org/10.1145/3544549.3585617
Yunna Kwan, Sungwon Choi, Tae Rim Eom, and Tae Hui Kim. 2021. Development of a structured interview to explore interpersonal schema of older adults living alone based on autobiographical memory. International Journal of Environmental Research and Public Health 18, 5 (2021), 2316. https://doi.org/10.3390/ijerph18052316
Nicole Lazzeri, Daniele Mazzei, Maher Ben Moussa, Nadia Magnenat-Thalmann, and Danilo De Rossi. 2018. The influence of dynamics and speech on understanding humanoid facial expressions. International Journal of Advanced Robotic Systems 15, 4 (2018). https://doi.org/10.1177/1729881418783158
Jina Lee, Zhiyang Wang, and Stacy Marsella. 2010. Evaluating models of speaker head nods for virtual agents. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems (Toronto, Canada) (AAMAS 2010, Vol. 1). International Foundation for Autonomous Agents and Multiagent Systems, 1257-1263.
Birgit Lugrin, Elisabeth Andre, Matthias Rehm, Afia Lipi, and Yukiko Nakano. 2011. Culture-related differences in aspects of behavior for virtual characters across Germany and Japan. In Proceedings of the 10th International Conference on Autonomous Agents and Multiagent Systems (Taipei, Taiwan) (AAMAS 2011, Vol. 1-3). Association for Computing Machinery, 441-448.
Birgit Lugrin, Julian Frommel, and Elisabeth André. 2018. Combining a Data-Driven and a Theory-Based Approach to Generate Culture-Dependent Behaviours for Virtual Characters. Vol. 134. Springer International Publishing, 111-142. https://doi.org/10.1007/978-3-319-67024-9_6
Audrey Marcoux, Marie-Hélène Tessier, and Philip L. Jackson. 2023. Nonverbal Markers of Empathy in Virtual Healthcare Professionals. In Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents (Würzburg, Germany) (IVA’23). Association for Computing Machinery, Article 43, 4 pages. https://doi.org/10.1145/3570945.3607291
Sébastien Mateo. 2020. A procedure for conduction of a successful literature review using the PRISMA method. Kinesitherapie 20, 226 (2020), 29-37. https://doi.org/10.1016/j.kine.2020.05.019
Robert R. McCrae and Oliver P John. 1992. An introduction to the five-factor model and its applications. Journal of Personality 60, 2 (1992), 175-215. https://doi.org/10.1111/j.1467-6494.1992.tb00970.x
Truong-Huy D. Nguyen, Elin Carstensdottir, Nhi Ngo, Magy Seif El-Nasr, Matt Gray, Derek Isaacowitz, and David Desteno. 2015. Modeling warmth and competence in virtual characters. In Proceedings of the 15th International Conference on Intelligent Virtual Agents (Delft, Netherlands) (IVA’15). 167-180. https://doi.org/10.1007/978-3-319-21996-7_18
Michael Nixon, Steve DiPaola, and Ulysses Bernardet. 2018. An Eye Gaze Model for Controlling the Display of Social Status in Believable Virtual Humans. In IEEE Conference on Computational Intelligence and Games (Maastricht, Netherlands) (CIG 2018). IEEE, 8 pages. https://doi.org/10.1109/CIG.2018.8490373
Magalie Ochs, Daniel Mestre, Grégoire De Montcheuil, Jean-Marie Pergandi, Jorane Saubesty, Evelyne Lombardo, Daniel Francon, and Philippe Blache. 2019. Training doctors’ social skills to break bad news: evaluation of the impact of virtual environment displays on the sense of presence. Journal on Multimodal User Interfaces 13 (2019), 41-51. https://doi.org/10.1007/s12193-018-0289-8
Magalie Ochs, Catherine Pelachaud, and Ken Prepin. 2013. SOCIAL STANCES BY VIRTUAL SMILES. In 14th International Workshop on Image Analysis for Multimedia Interactive Services (Paris, France) (WIAMIS 2013). IEEE, 1 - 4. https://doi.org/10.1109/WIAMIS.2013.6616144
Dhaval Parmar, Stefan Olafsson, Dina Utami, Prasanth Murali, and Timothy Bickmore. 2022. Designing empathic virtual agents: manipulating animation, voice, rendering, and empathy to create persuasive agents. Autonomous Agents and Multi-Agent Systems 36, Article 17 (2022). https://doi.org/10.1007/s10458-021-09539-1
Delphine Potdevin, Céline Clavel, and Nicolas Sabouret. 2021. Virtual intimacy in human-embodied conversational agent interactions: the influence of multimodality on its perception. Journal on Multimodal User Interfaces 15 (2021), 25-43. https://doi.org/10.1007/s12193-020-00337-9
Delphine Potdevin, Celine Clavel, and Nicolas Sabouret. 2021. A virtual tourist counselor expressing intimacy behaviors: A new perspective to create emotion in visitors and offer them a better user experience? International Journal of Human-Computer Studies 150 (2021), 102612. https://doi.org/10.1016/j.ijhcs.2021.102612
Ken Prepin, Magalie Ochs, and Catherine Pelachaud. 2013. Beyond backchannels: co-construction of dyadic stance by reciprocal reinforcement of smiles between virtual agents. In Proceedings of the Annual Meeting of the Cognitive Science Society. 1163-1168.
Astrid M. Rosenthal-von der Pütten, Carolin Straßmann, Ramin Yaghoubzadeh, Stefan Kopp, and Nicole C. Krämer. 2019. Dominant and submissive nonverbal behavior of virtual agents and its effects on evaluation and negotiation outcome in different age groups. Computers in Human Behavior 90 (2019), 397-409. https://doi.org/10.1016/j.chb.2018.08.047
James A. Russell and Albert Mehrabian. 1977. Evidence for a three-factor theory of emotions. Journal of Research in Personality 11, 3 (1977), 273-294. https://doi.org/10.1016/0092-6566(77)90037-X
Pejman Sajjadi, Laura Hoffmann, Philipp Cimiano, and Stefan Kopp. 2018. On the Effect of a Personality-Driven ECA on Perceived Social Presence and Game Experience in VR. In 10th International Conference on Virtual Worlds and Games for Serious Applications (VS-Games) (Würzburg, Germany). IEEE, Wurzburg, 1-8. https://doi.org/10.1109/VS-Games.2018.8493436
Sinan Sonlu, Ugur Gudukbay, and Funda Durupinar. 2021. A Conversational Agent Framework with Multi-modal Personality Expression. ACM Transactions on Graphics 40, 1, Article 7 (2021), 16 pages. https://doi.org/10.1145/3439795
Carolin Straßmann, Astrid M. Rosenthal-von der Pütten, Ramin Yaghoubzadeh, Raffael Kaminski, and Nicole C. Krämer. 2016. The effect of an intelligent virtual agent’s nonverbal behavior with regard to dominance and cooperativity. In Proceedings of the 16th International Conference on Intelligent Virtual Agents (Los Angeles, USA) (IVA’16). 15-28. https://doi.org/10.1007/978-3-319-47665-0_2
Astrid M. von der Pütten, Laura Hoffmann, Jennifer Klatt, and Nicole C. Krämer. 2011. Quid pro quo? Reciprocal self-disclosure and communicative accomodation towards a virtual interviewer. In Proceedings of the 10th International Conference on Intelligent Virtual Agents (Reykjavik, Iceland) (IVA’11). Springer-Verlag, 183-194. https://doi.org/10.1007/978- 3- 642- 23974- 8_20
Isaac Wang, Rodrigo Calvo, Heting Wang, and Jaime Ruiz. 2023. Stop Copying Me: Evaluating nonverbal mimicry in embodied motivational agents. In Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents (Würzburg, Germany) (IVA’23). Association for Computing Machinery, Article 49, 4 pages. https://doi.org/10.1145/3570945.3607322
Jane Webster and Richard T. Watson. 2002. Analyzing the Past to Prepare for the Future: Writing a Literature Review. MIS quarterly 26, 2 (2002), xiii-xxiii. http://www.jstor.org/stable/4132319
Zuhair Zafar, Ashita Ashok, and Karsten Berns. 2021. Personality Traits Assessment using P.A.D. Emotional Space in Human-robot Interaction. In Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (Vienna, Austria) (VISIGRAPP 2021, Vol. 2). SciTePress, 111-118. https://doi.org/10.5220/0010161801110118
Sahba Zojaji, Christopher Peters, and Catherine Pelachaud. 2020. Influence of virtual agent politeness behaviors on how users join small conversational groups. In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents (Virtual Event, Scotland, UK) (IVA’20). Association for Computing Machinery, Article 59, 8 pages. https://doi.org/10.1145/3383652.3423917
Sahba Zojaji, Adam Červeň, and Christopher Peters. 2023. Impact of Multimodal Communication on Persuasiveness and Perceived Politeness of Virtual Agents in Small Groups. In Proceedings of the 23th ACM International Conference on Intelligent Virtual Agents (Würzburg, Germany) (IVA’23). Association for Computing Machinery, Article 18, 8 pages. https://doi.org/10.1145/3570945.3607356
