[en] [en] BACKGROUND: Teaching and learning debriefing and feedback skills-especially to a level of mastery-is challenging without an agreed-upon standard. There are a number of rating scales and rubrics to identify and evaluate debriefing and feedback skills that focus on an entire feedback or debriefing conversation. However, there is no rubric to assess and provide feedback on one of these conversations' most widely used microskills, the Advocacy-Inquiry technique. This study aimed to develop and preliminarily test the Advocacy-Inquiry Rubric (AIR)-a tool designed to support the teaching, coaching, and assessment of Advocacy-Inquiry, a widely used yet challenging debriefing microskill-through an international expert consensus process.
METHOD: Using a four-round Delphi process, we achieved expert consensus on the behavioral markers of effective and ineffective Advocacy-Inquiry techniques. Thirty-nine experts from 13 countries identified and refined a set of key behavioral anchors for each of Advocacy-Inquiry's five elements: Preview, Observation, Point of View, Inquiry, and Listen. These descriptors were embedded first in a seven-point numeric Behaviorally Anchored Rating Scale, then in a three-point emoji-based version, and finally in a teaching and learning version. The AIR underwent two rounds of usability testing and inter-rater testing of the emoji version. Using an interpretation-use argument approach, evidence was collected for AIR's validity across scoring, generalization, extrapolation, and implication.
RESULTS: The Delphi process established descriptors for each element of Advocacy-Inquiry, categorized by proficiency level (beginner to advanced). Usability testing enhanced the AIR's graphic layout to support both numeric ratings and formative feedback. The AIR was adapted into three tailored versions: a numeric AIR for detailed evaluation and progress tracking, an emoji AIR for peer assessment, and a teaching and learning AIR. Evidence for validity was assessed, highlighting both strengths and gaps.
CONCLUSION: AIR is an empirical rubric based on expert-derived criteria to support teaching, coaching, and assessing Advocacy-Inquiry microskills. The AIR offers a structured framework for self-, peer-, and mentor-led feedback and assessment to enhance a core skill of facilitators. By anchoring assessments in clear behavioral descriptors, the AIR aims to improve the quality of feedback and debriefing conversations. Future work should focus on rater training, reliability testing, and exploring the AIR's impact on real-world outcomes.
Disciplines :
Education & instruction
Author, co-author :
Buleon, Clément ; Université de Liège - ULiège > Département des sciences de la santé publique > Médecine d'urgence ; Center for Medical Simulation, Boston, MA, USA. clement.buleon@unicaen.fr
Szyld, Demian; Center for Medical Simulation, Boston, MA, USA ; Department of Emergency Medicine, Boston University Chobanian and Avedisian School of Medicine, Boston, MA, USA
Simon, Robert ; Université de Liège - ULiège > Département des sciences de la santé publique > Médecine d'urgence ; Center for Medical Simulation, Boston, MA, USA
Setnik, Lon; Center for Medical Simulation, Boston, MA, USA
Eppich, Walter J; Faculty of Medicine, Dentistry and Health Sciences, University of Melbourne, Melbourne, Australia
Fey, Mary; Center for Medical Simulation, Boston, MA, USA ; Department of Anesthesia, Critical Care and Pain Medicine, Beth Israel Deaconess Hospital, Boston, MA, USA
Lipshaw, James A; Center for Medical Simulation, Boston, MA, USA
Palaganas, Janice C; Massachusetts General Hospital Institute for Health Professions, Boston, MA, USA
Rudolph, Jenny W; Center for Medical Simulation, Boston, MA, USA ; Department of Anesthesia, Critical Care and Pain Medicine, Beth Israel Deaconess Hospital, Boston, MA, USA ; Harvard Medical School, Boston, MA, USA
Advocacy Inquiry Interest Group
Language :
English
Title :
The Advocacy-Inquiry Rubric (AIR): a standard to build debriefing and feedback skills.
Alternative titles :
[fr] La rubrique Advocacy-Inquiry (AIR) : une norme pour développer les compétences en matière de débriefing et de feedback.
Arora S et al. Objective structured assessment of debriefing: bringing science to the art of debriefing in surgery Ann Surg 2012 256 6 982 988 10.1097/SLA.0b013e3182610c91 22895396
Brett-Fleegler M et al. Debriefing assessment for simulation in healthcare: development and psychometric properties Simul Healthc 2012 7 5 288 294 10.1097/SIH.0b013e3182620228 22902606
Seelandt JC Grande B Kriech S Kolbe M DE-CODE: a coding scheme for assessing debriefing interactions BMJ Simul Technol Enhanc Learn 2018 4 no. 2 51 58 10.1136/bmjstel-2017-000233 35515884 8990183
Góes FDSND Jackman D Development of an instructor guide tool: ‘Three Stages of Holistic Debriefing’ Rev Lat Am Enfermagem 2020 28 10.1590/1518-8345.3089.3229 32022149 7000188 e3229
Sargeant J et al. Facilitated reflective performance feedback: developing an evidence- and theory-based model that builds relationship, explores reactions and content, and coaches for performance change (R2C2) Acad Med 2015 90 12 1698 1706 10.1097/ACM.0000000000000809 26200584
Gauthier G St-Onge C Tavares W Rater cognition: review and integration of research findings Med Educ 2016 50 5 511 522 10.1111/medu.12973 27072440
Rudolph JW Simon R Dufresne RL Raemer DB There’s no such thing as ‘nonjudgmental’ debriefing: a theory and method for debriefing with good judgment Simul Healthc J Soc Simul Healthc 2006 1 1 49 55 10.1097/01266021-200600110-00006
Decker S et al. Standards of best practice: simulation standard VI: the debriefing process Clin Simul Nurs 2013 9 6 S26 S29 10.1016/j.ecns.2013.04.008
Eppich W Cheng A Promoting excellence and reflective learning in simulation (PEARLS): development and rationale for a blended approach to health care simulation debriefing Simul Healthc 2015 10 2 106 115 10.1097/SIH.0000000000000072 25710312
Gross Forneris S Fey MK Critical conversations: the NLN guide for teaching thinking Nurs Educ Perspect 2016 37 5 248 249 10.1097/01.NEP.0000000000000069 27740554
Palaganas JC Fey M Simon R Structured debriefing in simulation-based education AACN Adv Crit Care 2016 27 no. 1 78 85 10.4037/aacnacc2016328 26909457
Clark CM Fey MK Fostering civility in learning conversations: introducing the PAAIL communication strategy Nurse Educ 2020 45 3 139 143 10.1097/NNE.0000000000000731 31498196
C. Argyris, R. Putnam, and D. M. Smith, Action Science: Concepts, Methods and Skills for Research and Intervention., First. San Francisco: Jossey-Bass, 1985.
Morse KJ Structured model of debriefing on perspective transformation for NP students Clin Simul Nurs 2015 11 3 172 179 10.1016/j.ecns.2015.01.001
Fey MK Roussin CJ Rudolph JW Morse KJ Palaganas JC Szyld D Teaching, coaching, or debriefing With Good Judgment: a roadmap for implementing ‘With Good Judgment’ across the SimZones Adv Simul 2022 7 1 10.1186/s41077-022-00235-y 39
Fraser KL Meguerdichian MJ Haws JT Grant VJ Bajaj K Cheng A Cognitive load theory for debriefing simulations: implications for faculty development Adv Simul 2018 3 1 28 10.1186/s41077-018-0086-1
Edmondson AC Bohmer RM Pisano GP Disrupted routines: team learning and new technology implementation in hospitals Adm Sci Q 2001 46 no. 4 685 716 10.2307/3094828
Nembhard IM Edmondson AC Making it safe: the effects of leader inclusiveness and professional status on psychological safety and improvement efforts in health care teams J Organ Behav 2006 27 7 941 966 10.1002/job.413
Rock LK Morse KJ Eppich W Rudolph JW Transforming team culture Chest 2023 163 6 1448 1457 10.1016/j.chest.2022.12.046 36642367
W. R. Torbert and Associates, “Action Inquiry as a manner of speaking,” in Action Inquiry: The Secret of Timely and Transforming Leadership., San Francisco: Berret-Koehler, 2004, p. 28.
Loo ME Krishnasamy C Lim WS Considering face, rights, and goals: a critical review of rapport management in facilitator-guided simulation debriefing approaches Simul Healthc J Soc Simul Healthc 2018 13 1 52 60 10.1097/SIH.0000000000000258
van Merriënboer JJG Sweller J Cognitive load theory in health professional education: design principles and strategies Med Educ 2010 44 1 85 93 10.1111/j.1365-2923.2009.03498.x 20078759
Young JQ Van Merrienboer J Durning S Ten Cate O Cognitive load theory: implications for medical education: AMEE guide no. 86 Med Teach 2014 36 5 371 384 10.3109/0142159X.2014.889290 24593808
Senge PM The Leader’s new work: Building learning organizations Sloan Manage Rev 1990 32 1 7 23
Gattrell WT et al. ACCORD (ACcurate COnsensus Reporting Document): a reporting guideline for consensus methods in biomedicine developed via a modified Delphi PLoS Med 2024 21 1 10.1371/journal.pmed.1004326 38261576 10805282 e1004326
S. Debnath, B. B. Lee, and S. Tandon, “Fifty Years and Going Strong: What Makes Behaviorally Anchored Rating Scales So Perennial as an Appraisal Method?,” Int. J. Bus. Soc. Sci., vol. 6, no. 2, Accessed: Aug. 21, 2024. Available: https://www.semanticscholar.org/paper/Fifty-Years-and-Going-Strong%3A-What-Makes-Anchored-Debnath-Lee/9c381f049faa17915d904472fa09f67ed6b2ec46
Schmutz J Eppich WJ Hoffmann F Heimberg E Manser T Five steps to develop checklists for evaluating clinical performance: an integrative approach Acad Med 2014 89 7 996 1005 10.1097/ACM.0000000000000289 24826862
Blissett S Rodriguez S Qasim A O’Sullivan P Beyond the task: developing a tool to measure workplace characteristics that affect cognitive load and learning Acad Med 2024 99 10 1132 1139 10.1097/ACM.0000000000005763
Humphrey-Murto S et al. The use of the delphi and other consensus group methods in medical education research: a review Acad Med 2017 92 10 1491 1498 10.1097/ACM.0000000000001812 28678098
Cook DA Hatala R Validation of educational assessments: a primer for simulation and beyond Adv Simul 2016 1 1 10.1186/s41077-016-0033-y 31
Powell C The delphi technique: myths and realities J Adv Nurs 2003 41 4 376 382 10.1046/j.1365-2648.2003.02537.x 12581103
de Villiers MR de Villiers PJT Kent AP “The Delphi technique in health sciences education research” Med Teach 2005 27 no. 7 639 643 10.1080/13611260500069947 16332558
J. Bartunek and M. R. Louis, Insider/outsider team research. in Qualitative research methods, no. v. 40. Thousand Oaks: Sage Publications, 1996.
Louis MR Bartunek JM Insider/outsider research teams: collaboration across diverse perspectives J Manag Inq 1992 1 2 101 110 10.1177/105649269212002
Thomas MD Blacksmith J Reno J Utilizing insider-outsider research teams in qualitative research Qual Health Res 2000 10 6 819 828 10.1177/104973200129118840 11146861
Evered R Louis MR Alternative Perspectives in the Organizational Sciences: ‘Inquiry from the inside’ and ‘Inquiry from the outside’ Acad Manage Rev 1981 6 3 385 10.2307/257374
Brannick T Coghlan D In Defense of Being ‘Native’: The Case for Insider Academic Research Organ Res Methods 2007 10 no. 1 59 74 10.1177/1094428106289253
Raufaste E Eyrolle H Mariné C Pertinence generation in radiological diagnosis: spreading activation and the nature of expertise Cogn Sci 1998 22 no. 4 517 546 10.1207/s15516709cog2204_4
H. L. Dreyfus, “Intuitive, Deliberative, and Calculative Models of Expert Performance,” in Naturalistic Decision Making, Psychology Press, 1997, pp. 17–28.
“Open Coding - Identifying Concepts,” in Basics of qualitative research: techniques and procedures for developing grounded theory, Fourth edition., Los Angeles: SAGE, 2015, pp. 222–39.
Hruschka DJ Schwartz D St.john DC Picone-Decaro E Jenkins RA Carey JW Reliability in coding open-ended data: lessons learned from HIV behavioral research Field Methods 2004 16 3 307 331 10.1177/1525822X04266540
Boyatzis RE Transforming qualitative information: thematic analysis and code development Transforming qualitative information: thematic analysis and code development 1998 Thousand Oaks, CA, US Sage Publications Inc xvi 184
M. B. Miles and A. M. Huberman, Qualitative data analysis: An expanded sourcebook, 2nd ed. in Qualitative data analysis: An expanded sourcebook, 2nd ed. Thousand Oaks, CA, US: Sage Publications, Inc, 1994, pp. xiv, 338.
Dreifuerst KT Using debriefing for meaningful learning to foster development of clinical reasoning in simulation J Nurs Educ 2012 51 6 326 333 10.3928/01484834-20120409-02 22495923
Kell HJ Martin-Raugh MP Carney LM Inglese PA Chen L Feng G Exploring methods for developing behaviorally anchored rating scales for evaluating structured interview performance ETS Res Rep Ser 2017 2017 1 1 26 10.1002/ets2.12152
Fey MK Jenkins LS Debriefing practices in nursing education programs: results from a national study Nurs Educ Perspect 2015 36 6 361 366 10.5480/14-1520 26753294
Hayden JK Smiley RA Alexander M Kardong-Edgren S Jeffries PR The NCSBN national simulation study: a longitudinal, randomized, controlled study replacing clinical hours with simulation in prelicensure nursing education J Nurs Regul 2014 5 no. 2 S3 S40 10.1016/S2155-8256(15)30062-4
Vermylen J Wood G Wayne D McGaghie W Raising the Bar: Applying a Mastery Learning Approach to Communication Skills Training J Pain Symptom Manage 2017 53 2 388 389 10.1016/j.jpainsymman.2016.12.169
C. Buléon et al., “Strategy to Develop a Common Simulation Training Program: Illustration with Anesthesia and Intensive Care Residency in France,” Teach. Learn. Med., pp. 1–13, 2022, https://doi.org/10.1080/10401334.2022.2127730.
Buléon C Eng R Rudolph JW Minehart RD First steps towards international competency goals for residency training: a qualitative comparison of 3 regional standards in anesthesiology BMC Med Educ 2021 21 1 569 10.1186/s12909-021-03007-w 34758815 8582177
Buléon C et al. The state of distance healthcare simulation during the COVID-19 pandemic: results of an international survey Adv Simul 2022 7 1 10 10.1186/s41077-022-00202-7
Hatala R Gutman J Lineberry M Triola M Pusic M How well is each learner learning? Validity investigation of a learning curve-based assessment approach for ECG interpretation Adv Health Sci Educ 2019 24 1 45 63 10.1007/s10459-018-9846-x
W. C. McGaghie, J. H. Barsuk, and D. B. Wayne, Eds., Comprehensive Healthcare Simulation: Mastery Learning in Health Professions Education. in Comprehensive Healthcare Simulation. Cham: Springer International Publishing, 2020. https://doi.org/10.1007/978-3-030-34811-3.
