AI; large language models; reproducibility; Humans; Reproducibility of Results; Large Language Models; Generative Artificial Intelligence; Intelligent Systems; Cooperative Behavior; Social Sciences/methods; Artificial Intelligence
Abstract :
[en] Large Language Models (LLMs) such as ChatGPT are transforming how scientists conduct and validate research, offering promise as tools to improve scientific reproducibility. However, computational reproducibility and error detection remain expensive and labor-intensive. We experimentally test how collaboration between researchers and LLM assistants influences the reproduction of quantitative social science findings across different levels of AI autonomy. We randomly assigned 288 researchers to 103 teams working under three conditions: human-only, AI-assisted (using ChatGPT as a collaborative tool), or AI-led (ChatGPT operating with minimal human oversight). Teams reproduced published results from leading social science journals, detected coding errors, and proposed robustness checks. Human-only and AI-assisted teams achieved comparable reproduction rates (94% vs. 91%) and performed similarly on most outcomes, except human-only teams identified significantly more major coding errors. Both substantially outperformed AI-led teams, which achieved only a 37% reproduction rate, detected fewer errors across all categories, proposed weaker robustness checks, and required more time. This autonomous approach, however, likely represents only a lower bound of AI capabilities. Despite rapid model advances, expert human judgment currently remains indispensable for reliable empirical verification. While AI assistance did not degrade most outcomes, it provided no measurable advantages and was associated with reduced detection of major errors. However, the 37% autonomous reproduction rate indicates that AI could provide value in settings where scale or cost constraints preclude human review of papers, even though general-purpose LLMs offer no immediate advantages for human-supervised verification.
Disciplines :
Library & information sciences
Author, co-author :
Brodeur, Abel ; Department of Economics, Faculty of Social Sciences, University of Ottawa, Ottawa, ON K1N 6N5, Canada ; Institute for Replication, Ottawa, ON K1N 6N5, Canada
Valenta, David ; Department of Economics, Faculty of Social Sciences, University of Ottawa, Ottawa, ON K1N 6N5, Canada ; Institute for Replication, Ottawa, ON K1N 6N5, Canada
Marcoci, Alexandru ; Institute for Technology and Humanity, University of Cambridge, Cambridge CB2 1SB, United Kingdom
Aparicio, Juan P ; Department of Economics, Faculty of Social Sciences, University of Ottawa, Ottawa, ON K1N 6N5, Canada ; Institute for Replication, Ottawa, ON K1N 6N5, Canada
Mikola, Derek; Department of Economics, Faculty of Social Sciences, University of Ottawa, Ottawa, ON K1N 6N5, Canada ; Institute for Replication, Ottawa, ON K1N 6N5, Canada
Barbarioli, Bruno ; Department of Economics, Faculty of Social Sciences, University of Ottawa, Ottawa, ON K1N 6N5, Canada ; Institute for Replication, Ottawa, ON K1N 6N5, Canada
Alexander, Rohan ; Department of Statistical Sciences, Faculty of Information, University of Toronto, Toronto, ON M5S 3G6, Canada
Deer, Lachlan ; Department of Management and Marketing, University of Melbourne, Melbourne, Carlton, VIC 3010, Australia
Stafford, Tom ; School of Psychology, University of Sheffield, Sheffield S1 4DP, United Kingdom ; Research on Research Institute, London WC1E 6JA, United Kingdom
Vilhuber, Lars ; Department of Economics, Cornell University, Ithaca, NY 14853
Bensch, Gunther ; Climate and Development Policy Division, Leibniz Institute for Economic Research - Leibniz Institute for Economic Research, Essen, NRW 45128, Germany
Motoki, Fabio ; Robert C. Vackar College of Business and Entrepreneurship, University of Texas Rio Grande Valley, Edinburg, TX 78539 ; Norwich Business School, Accounting and Quantitative Methods, University of East Anglia, Norwich NR4 7TJ, United Kingdom
Abdelhady, Mohamed ; Department of Economics, Carleton University, Ottawa, ON K1S 5B6, Canada
Abdelmoula, Yousra; Statistics Canada, Ottawa, ON K1A 0T6, Canada
Baki, Ghina Abdul ; Department of Economics, Faculty of Social Sciences, University of Ottawa, Ottawa, ON K1N 6N5, Canada ; Institute for Replication, Ottawa, ON K1N 6N5, Canada
Aguirre, Tomás; GovAI, N1 9JY, London, United Kingdom
Aiyer, Sriraj; Department of Experimental Psychology, University of Oxford, Oxfordshire OX1 3EL, United Kingdom
Akhtar, Shumi ; Finance Discipline, University of Sydney Business School, The University of Sydney, Sydney, NSW 2006, Australia
Akhtar, Farida ; Department of Actuarial Studies and Business Analytics, Macquarie Business School, Macquarie University, Sydney, NSW 2019, Australia
Albada, Melle R ; Department of Socioeconomics, Vienna University of Economics and Business, Vienna 1020, Austria
Altman, Micah; Center for Research on Equitable and Open Scholarship, Massachusetts Institute of Technology, Cabmridge, MA 02139
Angenendt, David ; Technical University of Munich School of Management, Technical University of Munich, Munich 80333, Germany ; Centre for Business Research, University of Cambridge, Cambridge CB2 1QA, United Kingdom
De León Tejada, Jorge Armando; Facultad de Economía, Universidad del Rosario, Bogotá 111711, Colombia
Arana, David Rodriguez ; School of Economics, Universidad del Rosario, Bogotá 111711, Colombia
Asanov, Igor ; International Center for Higher Education Research and Faculty of Economics, University of Kassel, Kassel, Hessen 34125, Germany
Noha, Anastasiya-Mariya ; International Center for Higher Education Research and Faculty of Economics, University of Kassel, Kassel, Hessen 34125, Germany
Ashong, Rebecca ; Department of Economics, University of Ghana, Legon, Accra, Ghana
Auer, Tobias; Department of Economics, University of Basel, Basel 4052, Switzerland
Bahamonde-Birke, Francisco J; Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, Tilburg 5037 AB, The Netherlands
Baker, Bradley J ; Department of Sport, Tourism and Hospitality Management, Temple University, Philadelphia, PA 19312
Bartram, Söhnke M; Warwick Business School, University of Warwick, Coventry CV4 7AL, United Kingdom ; Center for Economic Policy Research, Coldbath Square, London EC1R 5HL, United Kingdom
Bao, Dongqi; Department of Economics, University of Zurich, Zurich CH-8001, Switzerland
Batinovic, Lucija; Department of Behavioural Sciences and Learning, Linköping University, Linköping 58183, Sweden
Batistoni, Tommaso ; Centre for Experimental Social Sciences, Nuffield College, University of Oxford, Oxford OX1 1NF, United Kingdom
Beeder, Monica ; Department of Economics, University of Southampton, Southampton SO17 1BJ, United Kingdom
Beland, Louis-Philippe ; Department of Economics, Carleton University, Ottawa, ON K1S 5B6, Canada
Gero Bienz, Carsten ; Department of Finance, Norwegian School of Economics, Bergen 5045, Norway
Aryanto, Christ Billy ; Faculty of Psychology, Atma Jaya Catholic University of Indonesia, Jakarta 12930, Indonesia
Bolibaugh, Cylcia ; Department of Education, University of York, York YO10 5DD, United Kingdom
Bonander, Carl ; Karlstad Business School, Karlstad University, Karlstad SE-651 88, Sweden ; Center for Societal Risk Research, Karlstad University, Karlstad 65188, Sweden
Bravo, Ramiro ; Faculty of Biology Medicine and Health, The University of Manchester, Manchester M13 9NT, United Kingdom
Bronnikov, Egor; School of Business and Economics, Maastricht University, Maastricht 6211 LM, The Netherlands ; Department of Political Science, University of Chicago, Chicago, IL 60637
Bruns, Stephan; Centre for Environmental Sciences, Hasselt University, Hasselt 3500, Belgium ; International Center for Higher Education Research, University of Kassel, Kassel, Hessen 34125, Germany ; Meta-Research Innovation Center at Stanford, Stanford University, Stanford, CA 94305
Buliskeria, Nino ; Department of Economics, Nazarbayev University, Astana 010000, Kazaakhstan
Caicedo-Silva, Sara; Facultad de Economía, Universidad de los Andes, Bogotá 111711, Colombia
Calef, Andrea ; University College London School of Management, University College London, London E14 5AA, United Kingdom
Sebastian Cano Arias, Juan; School of Economics, Universidad del Rosario, Bogotá 111711, Colombia
A Castillo Alvarez, Gustavo; Department of Economics, Universidad de Los Andes, Bogotá 111711, Colombia
Caulker, Solomon; United Methodist University Sierra Leone, Freetown, Sierra Leone
Cepenas, Simonas ; Department of Economics, International School of Management University of Management and Economics, Vilnius LT-01103, Lithuania
Chatton, Arthur ; Département de Médecine Sociale et Préventive, Université Laval, Québec, QC G1V0A6, Canada ; Département de Médecine Sociale et Préventive, Université de Montréal, Montréal, QC H3N1X9, Canada
Chen, Zirou ; Department of Management, Marketing and Information Systems, Hong Kong Baptist University, Kowloon Tong, Hong Kong Special Administrative Regions of China
Chioma Ewurum, Ngozi; Department of Economics, College of Management Sciences, Michael Okpara University of Agriculture, Umudike, Abia State 440109, Nigeria
Ciocîrlan, Anda-Bianca ; School of Psychology, University of Sheffield, Sheffield S1 4DP, United Kingdom
Clouth, Felix J; Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, Tilburg 5037 AB, The Netherlands
Collins, Jason ; Business School, University of Technology Sydney, Sydney, NSW 2007, Australia
Cook, Nikolai; Department of Economics, Wilfrid Laurier University, Waterloo, ON N2L 3C5, Canada
Cornejo, Cesar ; Department of Medical Statistics, The London School of Hygiene & Tropical Medicine, London WC1E 7HT, United Kingdom ; Department of Computer Science, University College London, London WC1E 6BT, United Kingdom
Craveiro, João; School of Psychology, University of Sheffield, Sheffield S1 4DP, United Kingdom
Créchet, Jonathan; Department of Economics, Faculty of Social Sciences, University of Ottawa, Ottawa ON, Canada
Cui, Jing ; Business School, Beijing Normal University, Beijing 100875, China ; Beijing Normal University, Belt and Road School, Zhuhai, Guangdong 519085, China
Chalil Vayalabron, Niveditha; School of Earth and Planetary Science, National Institute of Science Education and Research, Odisha 752050, India
Czymara, Christian ; Migration & Migrants, Netherlands Interdisciplinary Demographic Institute, The Hague NL-2511 CV, the Netherlands
Bermúdez Jaramillo, Carlos Daniel; Department of Economics, Universidad del Rosario, Bogotá 111711, Colombia
Datta, Hannes; Department of Marketing, Tilburg University, Tilburg 5000 LE, The Netherlands
Denoo, Lien ; Department of Strategy & Entrepreneurship, Tilburg University, Tilburg 5000 LE, The Netherlands
Dhaliwal, Arshia; Department of Economics, Carleton University, Ottawa, ON K1S 5B6, Canada
Dhameja, Nency ; Department of Economics, Binghamton University, Binghamton, NY 13902
Djemai, Elodie; Economics, Université Paris Dauphine-Paris Sciences et Lettres, Paris CEDEX 16 75775, France
Dujeancourt, Erwan ; Swedish Institute for Social Research, Stockholm University, Stockholm 103 91, Sweden ; Department of Economics and Statistics, Linnaeus University, Växjö 35195, Sweden
Dündar, Uǧurcan ; Department of Marketing, Vienna University of Economics and Business Vienna, Vienna 1020, Austria
Duprey, Thibaut ; Financial Stability Department, Bank of Canada, Ottawa, ON K1A 0G9, Canada
Eissa, Yasmine ; Onsi Sawiris School of Business, The American University in Cairo, New Cairo 11835, Egypt
El Fassi, Youssef; Department of Accounting and Control, Hautes études commerciales Lausanne, Lausanne 1015, Switzerland
El Fassi, Ismail ; Institute for Accounting, Controlling and Auditing, University of St. Gallen, St. Gallen 9000, Switzerland
Ellis, Keaton; Monash University, Melbourne, VIC 3168, Australia
Elminejad, Ali ; Department of Economics, Nazarbayev University, Astana 010000, Kazaakhstan
Elsherif, Mahmoud; School of Psychology and Vision Science, University of Leicester, Leicester LE1 7RH, United Kingdom ; Psychology, University of Birmingham, Birmingham B15 2TT, United Kingdom
Emirmahmutoglu, Aysil; Department of Business and Management Science, Norwegian School of Economics, Bergen 5045, Norway
Etingin-Frati, Giulian ; Konjunkturforschungsstelle Swiss Economic Institute, ETH Zurich, Zurich 8092, Switzerland
Eze, Emeka; Department of Economics, College of Management Sciences, Michael Okpara University of Agriculture, Umudike 440109, Abia State, Nigeria
Dollbaum, Jan Fabian ; School of Politics and International Relations, University College Dublin, Dublin 4, Ireland
Feld, Jan ; School of Economics and Finance, Victoria University of Wellington, Wellington 6011, New Zealand
Felipe Rengifo Jaramillo, Andres ; Business School, Universidad de los Andes, Bogota 111711, Colombia ; Universidad de los Andes, Bogota 111711, Colombia
Fenig, Guidon; Department of Economics, Faculty of Social Sciences, University of Ottawa, Ottawa, ON K1N 6N5, Canada
Fernandes, Victoria; Vancouver School of Economics, University of British Columbia, Vancouver, BC V6T 1Z4, Canada ; Centre for Health Economics, University of York, Heslington YO10 5DD, United Kingdom
Fiala, Lenka ; Department of Economics, Faculty of Social Sciences, University of Ottawa, Ottawa, ON K1N 6N5, Canada ; Institute for Replication, Ottawa, ON K1N 6N5, Canada ; Department of Economics, Tilburg University, Tilburg, North Brabant 5037 AB, The Netherlands
Fink, Lukas ; Department of Economics, School of Business and Economics, Freie Universität Berlin, Berlin 14195, Germany
Firouzjaeiangalougah, Mojtaba ; Department of Business Management, Masaryk University, Brno 602 00, Czech Republic
Fish, Sara; School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138
Fitzgerald, Jack; Department of Ethics, Governance, and Society, School of Business and Economics, Vrije Universiteit Amsterdam, Noord-Holland, Amsterdam 1081HV, The Netherlands ; Tinbergen Institute, Amsterdam, Noord-Holland 1018WB, The Netherlands
Forshaw, Rachel ; Department of Accountancy, Economics and Finance, Heriot-Watt University, Edinburgh EH14 4AS, United Kingdom
Fortier-Chouinard, Alexandre ; Department of Political Science, Université Laval, Québec, QC G1V0A6, Canada
Fréget, Louis ; laboratoire d'Économie de Dauphine, Centre Pour la Recherche EconoMique et ses Applications, Paris 75014, France
Frese, Joris ; Department of Political and Social Sciences, European University Institute, Fiesole 50014, Italy
Gabani, Jacopo ; Health, Nutrition, and Population Global Practice, World Bank, Washington, DC 20433 ; Centre for Health Economics, University of York, Heslington YO10 5DD, United Kingdom
Gallegos, Sebastian ; Business School, Universidad Adolfo Ibañez, Santiago 7910000, Chile
Gamill, Max C ; School of Chemical, Materials and Biological Engineering, University of Sheffield, Sheffield S1 3JD, United Kingdom
Gáspár, Attila ; Institute of Economics, Eötvös Loránd University Centre for Economic and Regional Studies, Budapest 1097, Hungary ; Department of Economics, Central European University, Vienna 1100, Austria
Gauriot, Romain ; Department of Economics, Deakin University, Burwood, VIC 3125, Australia
Gavrilova, Evelina ; Department of Business and Management Science, Norwegian School of Economics, Bergen 5045, Norway
Geraldes, Diogo ; School of Economics, University College Dublin, Dublin D04 F6X4, Ireland ; Centre for Business and Economics Research, University of Coimbra, Coimbra 3000-145, Portugal ; Behavioral Science, Geary Institute for Public Poliy, Dublin D04 P9C4, Ireland
Cantone, Giulio Giacomo ; Department of Law, Economics, and Sociology, "Magna Graecia" University of Catanzaro, Catanzaro 88100, Italy
Gibson, Grant ; Department of Economics, McMaster University, Hamilton, ON L8S 4M4, Canada ; The Canadian Research Data Centre Network, Hamilton, ON L8S4M4, Canada
Goldschmitt, Dirk ; School of Psychology, University of Sheffield, Sheffield S1 4DP, United Kingdom
Gourdon-Kanhukamwe, Amélie; Institute for Psychiatry, Psychology & Neuroscience, King's College London, London SE5 9RJ, United Kingdom
Gregor de Varda, Andrea; McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
Grigoryeva, Idaliya ; Economics, University of California San Diego, La Jolla, California 92092
Gugushvili, Alexi ; Department of Sociology and Human Geography, University of Oslo, Oslo N-0317, Norway
Fletcher, Aaron H A; School of Computer Science, University of Sheffield, Sheffield S1 4DP, United Kingdom
Habermann, Florian ; University College Dublin Lochlann Quinn School of Business, University College Dublin, Dublin 4, Ireland ; Department of Accounting and Control, University of Lausanne, Lausanne 1015, Switzerland
Hablicsek, Márton; Mathematics Institute, Leiden University, Leiden, South Holland 2333CA, The Netherlands
Haddad, Joanne ; Department of Economics and Economic History, Unit of Economic Analysis, Universitat Autònoma de Barcelona, Bellaterra, Cerdanyola de Vallès 08193, Spain
Hall, Jonathan D; Department of Economics, Finance, and Legal Studies, University of Alabama, Tuscaloosa, AL 35487
Hammar, Olle ; Department of Economics and Statistics, Linnaeus University, Växjö 35195, Sweden ; Institute for Futures Studies, Stockholm 10131, Sweden
Hassouneh, Malek; Department of Economics, University of Toronto, Toronto, ON M5S 3G7, Canada
Hausladen, Carina I ; Computational Social Science, ETH Zürich, Zurich 8057, Switzerland
Hendrikse, Sophie C F; Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, Tilburg 5037 AB, The Netherlands
Hepplewhite, Matthew ; Department of Politics and International Relations, University of Oxford, Oxford OX1 3UQ, United Kingdom
Ho, Anson T Y ; Department of Real Estate Management, Ted Rogers School of Management, Toronto Metropolitan University, Toronto, ON M5B 2K3, Canada
Hogan-Hennessy, Senan ; Department of Economics, Cornell University, Ithaca, NY 14853
Howley, Elliot ; School of Psychology, University of Nottingham, Nottingham NG7 2RD, United Kingdom
Huang, Gaoyang ; Department of Economics, University of Zurich, Zurich CH-8001, Switzerland ; Department of Health Sciences and Technology, ETH Zurich, Zurich 8001, Switzerland
Hulstaert, Héloïse ; Université de Liège - ULiège > HEC Liège Research > HEC Liège Research: Financial Management for the Future ; Centre for Environmental Sciences, Hasselt University, Hasselt 3500, Belgium
Ilchovska, Zlatomira G; School of Psychology, University of Nottingham, Nottingham NG7 2RD, United Kingdom ; Department of Psychology, University of York, York YO10 5DD, United Kingdom ; School of Psychology, University of Birmingham, Birmingham B15 2SA, United Kingdom
Jaimes Santamaria, Paola ; Political Economy Research Institute, University of Massachusetts Amherst, Amherst, MA 01003 ; Center for Economic and Policy Research, Washington, DC 20009
Jakobsson, Niklas ; Karlstad Business School, Karlstad University, Karlstad SE-651 88, Sweden
Jansson, Joakim ; Department of Economics and Statistics, Linnaeus University, Växjö 35195, Sweden ; Research Institute of Industrial Economics, Stockholm 10215, Sweden
Jarosz, Ewa; Faculty of Economic Sciences, University of Warsaw, Warsaw 00-927, Poland
Jebeli, Hossein ; Financial Stability Department, Climate Analysis Team, Bank of Canada, Ottawa, ON K1A 0G9, Canada
Jiang, Yanchen ; School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138
Junaid, Hiba ; Bart's Life Sciences, Bart's Health National Health Services Trust, London E14 5HJ, United Kingdom ; Queen Mary University of London, London E1 4NS, United Kingdom
Kalluraya, Rohan; School of Electrical and Computer Engineering, Cornell University, Ithaca, NY 14853
Karim, Sunny; Carleton University, Ottawa, ON K1S 5B6
Kelly, Edmund; University of Oxford, Oxford OX1 2JD, United Kingdom
Kimel, Eva ; Department of Psychology, University of York, York YO10 5DD, United Kingdom
Kingsuwankul, Sorravich; Tinbergen Institute, Amsterdam, Noord-Holland 1018WB, The Netherlands ; Department of Management and Organization, School of Business and Economics, Vrije Universiteit Amsterdam, Amsterdam, Noord-Holland 1081 HV, Netherlands
Klotzbücher, Valentin ; Department of Economics, University of Freiburg, Freiburg im Breisgau 79085, Germany ; Department of Clinical Research, University of Basel, Basel 4051, Switzerland ; University Hospital Basel, Basel 4031, Switzerland
Krähmer, Daniel ; Department of Sociology, Ludwig Maximilian University of Munich, Munich 80539, Germany
Krūminas, Pijus ; Department of Economics, International School of Management University of Management and Economics, Vilnius LT-01103, Lithuania
Kruus, Nicholas ; Department of Politics and International Relations, University of Oxford, Oxford OX1 3UQ, United Kingdom
Kujansuu, Essi; Department of Economics, Faculty of Economics and Statistics, University of Innsbruck, Innsbruck 6020, Austria ; Department of Economics, Turku School of Economics, University of Turku, Turku FI-20014, Finland
Kurz, Christoph F; Department of Health Economics, Ludwig-Maximilians-Universität Munich, Munich DE-80539, Germany
Küster, Stephan ; School of Business and Economics, Freie Universität Berlin, Berlin 14195, Germany
Lee-Whiting, Blake ; Department of Political Science, University of Toronto, Toronto, ON M5S 3G5, Canada
Lewandowski, Felix ; School of Psychology, University of Nottingham, Nottingham NG7 2RD, United Kingdom
Li, Tongzhe; Department of Food, Agricultural and Resource Economics, University of Guelph, Guelph, ON N1G 2W1, Canada
Li, Ruoxi; Department of Economics, Yale University, New Haven, CT 06520-8268
Liu, Dan ; Research School of Economics, Australian National University, Canberra, ACT 2600, Australia
Liu, Jiacheng ; School of Finance, Renmin University of China, Beijing 100872, China
Lo, Helix ; Department of Advanced Social and International Studies, University of Tokyo, Meguro City, Tokyo 153-8902, Japan
Loter, Katharina; Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, Tilburg 5037 AB, The Netherlands
Macedo Dias, Felipe; Dyson School of Applied Economics and Management, Cornell University, Ithaca, NY 14850
Madan, Christopher R ; School of Psychology, University of Nottingham, Nottingham NG7 2RD, United Kingdom
Mäder, Nicolas ; Department of Economics, Knauss School of Business, University of San Diego, San Diego, CA 92110
Mandas, Marco ; Department of Economics and Business Administration, University of Cagliari, Cagliari 09124, Italy
Mantilla, Cesar; Department of Economics, School of Administrative and Economic Sciences, Pontificia Universidad Javeriana, Bogotá 110231, Colombia
Marcus, Jan ; Department of Economics, School of Business and Economics, Freie Universität Berlin, Berlin 14195, Germany
Marino Fages, Diego ; Department of Economics, Durham University, Durham DH1 3LB, United Kingdom
Martin, Xavier; School of Economics and Management, Tilburg University, Tilburg 5000 LE, The Netherlands
McWay, Ryan ; Applied Economics, University of Minnesota, Saint Paul, MN 55108
Medina-Gaspar, Daniel ; Department of Economics, School of Finance, Economics and Government, Universidad Escuela Finanzas Economía y Gobierno, Antioquia, Medellín 050022, Colombia
Meng, Sisi; Samuel Curtis Johnson School of Business, Cornell University, Ithaca, NY 14853
Meng, Lingyu ; School of Economics, University of Sheffield, Sheffield S10 2TU, United Kingdom
Merz, Simon ; School of Economics and Business, University of Halle, Halle (Saale) 06108, Germany
Miller, Alex P ; Department of Marketing, Marshall School of Business, University of Southern California, Los Angeles, CA 90089-1424
Mirabel, Thibault ; Equalis Capital, Paris 75116, France
Mishra, Dibya Deepta; Department of Economics, Rice University, Houston, TX 77005
Mishra, Sumit ; Institute for Financial Management and Research Graduate School of Business, Krea University, Sri City, Andhra Pradesh 517646, India
Moges, Belay W ; Department of Psychology, Dilla University, Dilla, South Ethiopia 419, Ethiopia
Mohandes Mojarrad, Morteza; Center PhD Students, Research Group: Econometrics, Tilburg School of Economics and Management, Tilburg university, Tilburg 5037 AB, Netherlands
Mohnen, Myra; Department of Economics, University of Ottawa, Ottawa, ON K1N 6N5, Canada
Morin, Louis-Philippe ; Department of Economics, Faculty of Social Sciences, University of Ottawa, Ottawa, ON K1N 6N5, Canada
Muehlenbachs, Lucija; Department of Economics, University of Calgary, Calgary, AB T2N 1N4, Canada ; Resources for the Future, Washington, DC 20036
Mullin, Gastón ; Research Group: Econometrics, Tilburg School of Economics and Management, Tilburg University, Tilburg 5037 AB, The Netherlands
Musulan, Andreea ; Department of Political Science, University of Montreal, Montreal, QC H3T 1J4, Canada ; Institut de valorisation des données, Montreal, QC H3N 1V5, Canada
Muzzì, Sara; Department of Medicine and Surgery, University of Milano-Bicocca, Milan 20126, Italy ; Department of Statistics and Quantitative Methods, University of Milano-Bicocca, Milan 20126, Italy
Myers, James A C; Department of Health Sciences, University of York, York YO10 5DD, United Kingdom
Neubauer, Florian ; Climate and Development Policy Division, RWI - Leibniz Institute for Economic Research, Berlin 10115, Germany
Nguyen, Tuan ; Centre for Environmental Sciences, Hasselt University, Hasselt 3500, Belgium
Niazi, Ali; Department of Economics, University of Calgary, Calgary, AB T2N 1N4, Canada
Nordstrom, Ardyn; School of Public Policy and Administration, Carleton University, Ottawa, ON K1S 5B6, Canada
Nowak, Bartłomiej ; Institute of Psychology, Cardinal Stefan Wyszynski University, Warsaw 01-938, Poland
O'Habib, Daneal; Currency Department, Bank of Canada, Ottawa, ON K1A 0G9, Canada
Ölkers, Tim ; Department of Agricultural Economics and Rural Development, University of Göttingen, Göttingen, Niedersachsen 37083, Germany
Ong, Justin; School of Psychology, University of Sheffield, Sheffield S1 4DP, United Kingdom
Orozco Castiblanco, Valeria; Instituto de Estudios Superiores de la Empresa Business School, University of Navarra, Barcelona 08034, Spain ; Universidad de Navarra, Pamplona 31009, Spain
Özak, Ömer ; Department of Economics, Dedman College of Humanities and Sciences, Southern Methodist University, Dallas, TX 75275-0496
Ozkes, Ali I ; SKEMA Business School, Groupe de Recherche en Droit, Économie et Gestion, Université Côte d'Azur, Lille 59777, France ; Institute for Public Management and Governance, Vienna University of Economics and Business, Vienna 1020, Austria
Paaso, Mikael; Department of Finance, Rotterdam School of Management, Erasmus University Rotterdam, Rotterdam, Zuid Holland 3012 CC, The Netherlands
Pandey, Shubham ; Institute of Psychology, Universität Osnabrück, Osnabrück 49078, Germany
Papazoglou, Varvara ; School of Computer Science, University of Sheffield, Sheffield S1 4DP, United Kingdom
Penheiro, Romeo ; Department of Psychology, University of Houston, Houston, TX 77204
Pham, Linh; Department of Economics, Business and Finance, Lake Forest College, Lake Forest, IL 60048
Phieler, Ulrike ; Department of Marketing, Vienna University of Economics and Business, Vienna 1020, Austria
Pütz, Peter ; Department of Economics, Bielefeld University, Bielefeld 33615, Germany
Qi, Quan ; Department of Economics, University at Albany, State University of New York, Albany, NY 12222
Qiu, Jingyi ; School of Information, University of Michigan, Ann Arbor, MI 48109
Rein, Manuel T ; Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, Tilburg 5037 AB, The Netherlands
Reinstein, David A; The Unjournal, Camden, DE 19934
Repo, Juuso ; Finnish Flagship Inequalities, Interventions, and New Welfare State Flagship Research Centre, University of Turku, Turku 20014, Finland
Rudolf, Nicolas; Department of Accounting and Control, University of Lausanne, Lausanne 1015, Switzerland
Saha, Shree; Applied Economics and Management, Cornell University, Ithaca, NY 14853
Saka, Orkun; Department of Economics, City St George's, University of London, London EC1V 0HB, United Kingdom
Saponaro, Chiara ; Department of Psychology, University of Milano-Bicocca, Milan 20126, Italy
Sator, Georg; School of Economics, University of Nottingham, Nottingham NG7 2RD, United Kingdom
Schoenmakers, Martijn ; Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, Tilburg 5037 AB, The Netherlands
Seri, Raffaello ; InsIDE Lab, Dipartimento di Economia, Università degli Studi dell'Insubria, Varese 21100, Italy
Shah, Meet; Department of Real Estate Management, Ted Rogers School of Management, Toronto Metropolitan University, Toronto, ON M5B 2K3, Canada
Sibille, Paul ; Université de Liège - ULiège > HEC Liège Research > HEC Liège Research: Financial Management for the Future
Siemroth, Christoph; Department of Economics, University of Essex, Colchester CO4 3SQ, United Kingdom
Skavysh, Vladimir ; Data and Digital Services Department, Bank of Canada, Ottawa, ON K1A 0G9, Canada ; Systems and Computer Engineering Department, Carleton University, Ottawa, ON K1S 5B6, Canada
Slater, Ben; Leverhulme Centre for the Future of Intelligence, University of Cambridge, Cambridge CB2 1SB, United Kingdom
Song, Wenting; Department of Economics, University of California, Davis, CA 95616
Staubli, Stefan; Department of Economics, University of Calgary, Calgary, AB T2N 1N4, Canada
Steindl, Tobias ; Institute for Business Administration, University of Regensburg, Regensburg, Bavaria 93053, Germany
Waongo, Nomwendé Steven; Department of Economics, Faculty of Social Sciences, University of Ottawa, Ottawa, ON K1N 6N5, Canada
Stott, Paul ; Department of Linguistics, Ghent University, Gent 9000, Belgium ; Department of Linguistics and English Language, University of Manchester, Manchester M13 9PL, United Kingdom
Strobel, Stephenson; Department of Economics, McMaster University, Hamilton, ON L8S 4M4, Canada
Sudhaharan, Roshini; Department of Marketing, Tilburg School of Economics and Management, Tilburg University, Tilburg 5037 AB, The Netherlands
Sun, Pu ; School of Public Finance and Taxation, Dongbei University of Finance and Economics, Dalian 116025, China
Swain, Scott D ; Department of Marketing, Clemson University, Clemson, SC 29634
Talavera, Oleksandr; Department of Economics, University of Birmingham, Birmingham B15 2TT, United Kingdom
Tantiangco, Hanz M ; School of Information, Journalism and Communication, University of Sheffield, Sheffield S10 2AH, United Kingdom
Tarasenko, Georgy ; Department of Government, Cornell University, Ithaca, NY 14853
Tarlinton, Boyd ; Agri-Science Queensland, Department of Primary Industries, Brisbane, QLD 4102, Australia
Tarraf, Mariam; Department of Economics, Carleton University, Ottawa, ON K1S 5B6, Canada
Teoh, Ken; Asia and Pacific Department, International Monetary Fund, Washington, DC 20431
Thériault, Rémi ; Institute for Replication, Ottawa, ON K1N 6N5, Canada ; Department of Psychology, New York University, New York, NY 10003
Thompson, Bethan; School of Natural and Social Sciences, Scotland's Rural College, Edinburgh EH9 3JG, United Kingdom
Tian, Tonghui; Department of Economics, Carleton University, Ottawa, ON K1S 5B6, Canada
Tian, Wenjie; Department of Economics, Faculty of Social Sciences, University of Ottawa, Ottawa, ON K1N 6N5, Canada
Tolani, Emmanuel ; Chair of Agricultural Production and Resource Economics, Technical University of Munich, Freising 85354, Germany ; The Institute for Food and Resource Economics, University of Bonn, Bonn 53115, Germany
Borgen, Nicolai ; Department of Special Needs Education, Centre for Research on Equality in Education, University of Oslo, Oslo 0318, Norway
Topstad Borgen, Solveig ; Department of Sociology and Human Geography, University of Oslo, Oslo N-0317, Norway
Torralba, Javier; Department of Marketing, Tilburg University, Tilburg 5000 LE, The Netherlands
Velez-Ospina, Carolina; World Bank, Washington, DC 20433
Mak, Man Wai ; Department of Economics, Carleton University, Ottawa, ON K1S 5B6, Canada
Wallrich, Lukas; Birkbeck Business School, Birkbeck, University of London, London WC1E 7JL, United Kingdom
Wang, Zeyang; Department of Economics, Vanderbilt University, Nashville, TN 37235-1819
Ward, Leah; Division of Psychology & Mental Health, School of Health Sciences, University of Manchester, Manchester M13 9PL, United Kingdom
Webb, Matthew D ; Department of Economics, Carleton University, Ottawa, ON K1S 5B6, Canada
Webb, Duncan; Center for Health and Wellbeing, Princeton University, Princeton, NJ 08544
Weber, Bryan S ; Department of Economics, College of Staten Island, Staten Island, NY 10314 ; City University of New York, New York, NY 10017
Weber, Christoph ; Department Economics, Law, and Society, Ecole upérieure des sciences commerciales School of Management, Angers 49003, France
Weng, Wei-Chien ; Department of Economics, University of California, Davis, CA 95616
Westheide, Christian; Stockholm Business School, Stockholm University, Stockholm 10691, Sweden ; Leibniz Institute for Financial Research Sustainable Architecture for Finance in Europe, Frankfurt, Hesse 60323, Germany
Wilkinson, Tom ; School of Chemical, Materials and Biological Engineering, University of Sheffield, Sheffield S1 3JD, United Kingdom
Wong, Kwong-Yu ; Department of Economics, National University of Singapore, Singapore 117570, Singapore
Wroński, Marcin ; Collegium of World Economy, Szkoła Główna Handlowa Warsaw School of Economics, Warsaw 02-554, Poland
Wu, Zhuangchen; Department of Economics, University of Birmingham, Birmingham B15 2TT, United Kingdom
Wu, Qixia ; Department of Economics, Faculty of Social Sciences, University of Ottawa, Ottawa, ON K1N 6N5, Canada
Wu, Victor Y ; Department of Political Science, Stanford University, Stanford, CA 94305
Xiao, Bohan ; Department of Economics, Faculty of Social Sciences, University of Ottawa, Ottawa, ON K1N 6N5, Canada
Xu, Feihong ; Department of Engineering Sciences & Applied Mathematics, Northwestern University, Evanston, IL 60201
Xu, Cong; Real Estate Economics, National Chengchi University, Taipei 11605, Taiwan ; Aalto University, Espoo 02150, Finland
Yadav, Pranav ; Department of Accounting, Tilburg School of Economics and Management, Tilburg University, Tilburg 5037 AB, The Netherlands
Yang Chou, Yu ; Institute for Psychiatry, Psychology & Neuroscience, King's College London, London SE5 9RJ, United Kingdom
Yap, Luther; Department of Economics, Faculty of Arts and Social Sciences, National University of Singapore, Singapore 117570, Singapore
Yazbeck, Myra ; Department of Economics, Faculty of Social Sciences, University of Ottawa, Ottawa, ON K1N 6N5, Canada ; School of Economics, Faculty of Buisness Economics and Law, University of Queensland, Brisbane, St Lucia, QLD 4072, Australia
Yao, Bo; Department of Psychology, Faculty of Science and Technology, Fylde College, Lancaster University, Lancaster LA1 4YF, United Kingdom
Zagrodzka, Zuzanna ; School of Biosciences, University of Sheffield, Sheffield S10 2TN, United Kingdom
Zahra, Tahreen ; Department of Economics, Carleton University, Ottawa, ON K1S 5B6, Canada
Zaneva, Mirela; Christ Church College, University of Oxford, Oxfordshire OX1 1DP, United Kingdom
Open Philanthropy Project Alfred P. Sloan Foundation U of T - University of Toronto University of Ottawa CU - Cornell University Tilburg University Leverhulme Trust
A. Brodeur et al., Promoting reproducibility and replicability in political science. Res. Polit. 11, 20531680241233439 (2024).
D. L. Donoho, A. Maleki, I. U. Rahman, M. Shahram, V. Stodden, Reproducible research in computational harmonic analysis. Comput. Sci. Eng. 11, 8–18 (2008).
M. Fišar et al., Reproducibility in management science. Manag. Sci. 70, 1343–1356 (2024).
P. Gertler, S. Galiani, M. Romero, How to make replication the norm. Nature 554, 417–419 (2018).
S. N. Goodman, D. Fanelli, J. P. Ioannidis, What does research reproducibility mean? Sci. Transl. Med. 8, 341ps12-341ps12 (2016).
M. Milkowski, W. M. Hensel, M. Hohol, Replicability or reproducibility? On the replication crisis in computational neuroscience and sharing only relevant detail. J. Comput. Neurosci. 45, 163–172 (2018).
National Academies of Sciences, Engineering, and Medicine, Reproducibility and Replicability in Science (National Academies Press, 2019).
C. Pérignon, K. Gadouche, C. Hurlin, R. Silberman, E. Debonnel, Certify reproducibility with confidential data. Science 365, 127–128 (2019).
L. Vilhuber, Report by the AEA data editor. AEA Pap. Proc. 112, 813–823 (2022).
A. Brodeur et al., Reproducibility and robustness of economics and political science research. Nature 652, 151–156 (2026).
A. C. Chang, P. Li, Is economics research replicable? Sixty published papers from thirteen journals say “often not”. Crit. Finance Rev. 11, 185–206 (2022).
S. Crüwell et al., What’s in a badge? A computational reproducibility investigation of the open data badge policy in one issue of psychological science.. Psychol. Sci. 34, 512–522 (2023).
Open Science Collaboration, Estimating the reproducibility of psychological science. Science 349, aac4716 (2015).
P. Obels, D. Lakens, N. A. Coles, J. Gottfried, S. A. Green, Analysis of open data and computational reproducibility in registered reports in psychology. Adv. Methods Pract. Psychol. Sci. 3, 229–237 (2020).
C. Pérignon et al., Computational reproducibility in finance: Evidence from 1,000 tests. Rev. Financial Stud. 37, 3558–3593 (2024).
V. Stodden, J. Seiler, Z. Ma, An empirical analysis of journal policy effectiveness for computational reproducibility. Proc. Natl. Acad. Sci. U.S.A. 115, 2584–2589 (2018).
B. Wood, R. Müller, A. Brown, Push button replication: Is impact evaluation evidence for international development verifiable? PloS one 13, e0209416 (2018).
J. E. Colliard, C. Hurlin, C. Perignon, The Economics of Computational Reproducibility (2022). https://ssrn.com/abstract=3418896.
A. Hrycyshyn, H. Eassom, “ExplanAItions: An AI study” (Tech. Rep. 1, John Wiley & Sons, Hoboken, NJ, 2025).
M. Vaccaro, A. Almaatouq, T. Malone, When combinations of humans and ai are useful: A systematic review and meta-analysis. Nat. Hum. Behav. 8, 1–11 (2024).
G. Bansal et al., “Does the whole exceed its parts? The effect of AI explanations on complementary team performance” in Proceedings of the 2021 CHI conference on human factors in computing systems (2021), pp. 1–16.
G. Bansal, B. Nushi, E. Kamar, E. Horvitz, D. S. Weld, “Is the most accurate AI the best teammate? Optimizing AI for teamwork” in Proceedings of the AAAI Conference on Artificial Intelligence (PKP Publishing Services Network, 2021), vol. 35, pp. 11405–11414.
E. Bondi et al., “Role of human-AI interaction in selective prediction” in Proceedings of the AAAI Conference on Artificial Intelligence (PKP Publishing Services Network, 2022), vol. 36, pp. 5286–5294.
Á. A. Cabrera, A. Perer, J. I. Hong, Improving human-AI collaboration with descriptions of AI behavior. Proc. ACM Hum. Comput. Interact. 7, 1–21 (2023).
E. Goh et al., Influence of a large language model on diagnostic reasoning: A randomized clinical vignette study. medRxiv [Preprint] (2024). https://doi.org/10.1101/2024.03.12.24303785 (Accessed 20 February 2026).
B. Koepnick et al., De novo protein design by citizen scientists. Nature 570, 390–394 (2019).
H. Liu, V. Lai, C. Tan, Understanding the effect of out-of-distribution examples and interactive explanations on human-AI decision making. Proc. ACM Hum. Comput. Interact. 5, 1–45 (2021).
H. Mozannar et al., Effective human-AI teams via learned natural language rules and onboarding. Adv. Neural Inf. Process. Syst. 36, e01007 (2024).
S. Noy, W. Zhang, Experimental evidence on the productivity effects of generative artificial intelligence. Science 381, 187–192 (2023).
C. Reverberi et al., Experimental evidence of effective human-AI collaboration in medical decision-making. Sci. Rep. 12, 14952 (2022).
M. Schemmer, P. Hemmer, M. Nitsche, N. Kühl, M. Vössing, “A meta-analysis of the utility of explainable artificial intelligence in human-AI decision-making” in Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society (2022), pp. 617–626.
M. H. Tessler et al., AI can help humans find common ground in democratic deliberation. Science 386, eadq2852 (2024).
M. Vaccaro, J. Waldo, The effects of mixing machine learning and human judgment. Commun. ACM 62, 104–110 (2019).
B. Wilder, E. Horvitz, E. Kamar, “Learning to complement humans” in Proceedings of the 29th International Joint Conference on Artificial Intelligence (ACM Digital Library, 2020), pp. 1526–1533.
Z. Buçinca, M. B. Malaya, K. Z. Gajos, To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proc. ACM Hum. Comput. Interact. 5, 1–21 (2021).
L. J. Skitka, K. L. Mosier, M. Burdick, Does automation bias decision-making? Int. J. Hum. Comput. Stud. 51, 991–1006 (1999).
E. Brynjolfsson, D. Li, L. Raymond, Generative AI at work. Q. J. Econ. 140, 889–942 (2025).
M. Del Giudice, S. W. Gangestad, A traveler’s guide to the multiverse: Promises, pitfalls, and a framework for the evaluation of analytic decisions. Adv. Methods Pract. Psychol. Sci. 4, 2515245920954925 (2021).
X. Lu, H. White, Robustness checks and robustness tests in applied economics. J. Econom. 178, 194–206 (2014).
M. B. Nuijten, “Assessing and improving robustness of psychological research findings in four steps” in Avoiding Questionable Research Practices in Applied Psychology (Springer, 2022), pp. 379–400.
V. Lai, H. Liu, C. Tan,“‘why is’ chicago’deceptive?’ Towards building model-driven tutorials for human” in Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (ACM Digital Library, 2020), pp. 1–13.
Y. Zhang, Q. V. Liao, R. K. Bellamy, “Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making” in Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (ACM Digital Library, 2020), pp. 295–305.
H. Vasconcelos et al., Explanations can reduce overreliance on AI systems during decision-making. Proc. ACM Hum. Comput. Interact. 7, 1–38 (2023).
Y. K. Dwivedi et al., Opinion paper: “So what if chatgpt wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational ai for research, practice and policy. Int. J. Inf. Manag. 71, 102642 (2023).
B. D. Lund et al., Chatgpt and a new academic reality: Artificial intelligence-written research papers and the ethics of the large language models in scholarly publishing. J. Assoc. Inf. Sci. Technol. 74, 570–581 (2023).
N. Wadhwa et al., Core: Resolving code quality issues using LLMs. Proc. ACM Softw. Eng. 1, 789–811 (2024).
Y. Zhang, “Detecting code comment inconsistencies using LLM and program analysis” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (ACM Digital Library, 2024), pp. 683–685.
D. Nam, A. Macvean, V. Hellendoorn, B. Vasilescu, B. Myers, “Using an LLM to help with code understanding” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ACM Digital Library, 2024), pp. 1–13.
A. Brodeur et al., AI Replication Games. https://github.com/I4Replication/AI- Games. GitHub. Accessed 2 May 2026.
