Health policies; Non-pharmaceutical interventions; COVID-19; evaluation; realism
Abstract :
[en] A recent report by the Royal Society examined the literature on the effectiveness of non-pharmaceutical interventions (NPIs) aimed at reducing the transmission of SARS-CoV-2 during the COVID-19 pandemic. It emphatically concluded that they were ‘effective’, which we argue may be misleading to policymakers. This paper performs a critical analysis of the Royal Society report and explains what a proper evaluation of NPIs would mean, arguing that theory-based evaluations are required to approach complex issues and nurture democratic debates on societal choices. We argue that, first, testing the relationship between NPI interventions and their effectiveness in reducing SARS-CoV-2 transmission over a bounded period (a single outcome) is irrelevant for policymaking, because several criteria must be balanced to evaluate any intervention, including efficiency, equity, acceptability, long-term impacts, and sustainability. Second, ignoring the disruptive and unintended effects of NPIs leads to wrong conclusions regarding their overall value. Third, we question the mere objective of NPIs, that is, reducing viral transmission. Finally, we question the methodology used by the Royal Society Report regarding the literature selection and quality. To conclude, we argue that the Royal Society report is not only irrelevant and weak from a methodological point of view but also dangerously misleading in terms of policymaking. Four years after the start of the COVID-19 pandemic, it is high time for researchers, journals, and policymakers to stop debating over NPIs’ mere effectiveness against a parameter that requires a PCR test to determine, but engage in public health-based evaluations that weigh all criteria of interest.
Disciplines :
Public health, health care sciences & services Political science, public administration & international relations
Author, co-author :
Paul, Elisabeth ; Université de Liège - ULiège > Département de droit > Droit fiscal ; School of Public Health, Université libre de Bruxelles, Brussels, Belgium
Brown, Garrett W. ; School of Politics and International Studies, University of Leeds, Leeds, UK
Bell, David ; Independent Public Health Consultant, Lake Jackson, TX, USA
von Agris, Jean Merlin ; School of Politics and International Studies, University of Leeds, Leeds, UK
Ridde, Valéry ; Université Paris Cité, IRD, Ceped, Paris, France
Language :
English
Title :
Royal Society report: what would a comprehensive evaluation suggest about non-pharmaceutical interventions during COVID-19?
Abaluck, J., Kwong, L. H., Styczynski, A., Haque, A., Kabir, M. A., Bates-Jefferys, E., Crawford, E., Benjamin-Chung, J., Raihan, S., Rahman, S., Benhachmi, S., Bintee, N. Z., Winch, P. J., Hossain, M., Reza, H. M., Jaber, A. A., Momen, S. G., Rahman, A., Banti, F. L., … Mobarak, A. M., (2021). Impact of community masking on COVID-19: A cluster-randomized trial in Bangladesh. Science, 375(6577), eabi9069. https://doi.org/10.1126/science.abi9069
Abe, K., Kawachi, I., Iba, A., & Miyawaki, A., (2023). In-hospital deaths from ambulatory care–sensitive conditions before and during the COVID-19 pandemic in Japan. JAMA Network Open, 6(6), e2319583–e2319583. https://doi.org/10.1001/jamanetworkopen.2023.19583
Ahmed, T., Roberton, T., Vergeer, P., Hansen, P. M., Peters, M. A., Ofosu, A. A., Mwansambo, C., Nzelu, C., Wesseh, C. S., Smart, F., Alfred, J. P., Diabate, M., Baye, M., Yansane, M. L., Wendrad, N., Mohamud, N. A., Mbaka, P., Yuma, S., Ndiaye, Y., … Shapira, G., (2022). Healthcare utilization and maternal and child mortality during the COVID-19 pandemic in 18 low- and middle-income countries: An interrupted time-series analysis with mathematical modeling of administrative data. PLOS Medicine, 19(8), e1004070. https://doi.org/10.1371/journal.pmed.1004070
Andersson, F. N. G., & Jonung, L., (2024). The covid-19 lesson from Sweden: Don’t lock down. Economic Affairs, 44(1), 3–16. https://doi.org/10.1111/ecaf.12611
Bamberger, M. J., Rugh, J., & Mabry, L., (2006). RealWorld evaluation: Working under budget, time, data, and political constraints. SAGE Publications.
Bardosh, K., (2023). How did the COVID pandemic response harm society? A global evaluation and state of knowledge review (2020-21). SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4447806
Bavli, I., Sutton, B., & Galea, S., (2020). Harms of public health interventions against COVID-19 must not be ignored. BMJ: British Medical Journal, 371, m4074. https://doi.org/10.1136/bmj.m4074
Beauchamp, J. D., & Mayhew, C. A., (2023). Revisiting the rationale of mandatory masking. Journal of Breath Research, 17(4), 042001. https://doi.org/10.1088/1752-7163/acdf12
Bundervoet, T., Dávalos, M. E., & Garcia, N., (2022). The short-term impacts of COVID-19 on households in developing countries: An overview based on a harmonized dataset of high-frequency surveys. World Development, 153, 105844. https://doi.org/10.1016/j.worlddev.2022.105844
Bundgaard, H., Bundgaard, J. S., Raaschou-Pedersen, D. E. T., von Buchwald, C., Todsen, T., Norsk, J. B., Pries-Heje, M. M., Vissing, C. R., Nielsen, P. B., Winsløw, U. C., Fogh, K., Hasselbalch, R., Kristensen, J. H., Ringgaard, A., Porsborg Andersen, M., Goecke, N. B., Trebbien, R., Skovgaard, K., Benfield, T., … Iversen, K., (2021). Effectiveness of adding a mask recommendation to other public health measures to prevent SARS-CoV-2 infection in Danish mask wearers. Annals of Internal Medicine, 174(3), 335–343. https://doi.org/10.7326/M20-6817
Campeau, L., Degroote, S., Ridde, V., Carabali, M., & Zinszer, K., (2018). Containment measures for emerging and re-emerging vector-borne and other infectious diseases of poverty in urban settings: A scoping review. Infectious Diseases of Poverty, 7(1), Article 1. https://doi.org/10.1186/s40249-018-0478-4
Carillon, S., Gosselin, A., Coulibaly, K., Ridde, V., & Desgrées du Loû, A., (2020). Immigrants facing covid 19 containment in France: An ordinary hardship of disaffiliation. Journal of Migration and Health, 1‑2, 100032. https://doi.org/10.1016/j.jmh.2020.100032
Chou, R., & Dana, T., (2023). Major update: Masks for prevention of SARS-CoV-2 in health care and community settings—final update of a living, rapid review. Annals of Internal Medicine, 176(6), 827–835. https://doi.org/10.7326/M23-0570
Christakis, D. A., Van Cleve, W., & Zimmerman, F. J., (2020). Estimation of US children’s educational attainment and years of life lost associated with primary school closures during the coronavirus disease 2019 pandemic. JAMA Network Open, 3(11), Article 11. e2028786. https://doi.org/10.1001/jamanetworkopen.2020.28786
Craig, P., DiRuggiero, E., Frohlich, K., Mykhalovskiy, E., & White, M., (2018). Taking account of context in population health intervention research: Guidance for producers, users and funders of research[NIHR Evaluation, Trials and Studies Coordinating Centre]. on behalf of the Canadian Institutes of Health Research. CIHR)–National Institute for Health Research (NIHR. https://www.ncbi.nlm.nih.gov/books/NBK498645/pdf/Bookshelf_NBK498645.pdf
Diallo, I., Ndejjo, R., Leye, M. M. M., Egbende, L., Tusubira, A., Bamgboye, E. A., Fall, M., Namuhani, N., Bosonkie, M., Salawu, M. M., Ndiaye, Y., Kabwama, S. N., Sougou, N. M., Bello, S., Bassoum, O., Babirye, Z., Afolabi, R. F., Gueye, T., Kizito, S., … Seck, I., (2023). Unintended consequences of implementing non-pharmaceutical interventions for the COVID-19 response in Africa: Experiences from DRC, Nigeria, Senegal, and Uganda. Globalization and Health, 19(1), 36. https://doi.org/10.1186/s12992-023-00937-6
Donovan, K. P., (2018). The rise of the randomistas: On the experimental turn in international aid. Economy and Society, 47(1), 27–58. https://doi.org/10.1080/03085147.2018.1432153
Drummond, M. F., Sculpher, M. J., Claxton, K., Stoddart, G. L., & Torrance, G. W., (2015). Methods for the economic evaluation of health care programmes (4th ed.). Oxford University Press.
Felfe, C., Saurer, J., Schneider, P., Vornberger, J., Erhart, M., Kaman, A., & Ravens-Sieberer, U., (2023). The youth mental health crisis: Quasi-experimental evidence on the role of school closures. Science Advances, 9(33), eadh4030. https://doi.org/10.1126/sciadv.adh4030
Furlong, A., (2023, août24). Top review says COVID lockdowns and masks worked, period. https://www.politico.eu/article/masks-lockdowns-social-distancing-evidence-reduce-coronavirus-spread/
Gupta, S., Georgiou, A., Sen, S., Simon, K., & Karaca-Mandic, P., (2021). US trends in COVID-19–associated hospitalization and mortality rates before and after reopening economies. JAMA Health Forum, 2(6), e211262–e211262. https://doi.org/10.1001/jamahealthforum.2021.1262
Hale, T., Angrist, N., Goldszmidt, R., Kira, B., Petherick, A., Phillips, T., Webster, S., Cameron-Blake, E., Hallas, L., Majumdar, S., & Tatlow, H., (2021). A global panel database of pandemic policies (Oxford COVID-19 Government response tracker). Nature Human Behaviour, 5(4), 529–538. https://doi.org/10.1038/s41562-021-01079-8
Hawe, P., Shiell, A., & Riley, T., (2009). Theorising interventions as events in systems. American Journal of Community Psychology, 43(3‑4), 267–276. https://doi.org/10.1007/s10464-009-9229-9
Herby, J., Jonung, L., & Hanke, S. H., (2023). Did lockdowns work? The verdict on Covid restrictions. Institute of Economic Affairs (IEA). https://iea.org.uk/publications/did-lockdowns-work-the-verdict-on-covid-restrictions/?fbclid=IwAR2B0GDS5POsc08F91O_-tNykE-Gjx7P4VLpxNwWji4_CuqAvbcVjnAK9II
Horton, R., (2021). Offline: The case for No-COVID. The Lancet, 397(10272), 359. https://doi.org/10.1016/S0140-6736(21)00186-0
International Monetary Fund. (2023, avril). World economic outlook (April 2023). https://www.imf.org/external/datamapper/NGDP_RPCH@WEO/WEOWORLD
Ioannidis, J. P. A., Zonta, F., & Levitt, M., (2023). Variability in excess deaths across countries with different vulnerability during 2020–2023. Proceedings of the National Academy of Sciences, 120(49), e2309557120. https://doi.org/10.1073/pnas.2309557120
Jefferson, T., Dooley, L., Ferroni, E., Al-Ansary, L., van Driel, M., Bawazeer, G., Jones, M., Hoffmann, T., Clark, J., Beller, E., Glasziou, P. P., & Conly, J. M., (2023). Physical interventions to interrupt or reduce the spread of respiratory viruses. Cochrane Database of Systematic Reviews, 1(4). https://doi.org/10.1002/14651858.CD006207.pub6
Joffe, A. R., & Redman, D., (2021). The SARS-CoV-2 pandemic in high income countries such as Canada: A better way forward without lockdowns. Frontiers in Public Health, 9, 9. https://doi.org/10.3389/fpubh.2021.715904
Jones, M., (2021, February17). COVID response drives $24 trillion surge in global debt: IIF. Reuters. https://www.reuters.com/article/us-global-debt-iif/covid-response-drives-24-trillion-surge-in-global-debt-iif-idUSKBN2AH285
Juutinen, A., Sarvikivi, E., Laukkanen-Nevala, P., & Helve, O., (2023). Face mask recommendations in schools did not impact COVID-19 incidence among 10–12-year-olds in Finland–joinpoint regression analysis. BMC Public Health, 23(1), 730. https://doi.org/10.1186/s12889-023-15624-9
Kadri, S. S., Sun, J., Lawandi, A., Strich, J. R., Busch, L. M., Keller, M., Babiker, A., Yek, C., Malik, S., Krack, J., Dekker, J. P., Spaulding, A. B., Ricotta, E., Powers, J. H., Rhee, C., Klompas, M., Athale, J., Boehmer, T. K., Gundlapalli, A. V., … Warner, S., (2021). Association between caseload surge and COVID-19 survival in 558 U.S. Hospitals, March to August 2020. Annals of Internal Medicine, 174(9), 1240–1251. https://doi.org/10.7326/M21-1213
Kalk, A., Sturmberg, J., Van Damme, W., Brown, G., Ridde, V., Zizi, M., & Paul, E., (2022). Surfing corona waves–instead of breaking them: Rethinking the role of natural immunity in COVID-19 policy. F1000research, 11(337), Article 337. 337. https://doi.org/10.12688/f1000research.110593.1
Kapitsinis, N., (2021). The underlying factors of excess mortality in 2020: A cross-country analysis of pre-pandemic healthcare conditions and strategies to cope with covid-19. BMC Health Services Research, 21(1), Article 1. https://doi.org/10.1186/s12913-021-07169-7
Lartey, S. T., Jayawardene, W. P., Dickinson, S. L., Chen, X., Gletsu-Miller, N., & Lohrmann, D. K., (2023). Evaluation of unintended consequences of COVID-19 pandemic restrictions and obesity prevalence among youths. JAMA Network Open, 6(7), e2323596–e2323596. https://doi.org/10.1001/jamanetworkopen.2023.23596
León, T., Dorabawila, V., & Nelson, L., Lutterloh, E., Bauer, U. E., Backenson, B., Bassett, M. T., Henry, H., Bregman, B., Midgley, C. M., Myers, J. F., Plumb, I. D., Reese, H. E., Zhao, R., Briggs-Hagen, M., Hoefer, D., Watt, J. P., Silk, B. J., Jain, S., & Rosenberg, E. S., (2022). COVID-19 cases and hospitalizations by COVID-19 vaccination status and previous COVID-19 diagnosis—California and New York, May–November 2021. MMWR Morb Mortal Wkly Rep 71: 125-131; Numéro 71: 25-131. Centers for Disease Control and Prevention. https://www.cdc.gov/mmwr/volumes/71/wr/mm7104e1.htm?s_cid=mm7104e1_w
Levin, A. T., Hanage, W. P., Owusu-Boaitey, N., Cochran, K. B., Walsh, S. P., & Meyerowitz-Katz, G., (2020). Assessing the age specificity of infection fatality rates for COVID-19: Systematic review, meta-analysis, and public policy implications. European Journal of Epidemiology, 35(12), 1123–1138. https://doi.org/10.1007/s10654-020-00698-1
Li, L., Taeihagh, A., & Tan, S. Y., (2023). A scoping review of the impacts of COVID-19 physical distancing measures on vulnerable population groups. Nature Communications, 14(1), 599. https://doi.org/10.1038/s41467-023-36267-9
Murphy, C., Lim, W. W., Mills, C., Wong, J. Y., Chen, D., & Xie, Y., Li, M., Gould, S., Xin, H., Cheung, J. K., Bhatt, S., Cowling, B. J., & Donnelly, C. A., (2023). Appendix B from effectiveness of social distancing measures and lockdowns for reducing transmission of COVID-19 in non-healthcare, community-based settings. The Royal Society. https://doi.org/10.6084/m9.figshare.23283511.v1
OECD. (2020). Evaluation criteria. Organisation for Economic Cooperation and Development. https://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm
Ost, K., Duquesne, L., Duguay, C., Traverson, L., Mathevet, I., Ridde, V., & Zinszer, K., (2022). Large-scale infectious disease testing programs have little consideration for equity: Findings from a scoping review. Journal of Clinical Epidemiology, 143, 30–60. https://doi.org/10.1016/j.jclinepi.2021.11.030
Our World in Data. (2023). COVID-19 data explorer. https://ourworldindata.org/explorers/coronavirus-data-explorer?zoomToSelection=true&time=2020-03-01.latest&facet=none&country=USA~GBR~CAN~DEU~ITA~IND&pickerSort=asc&pickerMetric=location&Metric=Confirmed+cases&Interval=7-day+rolling+average&Relative+to+Population=true&Color+by+test+positivity=false
Paul, E., Brown, G. W., Kalk, A., Van Damme, W., Ridde, V., & Sturmberg, J., (2022). “When My Information Changes, I Alter My Conclusions.” What can we learn from the failures to adaptively respond to the SARS-CoV-2 pandemic and the under preparedness of health systems to manage COVID-19?International Journal of Health Policy and Management, 11(7), 1241–1245. https://doi.org/10.34172/ijhpm.2020.240
Pawson, R., & Tilley, N., (1997). Realistic evaluation. Sage.
The Royal Society. (2023). COVID-19: Examining the effectiveness of non-pharmaceutical interventions. https://royalsociety.org/topics-policy/projects/impact-non-pharmaceutical-interventions-on-covid-19-transmission/
Russell, F. M., Curtis, N., & Carlin, J., (2023). Lifting universal school masking—Covid-19 incidence among students and staff. New England Journal of Medicine, 389(6), 578–580. https://doi.org/10.1056/NEJMc2215560
Schmets, G., Rajan, D., & Kadandale, S., (2016). Strategizing national health in the 21st century: A handbook. World Health Organization.
Seal, A., Schaffner, A., Phelan, S., Brunner-Gaydos, H., Tseng, M., Keadle, S., Alber, J., Kiteck, I., & Hagobian, T., (2022). COVID-19 pandemic and stay-at-home mandates promote weight gain in US adults. Obesity, 30(1), 240–248. https://doi.org/10.1002/oby.23293
Skivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K. A., Craig, N., French, D. P., McIntosh, E., Petticrew, M., Rycroft-Malone, J., White, M., & Moore, L., (2021). A new framework for developing and evaluating complex interventions: Update of medical research council guidance. BMJ: British Medical Journal, 374, n2061. https://doi.org/10.1136/bmj.n2061
Sturmberg, J. P., (2023). Changing the paradigm of research. Journal of Evaluation in Clinical Practice, 29(5), 726–729. https://doi.org/10.1111/jep.13828
Talic, S., Shah, S., Wild, H., Gasevic, D., Maharaj, A., Ademi, Z., Li, X., Xu, W., Mesa-Eguiagaray, I., Rostron, J., Theodoratou, E., Zhang, X., Motee, A., Liew, D., & Ilic, D., (2021). Effectiveness of public health measures in reducing the incidence of covid-19, SARS-CoV-2 transmission, and COVID-19 mortality: Systematic review and meta-analysis. BMJ: British Medical Journal, 375, e068302. https://doi.org/10.1136/bmj-2021-068302
Teglia, F., Angelini, M., Astolfi, L., Casolari, G., & Boffetta, P., (2022). Global association of COVID-19 pandemic measures with cancer screening: A systematic review and meta-analysis. JAMA Oncology, 8(9), 1287–1293. https://doi.org/10.1001/jamaoncol.2022.2617
Turcotte-Tremblay, A.-M., Gali Gali, I. A., & Ridde, V., (2021). The unintended consequences of COVID-19 mitigation measures matter: Practical guidance for investigating them. BMC Medical Research Methodology, 21(1), 28. https://doi.org/10.1186/s12874-020-01200-x
United Nations Educational, Scientific and Cultural Organization (UNESCO). (2022). Survey on National education responses to COVID-19 school closures. https://covid19.uis.unesco.org/school-closures-survey/
Verity, R., Okell, L. C., Dorigatti, I., Winskill, P., Whittaker, C., Imai, N., Cuomo-Dannenburg, G., Thompson, H., Walker, P. G. T., Fu, H., Dighe, A., Griffin, J. T., Baguelin, M., Bhatia, S., Boonyasiri, A., Cori, A., Cucunubá, Z., FitzJohn, R., Gaythorpe, K., … Ferguson, N. M., (2020). Estimates of the severity of coronavirus disease 2019: A model-based analysis. The Lancet Infectious Diseases, 20(6), 669–677. https://doi.org/10.1016/S1473-3099(20)30243-7
Violato, M., Pollard, J., Lloyd, A., Roope, L. S. J., Duch, R., Becerra, M. F., Clarke, P. M., & Kola, L., (2023). The COVID-19 pandemic and health-related quality of life across 13 high- and low-middle-income countries: A cross-sectional analysis. PLOS Medicine, 20(4), 1–20. https://doi.org/10.1371/journal.pmed.1004146
Williamson, E. J., Walker, A. J., Bhaskaran, K., Bacon, S., Bates, C., Morton, C. E., Curtis, H. J., Mehrkar, A., Evans, D., Inglesby, P., Cockburn, J., McDonald, H. I., MacKenna, B., Tomlinson, L., Douglas, I. J., Rentsch, C. T., Mathur, R., Wong, A. Y. S., Grieve, R., … Goldacre, B., (2020). Factors associated with COVID-19-related death using Opensafely. Nature, 584(7821), 430–436. https://doi.org/10.1038/s41586-020-2521-4
World Health Organization, G. I. P. (2019). Non-pharmaceutical public health measures for mitigating the risk and impact of epidemic and pandemic influenza. https://apps.who.int/iris/bitstream/handle/10665/329438/9789241516839-eng.pdf?ua=1
Xiao, H., Wang, Z., Liu, F., & Unger, J. M., (2023). Excess all-cause mortality in China after ending the zero COVID policy. JAMA Network Open, 6(8), e2330877. https://doi.org/10.1001/jamanetworkopen.2023.30877
