[en] Objectives Meta-analyses (MAs) are often used because they are lauded to provide robust evidence that synthesises information from multiple studies. However, the validity of MA conclusions relies on the procedural rigour applied by the authors. Therefore, this meta-research study aims to characterise the methodological quality and meta-analytic practices of MAs indexed in PsycINFO.
Design A meta-epidemiological study.
Participants We evaluated a random sample of 206 MAs indexed in the PsycINFO database in 2016.
Primary and secondary outcomes Two authors independently extracted the methodological characteristics of all MAs and checked their quality according to the 16 items of the A MeaSurement Tool to Assess systematic Reviews (AMSTAR2) tool for MA critical appraisal. Moreover, we investigated the effect of mentioning Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) on the methodological quality of MAs.
Results According to AMSTAR2 criteria, 95% of the 206 MAs were rated as critically low quality. Statistical methods were appropriate and publication bias was well evaluated in 87% and 70% of the MAs, respectively. However, much improvement is needed in data collection and analysis: only 11% of MAs published a research protocol, 44% had a comprehensive literature search strategy, 37% assessed and 29% interpreted the risk of bias in the individual included studies, and 11% presented a list of excluded studies. Interestingly, the explicit mentioning of PRISMA suggested a positive influence on the methodological quality of MAs.
Conclusion The methodological quality of MAs in our sample was critically low according to the AMSTAR2 criteria. Some efforts to tremendously improve the methodological quality of MAs could increase their robustness and reliability.
Disciplines :
Social & behavioral sciences, psychology: Multidisciplinary, general & others Public health, health care sciences & services
Author, co-author :
Leclercq, Victoria ; Université de Liège - ULiège > Département des sciences de la santé publique > Epidémiologie clinique
Beaudart, Charlotte ; Université de Liège - ULiège > Département des sciences de la motricité > Département des sciences de la motricité
Ajamieh, Sara ; Université de Liège - ULiège > Département de Psychologie > Neuroscience comportementale et psychopharmacologie expér.
Tirelli, Ezio ; Université de Liège - ULiège > Département de Psychologie > Neuroscience comportementale et psychopharmacologie expér.
Bruyère, Olivier ; Université de Liège - ULiège > Département des sciences de la santé publique > Santé publique, Epidémiologie et Economie de la santé
Language :
English
Title :
The methodological quality of meta-analyses indexed in PsycINFO: leads for enhancements – a meta-epidemiological study
Gurevitch J, Koricheva J, Nakagawa S, et al. Meta-Analysis and the science of research synthesis. Nature 2018;555:175-82.
Page MJ, Altman DG, Shamseer L, et al. Reproducible research practices are underused in systematic reviews of biomedical interventions. J Clin Epidemiol 2018;94:8-18.
Dieckmann NF, Malle BF, Bodner TE. An empirical assessment of meta-analytic practice. Review of General Psychology 2009;13:101-15.
Zhang H, Han J, Zhu Y-B, et al. Reporting and methodological qualities of published surgical meta-analyses. J Clin Epidemiol 2016;70:4-16.
Ioannidis JPA. The mass production of redundant, misleading, and Conflicted systematic reviews and meta-analyses. Milbank Q 2016;94:485-514.
Nuijten MB. Practical tools and strategies for researchers to increase replicability. Dev Med Child Neurol 2019;61:535-9.
Page MJ, Shamseer L, Tricco AC. Registration of systematic reviews in prospero: 30,000 records and counting. Syst Rev 2018;7:32.
Sideri S, Papageorgiou SN, Eliades T. Registration in the International prospective register of systematic reviews (prospero) of systematic review protocols was associated with increased review quality. J Clin Epidemiol 2018;100:103-10.
Franco JVA, Garrote VL, Escobar Liquitay CM, et al. Identification of problems in search strategies in Cochrane reviews. Res Synth Methods 2018;9:408-16.
Ahn S, Ames AJ, Myers ND. A review of meta-analyses in education: methodological strengths and weaknesses. Rev Educ Res 2012;82:436-76.
Dechartres A, Atal I, Riveros C, et al. Association between publication characteristics and treatment effect estimates: a Meta-epidemiologic study. Ann Intern Med 2018;169:385.
Walpole SC. Including papers in languages other than English in systematic reviews: important, feasible, yet often omitted. J Clin Epidemiol 2019;111:127-34.
Jackson D, Turner R. Power analysis for random-effects meta-analysis. Res Synth Methods 2017.
Page MJ, Altman DG, McKenzie JE, et al. Flaws in the application and interpretation of statistical analyses in systematic reviews of therapeutic interventions were common: a cross-sectional analysis. J Clin Epidemiol 2018;95:7-18.
Stanley TD, Carter EC, Doucouliagos H. What meta-analyses reveal about the replicability of psychological research. Psychol Bull 2018;144:1325-46.
Cafri G, Kromrey JD, Brannick MT. A Meta-Meta-Analysis: empirical review of statistical power, type I error rates, effect sizes, and model selection of meta-analyses published in psychology. Multivariate Behav Res 2010;45:239-70.
Oliveras I, Losilla J-M, Vives J. Methodological quality is underrated in systematic reviews and meta-analyses in health psychology. J Clin Epidemiol 2017;86:59-70.
Johnson BT, Low RE, MacDonald HV. Panning for the gold in health research: incorporating studies' methodological quality in meta-analysis. Psychol Health 2015;30:135-52.
Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med 2009;6:e1000100.
Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017;358:j4008.
Leclercq V, Beaudart C, Ajamieh S, et al. Meta-Analyses indexed in PsycINFO had a better completeness of reporting when they mention PRISMA. J Clin Epidemiol 2019;115:46-54.
Wongpakaran N, Wongpakaran T, Wedding D, et al. A comparison of Cohen's Kappa and Gwet's AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Med Res Methodol 2013;13:1-7.
Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007;7:10.
Stroup DF. Meta-Analysis of observational studies in Epidemiology.A proposal for reporting. JAMA 2008;283.
Coxeter P, Del Mar CB, McGregor L, et al. Interventions to facilitate shared decision making to address antibiotic use for acute respiratory infections in primary care. Cochrane Database Syst Rev 2015;11:CD010907.
Cortese S, Ferrin M, Brandeis D, et al. Neurofeedback for attention-deficit/ hyperactivity disorder: meta-analysis of clinical and neuropsychological outcomes from randomized controlled trials. J Am Acad Child Adolesc Psychiatry 2016;55:444-55.
Cortese S, Moreira-Maia CR, St Fleur D, et al. Association between ADHD and obesity: a systematic review and meta-analysis. Am J Psychiatry 2016;173:34-43.
Aytug ZG, Rothstein HR, Zhou W, et al. Revealed or concealed? transparency of procedures, decisions, and judgment calls in meta-analyses. Organ Res Methods 2012;15:103-33.
Allers K, Hoffmann F, Mathes T, et al. Systematic reviews with published protocols compared to those without: more effort, older search. J Clin Epidemiol 2018;95:102-10.
Yaylali IE, Alacam T, AlaA§am T. Critical Assessment of Search Strategies in Systematic Reviews in Endodontics. J Endod 2016;42:854-60.
Hohn RE, Slaney KL, Tafreshi D. Primary study quality in psychological meta-analyses: an empirical assessment of recent practice. Front Psychol 2018;9:2667.
Ahn S, Becker BJ. Incorporating quality scores in meta-analysis. Journal of Educational and Behavioral Statistics 2011;36:555-85.
Schalken N, Rietbergen C. The reporting quality of systematic reviews and meta-analyses in industrial and organizational psychology: a systematic review. Front Psychol 2017;8:1395.
Hutton B, Wolfe D, Moher D, et al. Reporting guidance considerations from a statistical perspective: overview of tools to enhance the rigour of reporting of randomised trials and systematic reviews. Evid Based Ment Health 2017;20:46-52.
Robson RC, Pham Ba', Hwee J, et al. Few studies exist examining methods for selecting studies, abstracting data, and appraising quality in a systematic review. J Clin Epidemiol 2019;106:121-35.
Augusteijn HEM, van Aert RCM, van Assen MALM. The effect of publication bias on the Q test and assessment of heterogeneity. Psychol Methods 2019;24:116-34.