[en] Radiomics-the high throughput extraction of quantitative features from medical images and their correlation with clinical and biological endpoints- is the subject of active and extensive research. Although the field shows promise, the generalizability of radiomic signatures is affected significantly by differences in scan acquisition and reconstruction settings. Previous studies reported on the sensitivity of radiomic features (RFs) to test-retest variability, inter-observer segmentation variability, and intra-scanner variability. A framework involving robust radiomics analysis and the application of a post-reconstruction feature harmonization method using ComBat was recently proposed to address these challenges. In this study, we investigated the reproducibility of RFs across different scanners and scanning parameters using this framework. We analysed thirteen scans of a ten-layer phantom that were acquired differently. Each layer was subdivided into sixteen regions of interest (ROIs), and the scans were compared in a pairwise manner, resulting in seventy-eight different scenarios. Ninety-one RFs were extracted from each ROI. As hypothesized, we demonstrate that the reproducibility of a given RF is not a constant but is dependent on the heterogeneity found in the data under analysis. The number (%) of reproducible RFs varied across the pairwise scenarios investigated, having a wide range between 8 (8.8%) and 78 (85.7%) RFs. Furthermore, in contrast to what has been previously reported, and as hypothesized in the robust radiomics analysis framework, our results demonstrate that ComBat cannot be applied to all RFs but rather on a percentage of those-the "ComBatable" RFs-which differed depending on the data being harmonized. The number (%) of reproducible RFs following ComBat harmonization varied across the pairwise scenarios investigated, ranging from 14 (15.4%) to 80 (87.9%) RFs, and was found to depend on the heterogeneity in the data. We conclude that the standardization of image acquisition protocols remains the cornerstone for improving the reproducibility of RFs, and the generalizability of the signatures developed. Our proposed approach helps identify the reproducible RFs across different datasets.
Disciplines :
Radiology, nuclear medicine & imaging
Author, co-author :
Ibrahim, Abdalla ; The D-Lab, Department of Precision Medicine, GROW- School for Oncology, Maastricht University, Maastricht, The Netherlands ; Department of Radiology and Nuclear Medicine, Maastricht University Medical Centre+, Maastricht, The Netherlands ; Division of Nuclear Medicine and Oncological Imaging, Department of Medical Physics, University Hospital of Liège and GIGA CRC-in vivo imaging, University of Liège, Liege, Belgium ; Department of Nuclear Medicine and Comprehensive Diagnostic Centre Aachen (CDCA), University Hospital RWTH Aachen University, Aachen, Germany
Refaee, Turkey; The D-Lab, Department of Precision Medicine, GROW- School for Oncology, Maastricht University, Maastricht, The Netherlands ; Faculty of Applied Medical Sciences, Department of Diagnostic Radiology, Jazan University, Jazan, Saudi Arabia
Leijenaar, Ralph T H; Oncoradiomics SA, Liege, Belgium
Primakov, Sergey; The D-Lab, Department of Precision Medicine, GROW- School for Oncology, Maastricht University, Maastricht, The Netherlands ; Department of Nuclear Medicine and Comprehensive Diagnostic Centre Aachen (CDCA), University Hospital RWTH Aachen University, Aachen, Germany
HUSTINX, Roland ; Centre Hospitalier Universitaire de Liège - CHU > > Service médical de médecine nucléaire et imagerie onco
Mottaghy, Felix M; Department of Radiology and Nuclear Medicine, Maastricht University Medical Centre+, Maastricht, The Netherlands ; Department of Nuclear Medicine and Comprehensive Diagnostic Centre Aachen (CDCA), University Hospital RWTH Aachen University, Aachen, Germany
Woodruff, Henry C; The D-Lab, Department of Precision Medicine, GROW- School for Oncology, Maastricht University, Maastricht, The Netherlands ; Department of Radiology and Nuclear Medicine, Maastricht University Medical Centre+, Maastricht, The Netherlands
Maidment, Andrew D A; Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States of America
Lambin, Philippe ; Université de Liège - ULiège > Département des sciences cliniques ; The D-Lab, Department of Precision Medicine, GROW- School for Oncology, Maastricht University, Maastricht, The Netherlands ; Department of Radiology and Nuclear Medicine, Maastricht University Medical Centre+, Maastricht, The Netherlands
Language :
English
Title :
The application of a workflow integrating the variable reproducibility and harmonizability of radiomic features on a phantom dataset.
ERC - European Research Council EU - European Union Interreg EMR - Interreg Euregio Meuse-Rhin KWF - Kankerbestrijding UM - Universiteit Maastricht
Funding text :
Authors acknowledge financial support from ERC advanced grant (ERC-ADG-2015 n˚694812 - Hypoximmuno), ERC-2020-PoC: 957565-AUTO.DISTINCT. Authors also acknowledge financial support from SME Phase 2 (RAIL n˚673780), EUROSTARS (DART, DECIDE), the European Union’s Horizon 2020 research and innovation programme under grant agreement:
ImmunoSABR n˚ 733008, MSCA-ITN-PREDICT n˚766276, CHAIMELEON n˚ 952172, EuCanImage n˚952103, TRANSCAN Joint Transnational Call 2016
(JTC2016 CLEARLY n˚ UM 2017-8295) and Interreg V-A Euregio Meuse-Rhine (EURADIOMICS˚ EMR4). Authors further acknowledge financial support by the Dutch Cancer Society (KWF Kankerbestrijding), Project number 12085/2018-2, and Maastricht-Liege Imaging
Walsh S, de Jong EE, van Timmeren JE, Ibrahim A, Compter I, Peerlings J, et al. Decision support systems in oncology. JCO clinical cancer informatics. 2019; 3:1-9.
Aerts HJ, Velazquez ER, Leijenaar RT, Parmar C, Grossmann P, Carvalho S, et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nature communications. 2014; 5:4006. https://doi.org/10.1038/ncomms5006 PMID: 24892406.
Lambin P, Rios-Velazquez E, Leijenaar R, Carvalho S, Van Stiphout RG, Granton P, et al. Radiomics: Extracting more information from medical images using advanced feature analysis. European journal of cancer. 2012; 48(4):441-6. https://doi.org/10.1016/j.ejca.2011.11.036 PMID: 22257792.
Gillies RJ, Kinahan PE, Hricak H. Radiomics: Images are more than pictures, they are data. Radiology. 2015; 278(2):563-77. https://doi.org/10.1148/radiol.2015151169 PMID: 26579733.
Reiazi R, Abbas E, Faima P, Kwan JY, Rezaie A, Bratman SV, et al. The Impact of the Variation of Imaging Factors on the Robustness of Computed Tomography Radiomic Features: A Review. medRxiv. 2020.
van Timmeren JE, Carvalho S, Leijenaar RT, Troost EG, van Elmpt W, de Ruysscher D, et al. Challenges and caveats of a multi-center retrospective radiomics study: An example of early treatment response assessment for NSCLC patients using FDG-PET/CT radiomics. PloS one. 2019; 14(6): E0217536. https://doi.org/10.1371/journal.pone.0217536 PMID: 31158263.
Traverso A, Wee L, Dekker A, Gillies R. Repeatability and reproducibility of radiomic features: A systematic review. International Journal of Radiation Oncology∗ Biology∗ Physics. 2018; 102(4):1143-58. https://doi.org/10.1016/j.ijrobp.2018.05.053 PMID: 30170872.
van Timmeren JE, Leijenaar RT, van Elmpt W, Wang J, Zhang Z, Dekker A, et al. Test-retest data for radiomics feature stability analysis: Generalizable or study-specific? Tomography. 2016; 2(4):361. https://doi.org/10.18383/j.tom.2016.00208 PMID: 30042967.
Prayer F, Hofmanninger J, Weber M, Kifjak D, Willenpart A, Pan J, et al. Variability of computed tomography radiomics features of fibrosing interstitial lung disease: A test-retest study. Methods. 2020. https://doi.org/10.1016/j.ymeth.2020.08.007 PMID: 32891727.
Zhovannik I, Bussink J, Traverso A, Shi Z, Kalendralis P, Wee L, et al. Learning from scanners: Bias reduction and feature correction in radiomics. Clinical and translational radiation oncology. 2019; 19:33-8. https://doi.org/10.1016/j.ctro.2019.07.003 PMID: 31417963.
Mackin D, Fave X, Zhang L, Fried D, Yang J, Taylor B, et al. Measuring CT scanner variability of radiomics features. Investigative radiology. 2015; 50(11):757. https://doi.org/10.1097/RLI. 0000000000000180 PMID: 26115366.
Zwanenburg A, Leger S, Vallières M, Löck S. Image biomarker standardisation initiative. arXiv preprint arXiv:161207003. 2016.
Lambin P, Leijenaar RT, Deist TM, Peerlings J, De Jong EE, Van Timmeren J, et al. Radiomics: The bridge between medical imaging and personalized medicine. Nature reviews Clinical oncology. 2017; 14(12):749. https://doi.org/10.1038/nrclinonc.2017.141 PMID: 28975929.
Johnson WE, Li C, Rabinovic A. Adjusting batch effects in microarray expression data using empirical Bayes methods. Biostatistics. 2007; 8(1):118-27. https://doi.org/10.1093/biostatistics/kxj037 PMID: 16632515.
Ligero M, Jordi-Ollero O, Bernatowicz K, Garcia-Ruiz A, Delgado-Munõz E, Leiva D, et al. Minimizing acquisition-related radiomics variability by image resampling and batch effect correction to allow for large-scale data analysis. European radiology. 2021; 31(3):1460-70. https://doi.org/10.1007/s00330-020-07174-0 PMID: 32909055.
Foy JJ, Al-Hallaq HA, Grekoski V, Tran T, Guruvadoo K, Armato Iii SG, et al. Harmonization of radiomic feature variability resulting from differences in CT image acquisition and reconstruction: Assessment in a cadaveric liver. Physics in Medicine & Biology. 2020; 65(20):205008.
Orlhac F, Boughdad S, Philippe C, Stalla-Bourdillon H, Nioche C, Champion L, et al. A postreconstruction harmonization method for multicenter radiomic studies in PET. Journal of Nuclear Medicine. 2018; 59(8):1321-8. https://doi.org/10.2967/jnumed.117.199935 PMID: 29301932.
Fortin J-P, Cullen N, Sheline YI, Taylor WD, Aselcioglu I, Cook PA, et al. Harmonization of cortical thickness measurements across scanners and sites. NeuroImage. 2018; 167:104-20. https://doi.org/10. 1016/j.neuroimage.2017.11.024 PMID: 29155184.
Orlhac F, Humbert O, Boughdad S, Lasserre M, Soussan M, Nioche C, et al. Validation of a harmonization method to correct for SUV and radiomic features variability in multi-center studies. Journal of Nuclear Medicine. 2018; 59(supplement 1):288-.
Orlhac F, Frouin F, Nioche C, Ayache N, Buvat I. Validation of a method to compensate multicenter effects affecting CT Radiomics. Radiology. 2019:182023. https://doi.org/10.1148/radiol.2019182023 PMID: 30694160.
Ibrahim A, Primakov S, Beuque M, Woodruff H, Halilaj I, Wu G, et al. Radiomics for precision medicine: Current challenges, future prospects, and the proposal of a new framework. Methods. 2020.
Mackin D, Fave X, Zhang L, Fried D, Yang J, Taylor B, et al. Data From Credence Cartridge Radiomics Phantom CT Scans. The Cancer Imaging Archive. 2017. http://doi.org/10.7937/K9/TCIA.2017. zuzrml5b.
Clark K, Vendt B, Smith K, Freymann J, Kirby J, Koppel P, et al. The Cancer Imaging Archive (TCIA): Maintaining and operating a public information repository. Journal of digital imaging. 2013; 26(6):1045-57. https://doi.org/10.1007/s10278-013-9622-7 PMID: 23884657.
Stevenson M, Nunes T, Sanchez J, Thornton R, Reiczigel J, Robison-Cox J, et al. epiR: An R package for the analysis of epidemiological data. R package version 09-43. 2013.
Team RC. R: A language and environment for statistical computing. 2013.
Team R. RStudio: Integrated Development for R. Boston: RStudio Inc.; 2015. 2016.
Lawrence I, Lin K. A concordance correlation coefficient to evaluate reproducibility. Biometrics. 1989:255-68. PMID: 2720055.
McBride G. A proposal for strength-of-agreement criteria for Lin's concordance correlation coefficient. NIWA Client Report: HAM2005-062. 2005.
Leek JT, Johnson WE, Parker HS, Jaffe AE, Storey JD. The sva package for removing batch effects and other unwanted variation in high-throughput experiments. Bioinformatics. 2012; 28(6):882-3. https://doi.org/10.1093/bioinformatics/bts034 PMID: 22257669.
Kim H, Park CM, Lee M, Park SJ, Song YS, Lee JH, et al. Impact of reconstruction algorithms on CT radiomic features of pulmonary tumors: Analysis of intra-and inter-reader variability and inter-reconstruction algorithm variability. PLoS One. 2016; 11(10):e0164924. https://doi.org/10.1371/journal.pone. 0164924 PMID: 27741289.
Lu L, Ehmke RC, Schwartz LH, Zhao B. Assessing agreement between radiomic features computed for multiple CT imaging settings. PloS one. 2016; 11(12):e0166550. https://doi.org/10.1371/journal.pone. 0166550 PMID: 28033372.
Vetter TR, Schober P. Agreement analysis: What he said, she said versus you said. Anesthesia & Analgesia. 2018; 126(6):2123-8. https://doi.org/10.1213/ANE.0000000000002924 PMID: 29677066.
Leijenaar RT, Carvalho S, Velazquez ER, Van Elmpt WJ, Parmar C, Hoekstra OS, et al. Stability of FDG-PET radiomics features: An integrated analysis of test-retest and inter-observer variability. Acta oncologica. 2013; 52(7):1391-7. https://doi.org/10.3109/0284186X.2013.812798 PMID: 24047337.
Da-ano R, Masson I, Lucia F, Doré M, Robin P, Alfieri J, et al. Performance comparison of modified ComBat for harmonization of radiomic features for multicenter studies. Scientific Reports. 2020; 10 (1):10248. https://doi.org/10.1038/s41598-020-66110-w PMID: 32581221.
Andrearczyk V, Depeursinge A, Mü ller H. Neural network training for cross-protocol radiomic feature standardization in computed tomography. Journal of Medical Imaging. 2019; 6(2):024008. https://doi. org/10.1117/1.JMI.6.2.024008 PMID: 31205978.
Marcadent S, Hofmeister J, Preti MG, Martin SP, Van De Ville D, Montet X. Generative Adversarial Networks Improve the Reproducibility and Discriminative Power of Radiomic Features. Radiology: Artificial Intelligence. 2020; 2(3):e190035.
