CT; Generative modelling; Interpolation; Medical imaging; PixelCNN; Reproducibility of Results; Signal-To-Noise Ratio; Tomography, X-Ray Computed/methods; Image Processing, Computer-Assisted/methods; Accurate prediction; Analysis models; Computed tomography; Generative model; Imaging task; Objects detection; Objects segmentation; Quantitative image analysis; Root mean squared errors; Image Processing, Computer-Assisted; Tomography, X-Ray Computed; Health Informatics; Computer Science Applications
Abstract :
[en] Quantitative image analysis models are used for medical imaging tasks such as registration, classification, object detection, and segmentation. For these models to be capable of making accurate predictions, they need valid and precise information. We propose PixelMiner, a convolution-based deep-learning model for interpolating computed tomography (CT) imaging slices. PixelMiner was designed to produce texture-accurate slice interpolations by trading off pixel accuracy for texture accuracy. PixelMiner was trained on a dataset of 7829 CT scans and validated using an external dataset. We demonstrated the model's effectiveness by using the structural similarity index (SSIM), peak signal to noise ratio (PSNR), and the root mean squared error (RMSE) of extracted texture features. Additionally, we developed and used a new metric, the mean squared mapped feature error (MSMFE). The performance of PixelMiner was compared to four other interpolation methods: (tri-)linear, (tri-)cubic, windowed sinc (WS), and nearest neighbor (NN). PixelMiner produced texture with a significantly lowest average texture error compared to all other methods with a normalized root mean squared error (NRMSE) of 0.11 (p < .01), and the significantly highest reproducibility with a concordance correlation coefficient (CCC) ≥ 0.85 (p < .01). PixelMiner was not only shown to better preserve features but was also validated using an ablation study by removing auto-regression from the model and was shown to improve segmentations on interpolated slices.
Disciplines :
Computer science
Author, co-author :
Rogers, W ; The D-Lab, Department of Precision Medicine, GROW - School for Oncology and Developmental Biology, Maastricht University, Maastricht, the Netherlands
Keek, S A; The D-Lab, Department of Precision Medicine, GROW - School for Oncology and Developmental Biology, Maastricht University, Maastricht, the Netherlands
Beuque, M ; The D-Lab, Department of Precision Medicine, GROW - School for Oncology and Developmental Biology, Maastricht University, Maastricht, the Netherlands
Primakov, S; The D-Lab, Department of Precision Medicine, GROW - School for Oncology and Developmental Biology, Maastricht University, Maastricht, the Netherlands
Wu, G; The D-Lab, Department of Precision Medicine, GROW - School for Oncology and Developmental Biology, Maastricht University, Maastricht, the Netherlands
Yan, C; The D-Lab, Department of Precision Medicine, GROW - School for Oncology and Developmental Biology, Maastricht University, Maastricht, the Netherlands
Sanduleanu, S; The D-Lab, Department of Precision Medicine, GROW - School for Oncology and Developmental Biology, Maastricht University, Maastricht, the Netherlands
Gietema, H A; Department of Radiology and Nuclear Medicine, GROW - School for Oncology and Developmental Biology, Maastricht University Medical Centre+, Maastricht, the Netherlands
Casale, R; The D-Lab, Department of Precision Medicine, GROW - School for Oncology and Developmental Biology, Maastricht University, Maastricht, the Netherlands, Department of Radiology, Institut Jules Bordet, Université Libre de Bruxelles, Brussels, Belgium
Occhipinti, M; Radiomics, Clos Chanmurly 13, 4000, Liege, Belgium
Woodruff, H C ; The D-Lab, Department of Precision Medicine, GROW - School for Oncology and Developmental Biology, Maastricht University, Maastricht, the Netherlands, Department of Radiology and Nuclear Medicine, GROW - School for Oncology and Developmental Biology, Maastricht University Medical Centre+, Maastricht, the Netherlands
Jochems, A; The D-Lab, Department of Precision Medicine, GROW - School for Oncology and Developmental Biology, Maastricht University, Maastricht, the Netherlands
Lambin, P ; The D-Lab, Department of Precision Medicine, GROW - School for Oncology and Developmental Biology, Maastricht University, Maastricht, the Netherlands, Department of Radiology and Nuclear Medicine, GROW - School for Oncology and Developmental Biology, Maastricht University Medical Centre+, Maastricht, the Netherlands. Electronic address: philippe.lambin@maastrichtuniversity.nl
Authors acknowledge financial support from ERC advanced grant ( ERC-ADG-2015 n° 694812 Hypoximmuno), ERC-2020-PoC: 957565-AUTO.DISTINCT , Authors also acknowledge financial support from EUROSTARS (DART, DECIDE), the European Union's Horizon 2020 research and innovation programme under grant agreement: ImmunoSABR n° 733008 , MSCA-ITN-PREDICT n° 766276 , CHAIMELEON n° 952172 , EuCanImage n° 952103 , TRANSCAN Joint Transnational Call 2016 ( JTC2016 CLEARLY n° UM 2017–8295 ) and Interreg V-A Euregio Meuse-Rhine (EURADIOMICS n° EMR4). This work was supported by the Dutch Cancer Society (KWF Kankerbestrijding) , Project number 12085/2018–2 .
Aerts, H.J.W.L., et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat. Commun., 5, 2014.
Kurland, B.F., et al. Promise and pitfalls of quantitative imaging in oncology clinical trials. Magn. Reson. Imag. 30:9 (2012), 1301–1312.
Buckler, A., et al. A collaborative enterprise for multi-stakeholder participation in the advancement of quantitative imaging. Radiology 258 (2011), 906–914.
Buckler, A., et al. Quantitative imaging test Approval and biomarker qualification: Interrelated but distinct activities. Radiology 259 (2011), 875–884.
Lambin, P., et al. Predicting outcomes in radiation oncology-multifactorial decision support systems. Nat. Rev. Clin. Oncol., 10, 2012.
Goldman, L.W., Principles of CT: radiation dose and image quality. J. Nucl. Med. Technol. 35 (2007), 213–225.
Zhao, B., James, L.P., Moskowitz, C.S., Guo, P., Ginsberg, M.S., Lefkowitz, R.A., Qin, Y., Riely, G.J., Kris, M.G., Schwartz, L.H., Evaluating Variability In Tumor Measurements From Same-Day Repeat CT Scans Of Patients With Non–small Cell Lung Cancer 1. Radiology. 2009, Radiological Society of North America (RSNA).
Choe, J.A.-O., et al., Deep Learning-Based Image Conversion Of CT Reconstruction Kernels Improves Radiomics Reproducibility For Pulmonary Nodules Or Masses, (1527-1315 (Electronic)).
Mackin, D., et al. Correction: harmonizing the pixel size in retrospective computed tomography radiomics studies. PLoS One, 13, 2018, e0191597 e0191597.
Whybra, P., et al. Assessing Radiomic Feature Robustness to Interpolation in 18F-FDG PET Imaging, 9, 2019, Scientific Reports.
Leng, J., Xu, G., Zhang, Y., Medical image interpolation based on multi-resolution registration. Comput. Math. Appl. 66:1 (2013), 1–18.
Josien, P.W.P., Maintz, J.B.A., Max, A.V., Mutual information matching and interpolation artifacts. Proc. SPIE, 1999.
Parker, J.A., Kenyon, R.V., Troxel, D.E., Comparison of interpolating methods for image resampling. IEEE Trans. Med. Imag. 2:1 (1983), 31–39.
Crow, F.C., The aliasing problem in computer-generated shaded images. Commun. ACM 20:11 (1977), 799–805.
Strand, J., Taxt, T., Local frequency features for texture classification. Pattern Recogn. 27:10 (1994), 1397–1406.
van den Oord, A., Kalchbrenner, N., Kavukcuoglu, K., Pixel Recurrent Neural Networks. 2016, CoRR abs/1601.06759.
van den Oord, A., et al. Conditional Image Generation with PixelCNN Decoders. 2016, CoRR abs/1606.05328.
Salimans, T., et al. PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications. 2017, International Conference on Learning Representation.
Chen, X., et al. PixelSNAIL: an Improved Autoregressive Generative Model. 2017.
Dahl, R., Norouzi, M., Shlens, J., Pixel Recursive Super Resolution. 2017.
Mohamed, S., Lakshminarayanan, B., Learning in Implicit Generative Models. 2017.
Goodfellow, I., et al. Generative adversarial nets. Advances in Neural Information Processing Systems, 2014, Curran Associates, Inc.
Diederik, P.K., Max, W., An Introduction to Variational Autoencoders. 2019 now. 1.
Colak, E., et al. The RSNA pulmonary embolism CT dataset. Radiol. Artif. Intell., 3(2), 2021, e200254 e200254.
Henschke Ci, M.D., Yankelevitz, D.F., et al. Early lung cancer action project. Lancet, 354, 1999, 1205.
Armato, S.G. 3rd, McLennan, G., Bidaut, L., McNitt-Gray, M.F., Meyer, C.R., Reeves, A.P., Zhao, B., Aberle, D.R., Henschke, C.I., Hoffman, E.A., Kazerooni, E.A., MacMahon, H., Van Beeke, E.J., Yankelevitz, D., Biancardi, A.M., Bland, P.H., Brown, M.S., Engelmann, R.M., Laderach, G.E., Max, D., Pais, R.C., Qing, D.P., Roberts, R.Y., Smith, A.R., Starkey, A., Batrah, P., Caligiuri, P., Farooqi, A., Gladish, G.W., Jude, C.M., Munden, R.F., Petkovska, I., Quint, L.E., Schwartz, L.H., Sundaram, B., Dodd, L.E., Fenimore, C., Gur, D., Petrick, N., Freymann, J., Kirby, J., Hughes, B., Casteele, A.V., Gupte, S., Sallamm, M., Heath, M.D., Kuhn, M.H., Dharaiya, E., Burns, R., Fryd, D.S., Salganicoff, M., Anand, V., Shreter, U., Vastagh, S., Croft, B.Y., The lung image database Consortium (LIDC) and image database resource initiative (IDRI): a completed reference database of lung nodules on CT scans. Med. Phys. 38:2 (2011 Feb), 915–931, 10.1118/1.3528204.PMID:21452728 PMCID: PMC3041807.
Lehmann, E.L., Casella, G., Theory of Point Estimation. 2nd Ed. Springer Texts in Statistics. 1998, Springer, New York, 589 xxvi.
Myers, D.E., Kriging, cokriging, radial basis functions and the role of positive definiteness. Comput. Math. Appl. 24:12 (1992), 139–148.
Stytz, M.R., Parrott, R.W., Using kriging for 3d medical imaging. Comput. Med. Imag. Graph. 17:6 (1993), 421–442.
Keys, R., Cubic convolution interpolation for digital image processing. IEEE trans acoust speech signal process. Acoustics, speech and signal processing. IEEE Trans. 29 (1982), 1153–1160.
Hou, H., Andrews, H., Cubic splines for image interpolation and digital filtering. IEEE Trans. Acoust. Speech Signal Process. 26:6 (1978), 508–517.
Harris, F.J., On the use of windows for harmonic analysis with the discrete Fourier transform. Proc. IEEE 66:1 (1978), 51–83.
Lehmann, T.M., Gonner, C., Spitzer, K., Survey: interpolation methods in medical image processing. IEEE Trans. Med. Imag. 18:11 (1999), 1049–1075.
Whybra, P., et al. Assessing radiomic feature robustness to interpolation in 18F-FDG PET imaging. Sci. Rep., 9(1), 2019, 9649.
Johnson, H.J.a.M., Matthew, M., Ibanez, Luis, The ITK Software Guide Book 1: Introduction and Development Guidelines-Volume 1. 2015, Kitware, Inc.
van Griethuysen, J.J.M., et al. Computational radiomics system to decode the radiographic phenotype. Cancer Res. 77:21 (2017), e104–e107.
van der Walt S, S.J., Nunez-Iglesias, J., Boulogne, F., Warner, J.D., Yager, N., Gouillart, E., Yu, T., The scikit-image contributors, scikit-image: image processing in Python. PeerJ, 2, 2014, E453.
Hofmanninger, J., Prayer, F., Pan, J., Röhrich, S., Prosch, H., Langs, G., Automatic lung segmentation in routine imaging is primarily a data diversity problem, not a methodology problem. Eur. Radiol. Exp., 4(1), 2020 Aug 20, 50, 10.1186/s41747-020-00173-2 PMID: 32814998; PMCID: PMC7438418.
Ledig, C., et al. Photo-Realistic Single Image Super-resolution Using a Generative Adversarial Network. 2017.
Jian, X., et al. Image superresolution by midfrequency sparse representation and total variation regularization. J. Electron. Imag. 24:1 (2015), 1–29.
Rajan, D., Chaudhuri, S., Generalized Interpolation for Super-resolution. 2006, 45–72.
Lai, W., et al. Deep laplacian pyramid networks for fast and accurate super-resolution. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Zhang, X., et al. CT super-resolution using multiple dense residual block based GAN. Signals Image Video Process 15:4 (2021), 725–733.
Youssef, S., Pierre-Marc, J., Alain, L., GANs for Medical Image Synthesis: an Empirical Study. 2021 ArXiv, 2021. abs/2105.05318.
Howard, A.G., et al. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. ArXiv. 2017 abs/1704.04861.
Tan, M., Le, Q., EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. 2019.
Vaswani, A., et al. Attention Is All You Need. 2017.
Chen, M., et al. Generative Pretraining from Pixels. 2020.
Horé, A., Ziou, D., Image Quality Metrics: PSNR vs. SSIM. In 2010 20th International Conference On Pattern Recognition. 2010.
