[en] Image analysis is key to extracting quantitative information from scientific microscopy images, but the methods involved are now often so refined that they can no longer be unambiguously described by written protocols. We introduce BIAFLOWS, an open-source web tool enabling to reproducibly deploy and benchmark bioimage analysis workflows coming from any software ecosystem. A curated instance of BIAFLOWS populated with 34 image analysis workflows and 15 microscopy image datasets recapitulating common bioimage analysis problems is available online. The workflows can be launched and assessed remotely by comparing their performance visually and according to standard benchmark metrics. We illustrated these features by comparing seven nuclei segmentation workflows, including deep-learning methods. BIAFLOWS enables to benchmark and share bioimage analysis workflows, hence safeguarding research results and promoting high-quality standards in image analysis. The platform is thoroughly documented and ready to gather annotated microscopy datasets and workflows contributed by the bioimaging community.
Disciplines :
Computer science
Author, co-author :
Rubens, Ulysse ✱; Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Télécommunications
Mormont, Romain ✱; Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Algorith. des syst. en interaction avec le monde physique
Paavolainen, Lassi; University of Helsinki > HiLIFE > Institute for Molecular Medicine Finland (FIMM)
Ellenberg, J., Swedlow, J.R., Barlow, M., Cook, C.E., Sarkans, U., Patwardhan, A., Brazma, A., Birney, E., A call for public archives for biological image data. Nat. Methods 15 (2018), 849–854.
Allan, C., Burel, J.M., Moore, J., Blackburn, C., Linkert, M., Loynton, S., Macdonald, D., Moore, W.J., Neves, C., Patterson, A., et al. OMERO: flexible, model-driven data management for experimental biology. Nat. Methods 9 (2012), 245–253.
Kvilekval, K., Fedorov, D., Obara, B., Singh, A., Manjunath, B.S., Bisque: a platform for bioimage analysis and management. Bioinformatics 26 (2010), 544–552.
Williams, E., Moore, J., Li, S.W., Rustici, G., Tarkowska, A., Chessel, A., Leo, S., Antal, B., Ferguson, R.K., Sarkans, U., et al. Image Data Resource: a bioimage data integration and publication platform. Nat. Methods 14 (2017), 775–781.
Vandewalle, P., Code sharing is associated with research impact in image processing. Comput. Sci. Eng. 14 (2012), 42–47.
Maier-Hein, L., Eisenmann, M., Reinke, A., Onogur, S., Stankovic, M., Scholz, P., Arbel, T., Bogunovic, H., Bradley, A.P., Carass, A., et al. Why rankings of biomedical image analysis competitions should be interpreted with care. Nat. Commun., 9, 2018, 5217.
Meijering, E., Carpenter, A., Peng, H., Hamprecht, F.A., Olivo-Marin, J.C., Imagining the future of bioimage analysis. Nat. Biotechnol. 34 (2016), 1250–1255.
Perkel, J.M., A toolkit for data transparency takes shape. Nature 560 (2018), 513–515.
Grüning, B.A., Rasche, E., Rebolledo-Jaramillo, B., Eberhard, C., Houwaart, T., Chilton, J., Coraor, N., Backofen, R., Taylor, J., Nekrutenkoet, A., Jupyter and Galaxy: easing entry barriers into complex data analyses for biomedical researchers. PLoS Comput. Biol., 13, 2017, e1005425.
Marée, R., Rollus, L., Stévens, B., Hoyoux, R., Louppe, G., Vandaele, R., Begon, J.M., Kainz, P., Geurts, P., Wehenkel, L., Collaborative analysis of multi-gigapixel imaging data with Cytomine. Bioinformatics 32 (2016), 1395–1401.
Glatard, T., Kiar, G., Aumentado-Armstrong, T., Beck, N., Bellec, P., Bernard, R., Bonnet, A., Brown, S.T., Camarasu-Pop, S., Cervenansky, F., et al. Boutiques: a flexible framework to integrate command-line applications in computing platforms. GigaScience, 7, 2018, giy016.
Kurtzer, G.M., Sochat, V., Bauer, M.W., Singularity: scientific containers for mobility of compute. PLoS One, 12, 2017, e0177459.
Yoo, A., Jette, M., Grondona, M., SLURM: simple Linux utility for resource management, job scheduling strategies for parallel processing. Lect. Notes Comput. Sci. 2862 (2003), 44–60.
Kozubek, M., Challenges and benchmarks in bioimage analysis. Adv. Anat. Embryol. Cell Biol. 219 (2016), 231–262.
Brown, K.M., Barrionuevo, G., Canty, A.J., De Paola, V., Hirsch, J.A., Jefferis, G.S., Lu, J., Snippe, M., Sugihara, I., Ascoli, G.A., The DIADEM data sets: representative light microscopy images of neuronal morphology to advance automation of digital reconstructions. Neuroinformatics 9 (2011), 143–157.
Ulman, V., Maška, M., Magnusson, K.E.G., Ronneberger, O., Haubold, C., Harder, N., Matula, P., Matula, P., Svoboda, D., Radojevic, M., et al. An objective comparison of cell-tracking algorithms. Nat. Methods 14 (2017), 1141–1152.
Chenouard, N., Smal, I., de Chaumont, F., Maška, M., Sbalzarini, I.F., Gong, Y., Cardinale, J., Carthel, C., Coraluppi, S., Winter, M., et al. Objective comparison of particle tracking methods. Nat. Methods 11 (2014), 281–289.
Caicedo, J.C., Goodman, A., Karhohs, K.W., Cimini, B.A., Ackerman, J., Haghighi, M., Heng, C., Becker, T., Doan, M., McQuin, C., et al. Nucleus segmentation across imaging experiments: the 2018 data science bowl. Nat. Methods 16 (2019), 1247–1253.
Svoboda, D., Kozubek, M., Stejskal, S., Generation of digital phantoms of cell nuclei and simulation of image formation in 3D image cytometry. Cytometry A 75 (2009), 494–509.
Wiesner, D., Svoboda, D., Maška, M., Kozubek, M., CytoPacq: a web-interface for simulating multi-dimensional cell imaging. Bioinformatics 35 (2019), 4531–4533.
Cuntz, H., Forstner, F., Borst, A., Häusser, M., One rule to grow them all: a general theory of neuronal branching and its practical application. PLoS Comput. Biol., 6, 2010, e1000877.
Lehmussola, A., Ruusuvuori, P., Selinummi, J., Huttunen, H., Yli-Harja, O., Computational framework for simulating fluorescence microscope images with cell populations. IEEE Trans. Med. Imaging 26 (2007), 1010–1016.
Vandaele, R., Aceto, J., Muller, M., Péronnet, F., Debat, V., Wang, C.W., Huang, C.T., Jodogne, S., Martinive, P., Geurts, P., et al. Landmark detection in 2D bioimages for geometric morphometrics: a multi-resolution tree-based approach. Sci. Rep., 8, 2018, 538.
Schneider, C.A., Rasband, W.S., Eliceiri, K.W., NIH Image to ImageJ: 25 years of image analysis. Nat. Methods 9 (2012), 671–675.
de Chaumont, F., Dallongeville, S., Chenouard, N., Hervé, N., Pop, S., Provoost, T., Meas-Yedid, V., Pankajakshan, P., Lecomte, T., Le Montagner, Y., et al. Icy: an open bioimage informatics platform for extended reproducible research. Nat. Methods 9 (2012), 690–696.
McQuin, C., Goodman, A., Chernyshev, V., Kamentsky, L., Cimini, B.A., Karhohs, K.W., Doan, M., Ding, L., Rafelski, S.M., Thirstrup, D., et al. CellProfiler 3.0: next-generation image processing for biology. PLoS Biol., 16, 2018, e2005970.
Peng, H., Ruan, Z., Long, F., Simpson, J.H., Myers, E.W., V3D enables real-time 3D visualization and quantitative analysis of large-scale biological image data sets. Nat. Biotechnol. 28 (2010), 348–353.
Berg, S., Kutra, D., Kroeger, T., Straehle, C.N., Kausler, B.X., Haubold, C., Schiegg, M., Ales, J., Beier, T., Rudy, M., et al. ilastik: interactive machine learning for (bio)image analysis. Nat. Methods 16 (2019), 1226–1232.
Eaton, J.W., Bateman, D., Hauberg, S., Wehbring, R., GNU Octave Version 4.2.0 Manual: A High-Level Interactive Language for Numerical Computations. 2016, Free Software Foundation https://octave.org/doc/octave-4.2.0.pdf.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12 (2011), 2825–2830.
Chollet, F., Deep Learning with Python. 2017, Manning.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A. (2017). Automatic Differentiation in PyTorch. NIPS Autodiff Workshop, 2017.
Sirinukunwattana, K., Pluim, J.P.W., Chen, H., Qi, X., Heng, P.A., Guo, Y.B., Wang, L.Y., Matuszewski, B.J., Bruni, E., Sanchez, U., et al. Gland segmentation in colon histology images: the glas challenge contest. Med. Image Anal. 35 (2017), 489–502.
Multon, S., Pesesse, L., Weatherspoon, A., Florquin, S., Van de Poel, J.F., Martin, P., Vincke, G., Hoyoux, R., Marée, R., Verpoorten, D., et al. A Massive Open Online Course (MOOC) on practical histology: a goal, a tool, a large public! Return on a first experience. Ann. Pathol. 38 (2018), 76–84.
Ellenberg, J., Swedlow, J.R., Barlow, M., Cook, C.E., Sarkans, U., Patwardhan, A., Brazma, A., Birney, E., A call for public archives for biological image data. Nat. Methods 15 (2018), 849–854.
