[en] Estimation of biophysical vegetation variables is of interest for diverse applications, such as monitoring of crop growth and health or yield prediction. However, remote estimation of these variables remains challenging due to the inherent complexity of plant architecture, biology and surrounding environment, and the need for features engineering. Recent advancements in deep learning, particularly convolutional neural networks (CNN), offer promising solutions to address this challenge. Unfortunately, the limited availability of labeled data has hindered the exploration of CNNs for regression tasks, especially in the frame of crop phenotyping. In this study, the effectiveness of various CNN models in predicting wheat dry matter, nitrogen uptake, and nitrogen concentration from RGB and multispectral images taken from tillering to maturity was examined. To overcome the scarcity of labeled data, a training pipeline was devised. This pipeline involves transfer learning, pseudo-labeling of unlabeled data and temporal relationship correction. The results demonstrated that CNN models significantly benefit from the pseudolabeling method, while the machine learning approach employing a PLSr did not show comparable performance. Among the models evaluated, EfficientNetB4 achieved the highest accuracy for predicting above-ground biomass, with an R² value of 0.92. In contrast, Resnet50 demonstrated superior performance in predicting LAI, nitrogen uptake, and nitrogen concentration, with R² values of 0.82, 0.73, and 0.80, respectively. Moreover, the study explored multi-output models to predict the distribution of dry matter and nitrogen uptake between stem, inferior leaves, flag leaf, and ear. The findings indicate that CNNs hold promise as accessible and promising tools for phenotyping quantitative biophysical variables of crops. However, further research is required to harness their full potential.
Computational resources have been provided by the Consortium des Équipements de Calcul Intensif (CÉCI), funded by the Fonds de la Recherche Scientifique de Belgique (F.R.S.-FNRS) under Grant No. 2.5020.11 and by the Walloon Region. The authors thank the research and teaching support units Agriculture Is Life of TERRA Teaching and Research Centre, Liege` University for giving access to the field trials. The authors are grateful to Jerome Heens, Jesse Jap, Franc¸oise Thys and Gauthier Lepage for their help. The authors also thank CRA-W/Agromet.be for the meteorological data.This research was funded by the Agriculture, Natural Resources and Environment Research Direction of the Public Service of Wallonia (Belgium), project D65-1412/S1 PHENWHEAT, and the National Fund of Belgium F.R.S-FNRS (FRIA grant). Acknowledgments
Araus J. L. Buchaillot M. L. Kefauver S. C. (2022). “High Throughput Field Phenotyping,” in Wheat Improvement. Eds. Reynolds M. P. Braun H.-J. (Cham: Springer International Publishing), 495–512. doi: 10.1007/978-3-030-90673-327
Arya S. Sandhu K. S. Singh J. kumar S. (2022). Deep learning: As the new frontier in high-throughput plant phenotyping. Euphytica 218, 47. doi: 10.1007/s10681-022-02992-3
Berger K. Verrelst J. Féret J.-B. Wang Z. Wocher M. Strathmann M. et al. (2020). Crop nitrogen monitoring: Recent progress and principal developments in the context of imaging spectroscopy missions. Remote Sens. Environ. 242, 111758. doi: 10.1016/j.rse.2020.111758
Brocks S. Bareth G. (2018). Estimating barley biomass with crop surface models from oblique RGB imagery. Remote Sens. 10, 268. doi: 10.3390/rs10020268
Buxbaum N. Lieth J. H. Earles M. (2022). Non-destructive plant biomass monitoring with high spatio-temporal resolution via proximal RGB-D imagery and end-to-end deep learning. Front. Plant Sci. 13. doi: 10.3389/fpls.2022.758818
Carlier A. Dandrifosse S. Dumont B. Mercatoris B. (2022). Wheat ear segmentation based on a multisensor system and superpixel classification. Plant Phenomics 2022. doi: 10.34133/2022/9841985
Castro W. Marcato Junior J. Polidoro C. Osco L. P. Gonc¸alves W. Rodrigues L. et al. (2020). Deep learning applied to phenotyping of biomass in forages with UAV-based RGB imagery. Sensors 20, 4802. doi: 10.3390/s20174802
Chao Z. Liu N. Zhang P. Ying T. Song K. (2019). Estimation methods developing with remote sensing information for energy crop biomass: a comparative review. Biomass Bioenergy 122, 414–425. doi: 10.1016/j.biombioe.2019.02.002
Dandrifosse S. (2022). Dynamics of wheat organs by close-range multimodal machine vision. Ph.D. thesis, ULiège. GxABT - Liège Université. Gembloux Agro-Bio Tech Gembloux Belgium.
Dandrifosse S. Bouvry A. Leemans V. Dumont B. Mercatoris B. (2020). Imaging wheat canopy through stereo vision: overcoming the challenges of the laboratory to field transition for morphological features extraction. Front. Plant Sci. 11. doi: 10.3389/fpls.2020.00096
Dandrifosse S. Carlier A. Dumont B. Mercatoris B. (2021). Registration and fusion of closeRange multimodal wheat images in field conditions. Remote Sens. 13, 1380. doi: 10.3390/rs13071380
Dandrifosse S. Carlier A. Dumont B. Mercatoris B. (2022). In-field wheat reflectance: how to reach the organ scale? Sensors 22, 3342. doi: 10.3390/s22093342
David E. Serouart M. Smith D. Madec S. Velumani K. Liu S. et al. (2021). Global Wheat Head Detection 2021: an improved dataset for benchmarking wheat head detection methods. Plant Phenomics 2021. doi: 10.34133/2021/9846158
Deery D. Jimenez-Berni J. Jones H. Sirault X. Furbank R. (2014). Proximal remote sensing buggies and potential applications for field-based phenotyping. Agronomy 4, 349–379. doi: 10.3390/agronomy4030349
Deng J. Dong W. Socher R. Li L.-J. Li K. Fei-Fei L. (2009). “ImageNet: A large-scale hierarchical image database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE. 248–255. doi: 10.1109/CVPR.2009.5206848
de Oliveira G. S. Marcato Junior J. Polidoro C. Osco L. P. Siqueira H. Rodrigues L. et al. (2021). Convolutional neural networks to estimate dry matter yield in a Guineagrass breeding program using UAV remote sensing. Sensors 21, 3971. doi: 10.3390/s21123971
Duchene O. Dumont B. Cattani D. J. Fagnant L. Schlautman B. DeHaan L. R. et al. (2021). Processbased analysis of Thinopyrum intermedium phenological development highlights the importance of dual induction for reproductive growth and agronomic performance. Agric. For. Meteorol, 301–302. doi: 10.1016/j.agrformet.2021.108341
Freitas Moreira F. Rojas de Oliveira H. Lopez M. A. Abughali B. J. Gomes G. Cherkauer K. A. et al. (2021). High-throughput phenotyping and random regression models reveal temporal genetic control of soybean biomass production. Front. Plant Sci. 12. doi: 10.3389/fpls.2021.715983
Gaju O. Allard V. Martre P. Le Gouis J. Moreau D. Bogard M. et al. (2014). Nitrogen partitioning and remobilization in relation to leaf senescence, grain yield and grain nitrogen concentration in wheat cultivars. Field Crops Res. 155, 213–223. doi: 10.1016/j.fcr.2013.09.003
Gao Y. Li Y. Jiang R. Zhan X. Lu H. Guo W. et al. (2023). Enhancing green fraction estimation in rice and wheat crops: A self-supervised deep learning semantic segmentation approach. Plant Phenomics 5, 64. doi: 10.34133/plantphenomics.0064
Hawkesford M. Riche A. (2020). Impacts of G x E x M on nitrogen use efficiency in wheat and future prospects. Front. Plant Sci. 11. doi: 10.3389/fpls.2020.01157
He K. Zhang X. Ren S. Sun J. (2015). Deep Residual Learning for Image Recognition. doi: 10.48550/arXiv.1512.03385
Hickey L. T. Hafeez A. N. Robinson H. Jackson S. A. Leal-Bertioli S. C. M. et al. (2019). Breeding crops to feed 10 billion. Nat. Biotechnol. 37, 744–754. doi: 10.1038/s41587-019-0152-9
Jiang Y. Li C. (2020). Convolutional neural networks for image-based high-throughput plant phenotyping: a review. Plant Phenomics, 1–22. doi: 10.34133/2020/4152816
Justes E. (1994). Determination of a critical nitrogen dilution curve for winter wheat crops. Ann. Bot. 74, 397–407. doi: 10.1006/anbo.1994.1133
Kamilaris A. Prenafeta-Boldú F. X. (2018). Deep learning in agriculture: A survey. Comput. Electron. Agric. 147, 70–90. doi: 10.1016/j.compag.2018.02.016
Kattenborn T. Leitloff J. Schiefer F. Hinz S. (2021). Review on Convolutional Neural Networks (CNN) in vegetation remote sensing. ISPRS J. Photogrammetry Remote Sens. 173, 24–49. doi: 10.1016/j.isprsjprs.2020.12.010
Lee D.-H. (2013). Pseudo-label: the simple and efficient semi-supervised learning method for deep neural networks. ICML 2013 Workshop Challenges Representation Learn. (WREPL).
Lemaire G. Ciampitti I. (2020). Crop mass and N status as prerequisite covariables for unraveling nitrogen use efficiency across genotype-by-environment-by-management scenarios: a review. Plants 9, 1309. doi: 10.3390/plants9101309
Li Y. Liu H. Ma J. Zhang L. (2021). Estimation of leaf area index for winter wheat at early stages based on convolutional neural networks. Comput. Electron. Agric. 190, 106480. doi: 10.1016/j.compag.2021.106480
Ma J. Li Y. Chen Y. Du K. Zheng F. Zhang L. et al. (2019). Estimating above ground biomass of winter wheat at early growth stages using digital images and deep convolutional neural network. Eur. J. Agron. 103, 117–129. doi: 10.1016/j.eja.2018.12.004
Martre P. Porter J. R. Jamieson P. D. Tribo¨ı E. (2003). Modeling grain nitrogen accumulation and protein composition to understand the sink/source regulations of nitrogen remobilization for wheat. Plant Physiol. 133, 1959–1967. doi: 10.1104/pp.103.030585
Nabwire S. Suh H.-K. Kim M. S. Baek I. Cho B.-K. (2021). Review: application of artificial intelligence in phenomics. Sensors 21, 4363. doi: 10.3390/s21134363
Nguyen C. Sagan V. Bhadra S. Moose S. (2023). UAV multisensory data fusion and multi-task deep learning for high-throughput maize phenotyping. Sensors 23, 1827. doi: 10.3390/s23041827
Otsu N. (1979). A threshold selection method from gray-level histograms. IEEE Trans. Systems Man Cybernetics 9, 62–66. doi: 10.1109/TSMC.1979.4310076
Pedregosa F. Varoquaux G. Gramfort A. Michel V. Thirion B. Grisel O. et al. (2011). Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830. doi: 10.48550/arXiv.1201.0490
Pound M. P. Atkinson J. A. Wells D. M. Pridmore T. P. French A. P. (2017). “Deep learning for multi-task plant phenotyping,” in 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), 2055–2063. doi: 10.1109/ICCVW.2017.241
Raj R. Walker J. P. Pingale R. Nandan R. Naik B. Jagarlapudi A. (2021). Leaf area index estimation using top-of-canopy airborne RGB images. Int. J. Appl. Earth Observation Geoinformation 96, 102282. doi: 10.1016/j.jag.2020.102282
Reynolds M. Chapman S. Crespo-Herrera L. Molero G. Mondal S. Pequeno D. N. et al. (2020). Breeder friendly phenotyping. Plant Sci. 295, 110396. doi: 10.1016/j.plantsci.2019.110396
Roth L. Barendregt C. Bétrix C.-A. Hund A. Walter A. (2022). High-throughput field phenotyping of soybean: spotting an ideotype. Remote Sens. Environ. 269, 112797. doi: 10.1016/j.rse.2021
Roth L. Camenzind M. Aasen H. Kronenberg L. Barendregt C. Camp K.-H. et al. (2020). Repeated multiview imaging for estimating seedling tiller counts of wheat genotypes using drones. Plant Phenomics 2020. doi: 10.34133/2020/3729715
Sapkota B. Popescu S. Rajan N. Leon R. Reberg-Horton C. Mirsky S. et al. (2022). Use of synthetic images for training a deep learning model for weed detection and biomass estimation in cotton. Sci. Rep. 12. doi: 10.1038/s41598-022-23399-z
Schiefer F. Schmidtlein S. Kattenborn T. (2021). The retrieval of plant functional traits from canopy spectra through RTM-inversions and statistical models are both critically affected by plant phenology. Ecol. Indic. 121. doi: 10.1016/j.ecolind.2020.107062
Schreiber L. Atkinson Amorim J. Guimãraes L. Motta Matos D. Maciel da Costa C. Parraga A. (2022). Above-ground biomass wheat estimation: deep learning with UAV-based RGB images. Appl. Artif. Intell. doi: 10.1080/08839514.2022.2055392
Selvaraju R. R. Cogswell M. Das A. Vedantam R. Parikh D. Batra D. (2020). Grad-CAM: visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vision 128, 336–359. doi: 10.1007/s11263-019-01228-7
Serouart M. Madec S. David E. Velumani K. Lopez Lozano R. Weiss M. et al. (2022). SegVeg: segmenting RGB images into green and senescent vegetation by combining deep and shallow methods. Plant Phenomics 2022. doi: 10.34133/2022/9803570
Singh A. K. Ganapathysubramanian B. Sarkar S. Singh A. (2018). Deep learning for plant stress phenotyping: trends and future perspectives. Trends Plant Sci. 23, 883–898. doi: 10.1016/j.tplants.2018.07.004
Song X. Yang G. Xu X. Zhang D. Yang C. Feng H. (2022). Winter wheat nitrogen estimation based on ground-level and UAV-mounted sensors. Sensors 22. doi: 10.3390/s22020549
Standley T. Zamir A. R. Chen D. Guibas L. Malik J. Savarese S. (2020). Which tasks should be learned together in multi-task learning? doi: 10.48550/arXiv.1905.07553
Sun D. Robbins K. Morales N. Shu Q. Cen H. (2022). Advances in optical phenotyping of cereal crops. Trends Plant Sci. 27, 191–208. doi: 10.1016/j.tplants.2021.07.015
Tan M. Le Q. V. (2020). EfficientNet: rethinking model scaling for convolutional neural networks. doi: 10.48550/arXiv.1905.11946
Tanner F. Tonn S. de Wit J. Van den Ackerveken G. Berger B. Plett D. (2022). Sensorbased phenotyping of above-ground plant-pathogen interactions. Plant Methods 18, 35. doi: 10.1186/s13007-022-00853-7
Tilly N. Aasen H. Bareth G. (2015). Fusion of plant height and vegetation indices for the estimation of barley biomass. Remote Sens. 7, 11449–11480. doi: 10.3390/rs70911449
Vafaeikia P. Namdar K. Khalvati F. (2020). A brief review of deep multi-task learning and auxiliary task learning. doi: 10.48550/arXiv.2007.01126
Vandenhende S. Georgoulis S. Proesmans M. Dai D. Gool L. (2020). Revisiting multi-task learning in the deep learning era. ArXiv. doi: 10.1109/TPAMI.2021.3054719
van Eeuwijk F. A. Bustos-Korts D. Millet E. J. Boer M. P. Kruijer W. Thompson A. et al. (2019). Modelling strategies for assessing and increasing the effectiveness of new phenotyping techniques in plant breeding. Plant Sci. 282, 23–39. doi: 10.1016/j.plantsci.2018.06.018
Verrelst J. Malenovsky,´ Z. van der Tol C. Camps-Valls G. Gastellu-Etchegorry J.-P. Lewis P. et al. (2019). Quantifying vegetation biophysical variables from imaging spectroscopy data: a review on retrieval methods. Surveys Geophysics 40, 589–629. doi: 10.1007/s10712-018-9478-y
Wan L. Zhang J. Dong X. Du X. Zhu J. Sun D. et al. (2021). Unmanned aerial vehicle-based field phenotyping of crop biomass using growth traits retrieved from PROSAIL model. Comput. Electron. Agric. 187, 106304. doi: 10.1016/j.compag.2021.106304
Wang Z. Lu Y. Zhao G. Sun C. Zhang F. He S. (2022b). Sugarcane biomass prediction with multi-mode remote sensing data using deep archetypal analysis and integrated learning. Remote Sens. 14, 4944. doi: 10.3390/rs14194944
Wang W. Wu Y. Zhang Q. Zheng H. Yao X. Zhu Y. et al. (2021). AAVI: A novel approach to estimating leaf nitrogen concentration in rice from unmanned aerial vehicle multispectral imagery at early and middle growth stages. IEEE J. Selected Topics Appl. Earth Observations Remote Sens. 14, 6716–6728. doi: 10.1109/JSTARS.2021.3086580
Wang F. Yang M. Ma L. Zhang T. Qin W. Li W. et al. (2022a). Estimation of above-ground biomass of winter wheat based on consumer-grade multi-spectral UAV. Remote Sens. 14, 1251. doi: 10.3390/rs14051251
Xu R. Li C. (2022). A review of high-throughput field phenotyping systems: focusing on ground robots. Plant Phenomics 2022. doi: 10.34133/2022/9760269
Yang K.-W. Chapman S. Carpenter N. Hammer G. McLean G. Zheng B. et al. (2021). Integrating crop growth models with remote sensing for predicting biomass yield of sorghum. silico Plants 3, diab001. doi: 10.1093/insilicoplants/diab001
Yue J. Yang G. Tian Q. Feng H. Xu K. Zhou C. (2019). Estimate of winter-wheat above-ground biomass based on UAV ultrahigh-ground-resolution image textures and vegetation indices. ISPRS J. Photogrammetry Remote Sens. 150, 226–244. doi: 10.1016/j.isprsjprs.2019.02.022
Zbontar J. Jing L. Misra I. LeCun Y. Deny S. (2021). Barlow twins: self-supervised learning via redundancy reduction. doi: 10.48550/arXiv.2103.03230
Zhang J. Tian H. Wang P. Tansey K. Zhang S. Li H. (2022). Improving wheat yield estimates using data augmentation models and remotely sensed biophysical indices within deep neural networks in the Guanzhong Plain, PR China. Comput. Electron. Agric. 192, 106616. doi: 10.1016/j.compag.2021.106616
Zheng C. Abd-Elrahman A. Whitaker V. M. Dalid C. (2022). Deep learning for strawberry canopy delineation and biomass prediction from high-resolution images. Plant Phenomics 2022. doi: 10.34133/2022/9850486
