Registration and fusion of close-range multimodal wheat images in field conditions

Dandrifosse, Sébastien; Carlier, Alexis; Dumont, Benjamin; Mercatoris, Benoît

doi:10.3390/rs13071380

Download

Article (Scientific journals)

Registration and fusion of close-range multimodal wheat images in field conditions

Dandrifosse, Sébastien; Carlier, Alexis; Dumont, Benjamin et al.

2021 • In Remote Sensing

Peer Reviewed verified by ORBi

Permalink
https://hdl.handle.net/2268/258867

DOI
10.3390/rs13071380

Files (1)Send to Details Statistics Bibliography Similar publications

Files

Full Text

Dandrifosse2021_Registration.pdf

Publisher postprint (8.63 MB)

Download

All documents in ORBi are protected by a user license.

Send to

RIS BibTex APA Chicago Permalink X Linkedin

Details

Keywords :

image registration; proxy-sensing; high-throughput phenotyping; winter wheat; thermography; multispectral

Abstract :

[en] Multimodal images fusion has the potential to enrich the information gathered by multisensor plant phenotyping platforms. Fusion of images from multiple sources is, however, hampered by the technical lock of image registration. The aim of this paper is to provide a solution to the registration and fusion of multimodal wheat images in field conditions and at close range. Eight registration methods were tested on nadir wheat images acquired by a pair of red, green and blue (RGB) cameras, a thermal camera and a multispectral camera array. The most accurate method, relying on a local transformation, aligned the images with an average error of 2 mm but was not reliable for thermal images. More generally, the suggested registration method and the preprocesses necessary before fusion (plant mask erosion, pixel intensity averaging) would depend on the application. As a consequence, the main output of this study was to identify four registration-fusion strategies: (i) the REAL-TIME strategy solely based on the cameras’ positions, (ii) the FAST strategy suitable for all types of images tested, (iii) and (iv) the ACCURATE and HIGHLY ACCURATE strategies handling local distortion but unable to deal with images of very different natures. These suggestions are, however, limited to the methods compared in this study. Further research should investigate how recent cutting-edge registration methods would perform on the specific case of wheat canopy.

Disciplines :

Computer science

Author, co-author :

Dandrifosse, Sébastien ; Université de Liège - ULiège > Département GxABT > Biosystems Dynamics and Exchanges

Carlier, Alexis ; Université de Liège - ULiège > Département GxABT > Biosystems Dynamics and Exchanges

Dumont, Benjamin ; Université de Liège - ULiège > Département GxABT > Phytotechnie tempérée

Mercatoris, Benoît ; Université de Liège - ULiège > Département GxABT > Biosystems Dynamics and Exchanges

Language :

English

Title :

Registration and fusion of close-range multimodal wheat images in field conditions

Publication date :

03 April 2021

Journal title :

Remote Sensing

eISSN :

2072-4292

Publisher :

Multidisciplinary Digital Publishing Institute (MDPI), Basel, Switzerland

Peer reviewed :

Peer Reviewed verified by ORBi

Name of the research project :

Natural Resources and Environment Research Direction of the Public Service of Wallonia (Belgium): project D31-1385 PHENWHEAT

Funders :

SPW - Public Service of Wallonia
F.R.S.-FNRS - Fonds de la Recherche Scientifique
FRIA - Fonds pour la Formation à la Recherche dans l'Industrie et dans l'Agriculture

Available on ORBi :

since 14 April 2021

Statistics

Number of views

187 (23 by ULiège)

Number of downloads

114 (11 by ULiège)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

Kirchgessner, N.; Liebisch, F.; Yu, K.; Pfeifer, J.; Friedli, M.; Hund, A.; Walter, A. The ETH field phenotyping platform FIP: A cable-suspended multi-sensor system. Funct. Plant Biol. 2017, 44, 154-168.
Shafiekhani, A.; Kadam, S.; Fritschi, F.; DeSouza, G. Vinobot and Vinoculer: Two Robotic Platforms for High-Throughput Field Phenotyping. Sensors 2017, 17, 214.
Virlet, N.; Sabermanesh, K.; Sadeghi-Tehran, P.; Hawkesford, M.J. Field Scanalyzer: An automated robotic field phenotyping platform for detailed crop monitoring. Funct. Plant Biol. 2017, 44, 143-153.
Jiang, Y.; Li, C.; Robertson, J.S.; Sun, S.; Xu, R.; Paterson, A.H. GPhenoVision: A ground mobile system with multi-modal imaging for field-based high throughput phenotyping of cotton. Sci. Rep. 2018, 8, 1-15.
Bai, G.; Ge, Y.; Scoby, D.; Leavitt, B.; Stoerger, V.; Kirchgessner, N.; Irmak, S.; Graef, G.; Schnable, J.; Awada, T. NU-Spidercam:A large-scale, cable-driven, integrated sensing and robotic system for advanced phenotyping, remote sensing, and agronomic research. Comput. Electron. Agric. 2019, 160, 71-81.
Beauchêne, K.; Leroy, F.; Fournier, A.; Huet, C.; Bonnefoy, M.; Lorgeou, J.; de Solan, B.; Piquemal, B.; Thomas, S.; Cohan, J.-P. Management and Characterization of Abiotic Stress via PhénoField®, a High-Throughput Field Phenotyping Platform. Front. Plant Sci. 2019, 10, 1-17.
Pérez-Ruiz, M.; Prior, A.; Martinez-Guanter, J.; Apolo-Apolo, O.E.; Andrade-Sanchez, P.; Egea, G. Development and evaluation of a self-propelled electric platform for high-throughput field phenotyping in wheat breeding trials. Comput. Electron. Agric. 2020, 169, 105237.
Leinonen, I.; Jones, H.G. Combining thermal and visible imagery for estimating canopy temperature and identifying plant stress. J. Exp. Bot. 2004, 55, 1423-1431.
Jerbi, T.; Wuyts, N.; Cane, M.A.; Faux, P.-F.; Draye, X. High resolution imaging of maize (Zea maize) leaf temperature in the field:The key role of the regions of interest. Funct. Plant Biol. 2015, 42, 858.
Huang, P.; Luo, X.; Jin, J.; Wang, L.; Zhang, L.; Liu, J.; Zhang, Z. Improving high-throughput phenotyping using fusion of close-range hyperspectral camera and low-cost depth sensor. Sensors 2018, 18, 2711.
Khanna, R.; Schmid, L.; Walter, A.; Nieto, J.; Siegwart, R.; Liebisch, F. A spatio temporal spectral framework for plant stress phenotyping. Plant Methods 2019, 15, 1-18.
Roitsch, T.; Cabrera-Bosquet, L.; Fournier, A.; Ghamkhar, K.; Jiménez-Berni, J.; Pinto, F.; Ober, E.S. Review: New sensors and data-driven approaches-A path to next generation phenomics. Plant Sci. 2019, 282, 2-10.
Mishra, P.; Asaari, M.S.M.; Herrero-Langreo, A.; Lohumi, S.; Diezma, B.; Scheunders, P. Close range hyperspectral imaging of plants: A review. Biosyst. Eng. 2017, 164, 49-67.
Busemeyer, L.; Mentrup, D.; Möller, K.;Wunder, E.; Alheit, K.; Hahn, V.; Maurer, H.P.; Reif, J.C.; Würschum, T.; Müller, J.; et al. Breedvision-A multi-sensor platform for non-destructive field-based phenotyping in plant breeding. Sensors 2013, 13, 2830-2847.
Deery, D.; Jimenez-Berni, J.; Jones, H.; Sirault, X.; Furbank, R. Proximal Remote Sensing Buggies and Potential Applications for Field-Based Phenotyping. Agronomy 2014, 5, 349-379.
Behmann, J.; Acebron, K.; Emin, D.; Bennertz, S.; Matsubara, S.; Thomas, S.; Bohnenkamp, D.; Kuska, M.T.; Jussila, J.; Salo, H.; et al. Specim IQ: Evaluation of a new, miniaturized handheld hyperspectral camera and its application for plant phenotyping and disease detection. Sensors 2018, 18, 441.
Whetton, R.L.; Waine, T.W.; Mouazen, A.M. Hyperspectral measurements of yellow rust and fusarium head blight in cereal crops:Part 2: On-line field measurement. Biosyst. Eng. 2018, 167, 144-158.
Leemans, V.; Marlier, G.; Destain, M.-F.; Dumont, B.; Mercatoris, B. Estimation of leaf nitrogen concentration on winter wheat by multispectral imaging. In Proceedings of the Hyperspectral Imaging Sensors: Innovative Applications and Sensor Standards 2017, Anaheim, CA, USA, 12 April 2017; SPIE-International Society for Optics and Photonics: Bellingham, WA, USA, 2017.
Bebronne, R.; Carlier, A.; Meurs, R.; Leemans, V.; Vermeulen, P.; Dumont, B.; Mercatoris, B. In-field proximal sensing of septoria tritici blotch, stripe rust and brown rust in winter wheat by means of reflectance and textural features from multispectral imagery. Biosyst. Eng. 2020, 197, 257-269.
Genser, N.; Seiler, J.; Kaup, A. Camera Array for Multi-Spectral Imaging. IEEE Trans. Image Process. 2020, 29, 9234-9249.
Jiménez-Bello, M.A.; Ballester, C.; Castel, J.R.; Intrigliolo, D.S. Development and validation of an automatic thermal imaging process for assessing plant water status. Agric. Water Manag. 2011, 98, 1497-1504.
Möller, M.; Alchanatis, V.; Cohen, Y.; Meron, M.; Tsipris, J.; Naor, A.; Ostrovsky, V.; Sprintsin, M.; Cohen, S. Use of thermal and visible imagery for estimating crop water status of irrigated grapevine. J. Exp. Bot. 2007, 58, 827-838.
Wang, X.; Yang, W.; Wheaton, A.; Cooley, N.; Moran, B. Efficient registration of optical and IR images for automatic plant water stress assessment. Comput. Electron. Agric. 2010, 74, 230-237.
Zitová, B.; Flusser, J. Image registration methods: A survey. Image Vis. Comput. 2003, 21, 977-1000.
Rabatel, G.; Labbé, S. Registration of visible and near infrared unmanned aerial vehicle images based on Fourier-Mellin transform. Precis. Agric. 2016, 17, 564-587.
Klein, S.; Staring, M.; Murphy, K.; Viergever, M.A.; Pluim, J. Elastix: A Toolbox for Intensity-Based Medical Image Registration. IEEE Trans. Med. Imaging 2010, 29, 196-205.
Sotiras, A.; Davatzikos, C.; Paragios, N. Deformable medical image registration: A survey. IEEE Trans. Med. Imaging 2013, 32, 1153-1190.
De Vylder, J.; Douterloigne, K.; Vandenbussche, F.; Van Der Straeten, D.; Philips, W. A non-rigid registration method for multispectral imaging of plants. Sens. Agric. Food Qual. Saf. IV 2012, 8369, 836907.
Raza, S.E.A.; Sanchez, V.; Prince, G.; Clarkson, J.P.; Rajpoot, N.M. Registration of thermal and visible light images of diseased plants using silhouette extraction in the wavelet domain. Pattern Recognit. 2015, 48, 2119-2128.
Henke, M.; Junker, A.; Neumann, K.; Altmann, T.; Gladilin, E. Comparison of feature point detectors for multimodal image registration in plant phenotyping. PLoS ONE 2019, 14, 1-16.
Henke, M.; Junker, A.; Neumann, K.; Altmann, T.; Gladilin, E. Comparison and extension of three methods for automated registration of multimodal plant images. Plant Methods 2019, 15, 1-15.
Meier, U. Growth Stages of Mono and Dicotyledonous Plants. BBCH Monograph 2nd Edition; Federal Biological Research Centre for Agriculture and Forestry: Quedlinburg, Germany, 2001; ISBN 9783955470715.
Berenstein, R.; Hočevar, M.; Godeša, T.; Edan, Y.; Ben-Shahar, O. Distance-dependent multimodal image registration for agriculture tasks. Sensors 2015, 15, 20845-20862.
Dandrifosse, S.; Bouvry, A.; Leemans, V.; Dumont, B.; Mercatoris, B. Imaging wheat canopy through stereo vision: Overcoming the challenges of the laboratory to field transition for morphological features extraction. Front. Plant Sci. 2020, 11, 1-15.
Bradski, G.; Kaehler, A. Learning OpenCV; O’Reilly Media, Inc.: Newton, MA, USA, 2008; ISBN 978-1-4493-1465-1.
Hirschmüller, H. Stereo Processing by Semi-Global Matching and Mutual Information. IEEE Trans. Pattern Anal. Mach. Intell. 2007, 30, 328-341.
Xiong, Z.; Zhang, Y. A critical review of image registration methods. Int. J. Image Data Fusion 2010, 1, 137-158.
Low, D.G. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 2004, 60, 91-110.
Bay, H.; Ess, A.; Tuytelaars, T.; Gool, L. Van Speeded-Up Robust Features (SURF). In Computer Vision and Image Understanding; Springer: Berlin/Heidelberg, Germany, 2008; Volume 110, pp. 346-359.
Rublee, E.; Rabaud, V.; Konolige, K.; Bradski, G. ORB: An efficient alternative to SIFT or SURF. In Proceedings of the IEEE International Conference on Computer Vision; IEEE: Manhattan, NY, USA, 2011; pp. 2564-2571.
Alcantarilla, P.F.; Nuevo, J.; Bartoli, A. Fast explicit diffusion for accelerated features in nonlinear scale spaces. In Proceedings of the BMVC 2013-Electronic Proceedings of the British Machine Vision Conference 2013, Bristol, UK, 9-13 September 2013; BMVA Press: Swansea, UK, 2013.
Srinivasa Reddy, B.; Chatterji, B.N. An FFT-based technique for translation, rotation, and scale-invariant image registration. IEEE Trans. Image Process. 1996, 5, 1266-1271.
Evangelidis, G.D.; Psarakis, E.Z. Parametric image alignment using enhanced correlation coefficient maximization. IEEE Trans. Pattern Anal. Mach. Intell. 2008, 30, 1858-1865.
Rueckert, D.; Sonoda, L.I.; Hayes, C.; Hill, D.L.; Leach, M.O.; Hawkes, D.J. Nonrigid Registration Using Free-Form Deformations:Application to Breast MR Images. IEEE Trans. Med. Imaging 1999, 18, 712-721.
Studholme, C.; Hill, D.L.G.; Hawkes, D.J. An overlap invariant entropy measure of 3D medical image alignment. Pattern Recognit. 1999, 32, 71-86.
Keszei, A.P.; Berkels, B.; Deserno, T.M. Survey of Non-Rigid Registration Tools in Medicine. J. Digit. Imaging 2017, 30, 102-116.
Yang,W.P.;Wang, X.Z.; Wheaton, A.; Cooley, N.; Moran, B. Ieee Automatic Optical and IR Image Fusion for PlantWater Stress Analysis. In Proceedings of the 12th International Conference on Information Fusion, Seattle, WA, USA, 6-9 July 2009; IEEE:Manhattan, NY, USA, 2009; pp. 1053-1059.
Yang,W.;Wang, X.; Moran, B.; Wheaton, A.; Cooley, N. Efficient registration of optical and infrared images via modified Sobel edging for plant canopy temperature estimation. Comput. Electr. Eng. 2012, 38, 1213-1221.
Rohlfing, Torsten, 2013 Image Similarity and Tissue Overlaps as Surrogates for Image Registration Accuracy: Widely Used but Unreliable. IEEE Trans. Med. Imaging 2012, 31, 153-163.
Feng, R.; Du, Q.; Li, X.; Shen, H. ISPRS Journal of Photogrammetry and Remote Sensing Robust registration for remote sensing images by combining and localizing feature- and area-based methods. ISPRS J. Photogramm. Remote Sens. 2019, 151, 15-26.
Zach, C.; Pock, T.; Bischof, H. A Duality Based Approach for Realtime TV-L1 Optical Flow. In Proceedings of the Pattern Recognition; Hamprecht, F.A., Schnörr, C., Jähne, B., Eds.; Springer: Berlin/Heidelberg, Germany, 2007; pp. 214-223.
Javier, S.; Meinhardt-llopis, E.; Facciolo, G. TV-L1 Optical Flow Estimation. Image Process. Line 2013, 1, 137-150.
Nguyen, T.; Chen, S.W.; Shivakumar, S.S.; Taylor, C.J.; Kumar, V. Unsupervised Deep Homography: A Fast and Robust Homography Estimation Model. IEEE Robot. Autom. Lett. 2018, 3, 2346-2353.
Balakrishnan, G.; Zhao, A.; Sabuncu, M.R.; Guttag, J.; Dalca, A. V An Unsupervised Learning Model for Deformable Medical Image Registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2018, Salt Lake City, UT, USA, 18-23 June 2018; IEEE: Manhattan, NY, USA, 2018.