Convolutional neural networks; Deep learning; Information fusion; Precision livestock farming; Recurrent neural networks; Ruminant foraging behaviour; Cattles; Convolutional neural network; F1 scores; Feeding behavior; Foraging behaviours; Fusion model; Neural-networks; Ruminant foraging behavior; Control and Systems Engineering; Artificial Intelligence; Electrical and Electronic Engineering
Abstract :
[en] Monitoring feeding behaviour is a relevant task for efficient herd management and the effective use of available resources in grazing cattle. The ability to automatically recognise animals’ feeding activities through the identification of specific jaw movements allows for the improvement of diet formulation, as well as early detection of metabolic problems and symptoms of animal discomfort, among other benefits. The use of sensors to obtain signals for such monitoring has become popular in the last two decades. The most frequently employed sensors include accelerometers, microphones, and cameras, each with its own set of advantages and drawbacks. An unexplored aspect is the simultaneous use of multiple sensors with the aim of combining signals in order to enhance the precision of the estimations. In this direction, this work introduces a deep neural network based on the fusion of acoustic and inertial signals, composed of convolutional, recurrent, and dense layers. The main advantage of this model is the combination of signals through the automatic extraction of features independently from each of them. The model has emerged from an exploration and comparison of different neural network architectures proposed in this work, which carry out information fusion at different levels. Feature-level fusion has outperformed data and decision-level fusion by at least a 0.14 based on the F1-score metric. Moreover, a comparison with state-of-the-art machine learning methods is presented, including traditional and deep learning approaches. The proposed model yielded an F1-score value of 0.802, representing a 14% increase compared to previous methods. Finally, results from an ablation study and post-training quantisation evaluation are also reported.
Disciplines :
Agriculture & agronomy
Author, co-author :
Ferrero, Mariano ; Instituto de Investigación en Señales, Sistemas e Inteligencia Computacional, sinc(i), FICH-UNL/CONICET, Santa Fe, Argentina
Chelotti, José ; Université de Liège - ULiège > Département GxABT > Animal Sciences (AS) ; Instituto de Investigación en Señales, Sistemas e Inteligencia Computacional, sinc(i), FICH-UNL/CONICET, Santa Fe, Argentina
Martinez-Rau, Luciano S. ; Instituto de Investigación en Señales, Sistemas e Inteligencia Computacional, sinc(i), FICH-UNL/CONICET, Santa Fe, Argentina ; Department of Computer and Electrical Engineering, Mid Sweden University, Sundsvall, Sweden
Vignolo, Leandro D. ; Instituto de Investigación en Señales, Sistemas e Inteligencia Computacional, sinc(i), FICH-UNL/CONICET, Santa Fe, Argentina
Pires, Martín; Facultad de Ciencias Agrarias, Univ. Nacional de Rosario, Zavalla, Argentina
Galli, Julio R. ; Facultad de Ciencias Agrarias, Univ. Nacional de Rosario, Zavalla, Argentina ; Instituto de Investigaciones en Ciencias Agrarias de Rosario, IICAR, Facultad de Ciencias Agrarias, UNR-CONICET, Zavalla, Argentina
Giovanini, Leonardo L. ; Instituto de Investigación en Señales, Sistemas e Inteligencia Computacional, sinc(i), FICH-UNL/CONICET, Santa Fe, Argentina
Rufiner, H. Leonardo ; Instituto de Investigación en Señales, Sistemas e Inteligencia Computacional, sinc(i), FICH-UNL/CONICET, Santa Fe, Argentina ; Laboratorio de Cibernética, Facultad de Ingeniería, Univ. Nacional de Entre Ríos, Oro Verde, Argentina
Language :
English
Title :
A multi-head deep fusion model for recognition of cattle foraging events using sound and movement signals
Publication date :
2025
Journal title :
Engineering Applications of Artificial Intelligence
Santa Fe Science, Technology and Innovation Agency CONICET - National Scientific and Technical Research Council UNL - National University of the Littoral UNR - National University of Rosario Nvidia Knowledge Foundation
Funding text :
This work has been funded by Universidad Nacional del Litoral, Argentina, CAID 50620190100080LI and 50620190100151LI, Universidad Nacional de Rosario, Argentina, projects 2013-AGR216, 2016-AGR266 and 80020180300053UR, Agencia Santafesina de Ciencia, Tecnolog\u00EDa e Innovaci\u00F3n (ASACTEI), project IO\u20132018\u201400082,Consejo Nacional de Investigaciones Cient\u00EDficas y T\u00E9cnicas (CONICET), project 2017-PUE sinc(i). Authors would like to thank the dedication and perceptive help by Campo Experimental J. Villarino Dairy Farm staff for their assistance and support during the completion of this study. Authors also gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan XP GPU used for this research. Finally, our thanks to Constanza Quaglia for her enormous contribution to this work through the development of the Android application used to capture signals during fieldwork.This work has been funded by Universidad Nacional del Litoral, Argentina , CAID 50620190100080LI and 50620190100151LI , Universidad Nacional de Rosario, Argentina , projects 2013-AGR216 , 2016-AGR266 and 80020180300053UR , Agencia Santafesina de Ciencia, Tecnolog\u00EDa e Innovaci\u00F3n (ASACTEI) , project IO\u20132018\u201400082 , Consejo Nacional de Investigaciones Cient\u00EDficas T\u00E9cnicas (CONICET) , project 2017-PUE sinc(i) . Authors would like to thank the dedication and perceptive help by Campo Experimental J. Villarino Dairy Farm staff for their assistance and support during the completion of this study. Authors also gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan XP GPU used for this research. Finally, our thanks to Constanza Quaglia for her enormous contribution to this work through the development of the Android application used to capture signals during fieldwork.
Alvarenga, F.A.P., Borges, I., Oddy, V.H., Dobos, R.C., Discrimination of biting and chewing behaviour in sheep using a tri-axial accelerometer. Comput. Electron. Agric., 168, 2020, 105051.
Andriamandroso, A.L.H., Bindelle, J., Mercatoris, B., Lebeau, F., A review on the use of sensors to monitor cattle jaw movements and behavior when grazing. Biotechnol. Agron. Soc. Env. Pages 27 (2016), 3–286.
Andriamandroso, A.L.H., Lebeau, F., Beckers, Y., Froidmont, E., Dufrasne, I., Heinesch, B., Dumortier, P., Blanchy, G., Blaise, Y., Bindelle, J., Development of an open-source algorithm based on inertial measurement units (IMU) of a smartphone to detect cattle grass intake and ruminating behaviors. Comput. Electron. Agric. 139 (2017), 126–137.
Andriamandroso, A., Lebeau, F., Bindelle, J., Changes in biting characteristics recorded using the inertial measurement unit of a smartphone reflect differences in sward attributes. 7th Conference on Precision Livestock Farming, vol. 28, 2015, 3–289.
Aquilani, C., Confessore, A., Bozzi, R., Sirtori, F., Pugliese, C., Review: Precision livestock farming technologies in pasture-based livestock systems. Animal, 16(1), 2022, 100429.
Arablouei, R., Wang, Z., Bishop-Hurley, G.J., Liu, J., Multimodal sensor data fusion for in-situ classification of animal behavior using accelerometry and GNSS data. Smart Agric. Technol., 4(100163), 2023, 100163.
Aydogmus, O., Bingol, M.C., Boztas, G., Tuncer, T., An automated voice command classification model based on an attention-deep convolutional neural network for industrial automation system. Eng. Appl. Artif. Intell., 126, 2023, 107120.
Bishop, C.M., Pattern Recognition and Machine Learning. 2006, Springer New York.
Bloch, V., Frondelius, L., Arcidiacono, C., Mancino, M., Pastell, M., Development and analysis of a CNN- and Transfer-Learning-Based classification model for automated dairy cow feeding behavior recognition from accelerometer data. Sensors, 23(5), 2023.
Bosi, M., Brandenburg, K., Quackenbush, S., Fielder, L., Akagiri, K., Fuchs, H., Dietz, M., ISO/IEC MPEG-2 advanced audio coding. J. Audio Eng. Soc. 45 (1997), 789–814.
Bristow, D.J., Holmes, D.S., Cortisol levels and anxiety-related behaviors in cattle. Physiol. Behav. 90:4 (2007), 626–628.
Büchel, S., Sundrum, A., Short communication: Decrease in rumination time as an indicator of the onset of calving. J. Dairy Sci. 97:5 (2014), 3120–3127.
Calamari, L., Soriani, N., Panella, G., Petrera, F., Minuti, A., Trevisi, E., Rumination time around calving: An early signal to detect cows at greater risk of disease. J. Dairy Sci. 97:6 (2014), 3635–3647.
Chelotti, J.O., Martinez-Rau, L.S., Ferrero, M., Vignolo, L.D., Galli, J.R., Planisich, A.M., Rufiner, H.L., Giovanini, L.L., Livestock feeding behaviour: A review on automated systems for ruminant monitoring. Biosyst. Eng. 246 (2024), 150–177.
Chelotti, J.O., Vanrell, S.R., Galli, J.R., Giovanini, L.L., Rufiner, H.L., A pattern recognition approach for detecting and classifying jaw movements in grazing cattle. Comput. Electron. Agric. 145 (2018), 83–91.
Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y., Learning phrase representations using RNN encoder–decoder for statistical machine translation. Proc. 2014 Conf. Empir. Methods Nat. Lang. Process, EMNLP, 2014, Association for Computational Linguistics., Doha, Qatar, 1724–1734.
Clark, C.E.F., Lyons, N.A., Millapan, L., Talukder, S., Cronin, G.M., Kerrisk, K.L., Garcia, S.C., Rumination and activity levels as predictors of calving for dairy cows. Animal 9:4 (2015), 691–695.
Dietterich, T.G., Ensemble methods in machine learning. Multiple Classifier Systems, 2000, Springer Berlin Heidelberg., 1–15.
Ferrero, M., Vignolo, L.D., Vanrell, S.R., Martinez-Rau, L.S., Chelotti, J.O., Galli, J.R., Giovanini, L.L., Rufiner, H.L., A full end-to-end deep approach for detecting and classifying jaw movements from acoustic signals in grazing cattle. Eng. Appl. Artif. Intell., 121, 2023, 106016.
Garcia-Ceja, E., Galván-Tejada, C.E., Brena, R., Multi-view stacking for activity recognition with sound and accelerometer data. Inf. Fusion 40 (2018), 45–56.
Giovanetti, V., Decandia, M., Molle, G., Acciaro, M., Mameli, M., Cabiddu, A., Cossu, R., Serra, M.G., Manca, C., Rassu, S.P.G., Dimauro, C., Automatic classification system for grazing, ruminating and resting behaviour of dairy sheep using a tri-axial accelerometer. Livest. Sci. 196 (2017), 42–48.
Greenwood, P.L., Paull, D.R., McNally, J., Kalinowski, T., Ebert, D., Little, B., Smith, D.V., Rahman, A., Valencia, P., Ingham, A.B., Bishop-Hurley, G.J., Use of sensor-determined behaviours to develop algorithms for pasture intake by individual grazing cattle. Crop. Pasture Sci., 68(12), 2017, 1091.
Hall, D.L., Llinas, J., An introduction to multisensor data fusion. Proc. IEEE 85:1 (1997), 6–23.
Herskin, M.S., Munksgaard, L., Ladewig, J., Effects of acute stressors on nociception, adrenocortical responses and behavior of dairy cows. Physiol. Behav. 83:3 (2004), 411–420.
Hinton, G., Deng, L., Yu, D., Dahl, G.E., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T.N., Kingsbury, B., Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Process. Mag. 29:6 (2012), 82–97.
Laca, E.A., Ungar, E.D., Seligman, N.G., Ramey, M.R., Demment, M.W., An integrated methodology for studying short-term grazing behaviour of cattle. Grass Forage Sci. 47:1 (1992), 81–90.
Laca, E.A., WallisDeVries, M.F., Acoustic measurement of intake and grazing behaviour of cattle. Grass Forage Sci. 55 (2000), 97–104.
Lecun, Y., Bottou, L., Bengio, Y., Haffner, P., Gradient-based learning applied to document recognition. Proc. IEEE 86:11 (1998), 2278–2324.
Li, Y., Shu, H., Bindelle, J., Xu, B., Zhang, W., Jin, Z., Guo, L., Wang, W., Classification and analysis of multiple cattle unitary behaviors and movements based on machine learning methods. Anim. (Basel), 12(9), 2022.
Li, G., Xiong, Y., Du, Q., Shi, Z., Gates, R.S., Classifying ingestive behavior of dairy cows via automatic sound recognition. Sensors, 21(15), 2021.
Liu, M., Wu, Y., Li, G., Liu, M., Hu, R., Zou, H., Wang, Z., Peng, Y., Classification of cow behavior patterns using inertial measurement units and a fully convolutional network model. J. Dairy Sci. 106:2 (2023), 1351–1359.
Martinez-Rau, L.S., Chelotti, J.O., Ferrero, M., Utsumi, S.A., Planisich, A.M., Vignolo, L.D., Giovanini, L.L., Rufiner, H.L., Galli, J.R., Daylong acoustic recordings of grazing and rumination activities in dairy cows. Sci Data, 10(1), 2023, 782.
Martinez-Rau, L.S., Chelotti, J.O., Vanrell, S.R., Galli, J.R., Utsumi, S.A., Planisich, A.M., Rufiner, L.L., A robust computational approach for jaw movement detection and classification in grazing cattle using acoustic signals. Comput. Electron. Agric., 192, 2022, 106569.
Martiskainen, P., Järvinen, M., Skön, J.-P., Tiirikainen, J., Kolehmainen, M., Mononen, J., Cow behaviour pattern recognition using a three-dimensional accelerometer and support vector machines. Appl. Anim. Behav. Sci. 119:1–2 (2009), 32–38.
Mesaros, A., Heittola, T., Virtanen, T., Metrics for polyphonic sound event detection. NATO Adv. Sci. Inst. Ser. E Appl. Sci., 6(6), 2016, 162.
Mesaros, A., Heittola, T., Virtanen, T., Plumbley, M.D., Sound event detection: A tutorial. IEEE Signal Process. Mag. 38:5 (2021), 67–83.
Morgan-Davies, C., Tesnière, G., Gautier, J., Jørgensen, G., González-García, E., Patsios, S., Sossidou, E., Keady, T., McClearn, B., Kenyon, G., Grøva, L., Decandia, L., Halachmi, I., Dwyer, C., Review: Exploring the use of precision livestock farming for small ruminant welfare management. Animal, 18, 2024, 101233 Selected keynote lectures of the 74th Annual Meeting of the European Federation of Animal Science (Lyon, France).
Mroueh, Y., Marcheret, E., Goel, V., Deep multimodal learning for audio-visual speech recognition. 2015 IEEE Int. Conf. Acoust. Speech Signal Process, ICASSP, 2015, 2130–2134.
Navon, S., Mizrach, A., Hetzroni, A., Ungar, E.D., Automatic recognition of jaw movements in free-ranging cattle, goats and sheep, using acoustic monitoring. Biosyst. Eng. 114:4 (2013), 474–483 (Special Issue: Sensing Technologies for Sustainable Agriculture).
Nunes, L., Ampatzidis, Y., Costa, L., Wallau, M., Horse foraging behavior detection using sound recognition techniques and artificial intelligence. Comput. Electron. Agric., 183, 2021, 106080.
Nweke, H.F., Teh, Y.W., Mujtaba, G., Al-garadi, M.A., Data fusion and multiple classifier systems for human activity detection and health monitoring: Review and open research directions. Inf. Fusion 46 (2019), 147–170.
Oudshoorn, F.W., Cornou, C., Hellwing, A.L.F., Hansen, H.H., Munksgaard, L., Lund, P., Kristensen, T., Estimation of grass intake on pasture for dairy cows using tightly and loosely mounted di- and tri-axial accelerometers combined with bite count. Comput. Electron. Agric. 99 (2013), 227–235.
Paudyal, S., Maunsell, F., Richeson, J., Risco, C., Donovan, D., Pinedo, P., Rumination time and monitoring of health disorders during early lactation. Animal 12:7 (2018), 1484–1492.
Pavlovic, D., Davison, C., Hamilton, A., Marko, O., Atkinson, R., Michie, C., Crnojević, V., Andonovic, I., Bellekens, X., Tachtatzis, C., Classification of cattle behaviours using Neck-Mounted Accelerometer-Equipped collars and convolutional neural networks. Sensors, 21(12), 2021.
Peng, Y., Kondo, N., Fujiura, T., Suzuki, T., Wulandari, H., Itoyama, E., Classification of multiple cattle behavior patterns using a recurrent neural network with long short-term memory and inertial measurement units. Comput. Electron. Agric. 157 (2019), 247–253.
Qiu, S., Zhao, H., Jiang, N., Wang, Z., Liu, L., An, Y., Zhao, H., Miao, X., Liu, R., Fortino, G., Multi-sensor information fusion based on machine learning for real applications in human activity recognition: State-of-the-art and research challenges. Inf. Fusion 80 (2022), 241–265.
Schuster, M., Paliwal, K.K., Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45:11 (1997), 2673–2681.
Serizel, R., Turpault, N., Shah, A., Salamon, J., Sound event detection in synthetic domestic environments. ICASSP 2020-2020 IEEE Int. Conf. Acoust. Speech Signal Process, ICASSP, 2020, IEEE, 86–90.
da Silva Santos, A., de Medeiros, V.W.C., Gonçalves, G.E., Monitoring and classification of cattle behavior: a survey. Smart Agric. Technol., 3, 2023, 100091.
Sokolova, M., Lapalme, G., A systematic analysis of performance measures for classification tasks. Inf. Process. Manag. 45:4 (2009), 427–437.
Son, C.-S., Kang, W.-S., Multivariate CNN model for human locomotion activity recognition with a wearable exoskeleton robot. Bioeng. (Basel), 10(9), 2023.
Spinsante, S., Angelici, A., Lundström, J., Espinilla, M., Cleland, I., Nugent, C., A mobile application for easy design and testing of algorithms to monitor physical activity in the workplace. Mob. Inf. Syst., 2016.
Tan, T.-H., Chang, Y.-L., Wu, J.-R., Chen, Y.-F., Alkhaleefah, M., Convolutional neural network with multihead attention for human activity recognition. EEE Internet Things J. 11 11:2 (2024), 3032–3043.
Tani, Y., Yokota, Y., Yayota, M., Ohtani, S., Automatic recognition and classification of cattle chewing activity by an acoustic monitoring method with a single-axis acceleration sensor. Comput. Electron. Agric. 92 (2013), 54–65.
Topaloglu, I., Barua, P.D., Yildiz, A.M., Keles, T., Dogan, S., Baygin, M., Gul, H.F., Tuncer, T., Tan, R.-S., Acharya, U.R., Explainable attention resnet18-based model for asthma detection using stethoscope lung sounds. Eng. Appl. Artif. Intell., 126, 2023, 106887.
Tzirakis, P., Trigeorgis, G., Nicolaou, M., Schuller, B., Zafeiriou, S., End-to-end multimodal emotion recognition using deep neural networks. IEEE J. Sel. Top. Signal Process., 2017.
Ungar, E., Ravid, N., Zada, T., Ben-Moshe, E., Yonatan, R., Baram, H., Genizi, A., The implications of compound chew–bite jaw movements for bite rate in grazing cattle. Appl. Anim. Behav. Sci. 98:3–4 (2006), 183–195.
Vanrell, S.R., Chelotti, J.O., Bugnon, L.A., Rufiner, H.L., Milone, D.H., Laca, E.A., Galli, J.R., Audio recordings dataset of grazing jaw movements in dairy cattle. Data Brief, 30, 2020, 105623.
Venkatesh, S., Moffat, D., Miranda, E.R., You only hear once: A YOLO-like algorithm for audio segmentation and sound event detection. NATO Adv. Sci. Inst. Ser. E Appl. Sci., 12(7), 2022, 3293.
Wang, K., Wu, P., Cui, H., Xuan, C., Su, H., Identification and classification for sheep foraging behavior based on acoustic signal and deep learning. Comput. Electron. Agric., 187(106275), 2021, 106275.
Wu, Y., Liu, M., Peng, Z., Liu, M., Wang, M., Peng, Y., Recognising cattle behaviour with deep residual bidirectional LSTM model using a wearable movement monitoring collar. Collect. FAO Agric., 12(8), 2022, 1237.
Yihan, C., Min, G., Zhiqiang, L., Sound event detection based on bidirectional temporal convolutional network and gated recurrent unit. 2021 20th International Conference on Ubiquitous Computing and Communications (IUCC/CIT/DSCI/SmartCNS), 2021, IEEE, pages 445–450.
Zhu, Z., Dai, W., Hu, Y., Li, J., Speech emotion recognition model based on Bi-GRU and focal loss. Pattern Recognit. 140 (2020), 358–365.
