An inertial human upper limb motion tracking method for robot programming by demonstration

[en] This paper proposes an inertial human motion tracking for robot programming by demonstration (PbD). An original element called heading reset is proposed to catch the drift around gravity direction. It is based on a hypothesis made on the human arm motion during a task demonstration. It is used to overcome the non-use of the magnetometer due to magnetic disturbances from robotic environment. This element is implemented in an orientation estimation algorithm and compared with three other IMU algorithms and a commercial MARG algorithm. The human arm trajectory is estimated through three IMUs sensors directly set on the arm to estimate the segment orientation (hand, forearm and arm). A specific inertial-2-segment procedure is presented as well as a procedure to estimate the transformation from human reference frame to task frame, necessary for a PbD process. Experimental tests, using a robot as a reference, have been conducted to validate the different part of the method. The heading reset and the orientation algorithm show good results. The inertial-2-segment procedure is shown to be robust. Finally, experimental tests on a human arm and physical robot validate the complete method.

Disciplines :

Engineering, computing & technology: Multidisciplinary, general & others

Author, co-author :

Pellois, Robin ; Université de Liège - ULiège > Aérospatiale et Mécanique (A&M)

Brüls, Olivier

Language :

English

Title :

An inertial human upper limb motion tracking method for robot programming by demonstration

Publication date :

11 July 2022

Journal title :

Robotics and Autonomous Systems

ISSN :

0921-8890

Publisher :

Elsevier BV

Peer reviewed :

Peer Reviewed verified by ORBi

Additional URL :

https://api.elsevier.com/content/article/PII:S0921889022001154?httpAccept=text/xml

Available on ORBi :

since 19 July 2022

Statistics

Number of views

61 (5 by ULiège)

Number of downloads

225 (2 by ULiège)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

Billard, A., Calinon, S., Dillmann, R., Schaal, S., Robot programing by demonstration. Springer Handbook of Robotics, 2008, Springer, 1371–1394, 10.1007/978-3-540-30301-5_60.
M. Ehrenmann, R. Zöllner, O. Rogalla, R. Dillmann, Programming service tasks in household environments by human demonstration, in: Proceedings of the IEEE International Workshop on Robot and Human Interactive Communication, 2002, pp. 460–467.
Argall, B.D., Chernova, S., Veloso, M., Browning, B., A survey of robot learning from demonstration. Robot. Auton. Syst. 57:5 (2009), 469–483, 10.1016/j.robot.2008.10.024.
Abbeel, P., Ng, A., Apprenticeship learning via inverse reinforcement learning. Proceedings of the Twenty-First International Conference on Machine Learning, 2004, 10.1145/1015330.1015430.
Breazeal, C., Berlin, M., Brooks, A., Gray, J., Thomaz, A.L., Using perspective taking to learn from ambiguous demonstrations. Robot. Auton. Syst. 54:5 (2006), 385–393, 10.1016/j.robot.2006.02.004.
Chernova, S., Veloso, M., Multi-thresholded approach to demonstration selection for interactive robot learning. Proceedings of the 3rd International Conference on Human-Robot Interaction, 2008, 225–232, 10.1145/1349822.1349852.
Vakanski, A., Mantegh, I., Irish, A., Janabi-Sharifi, F., Trajectory learning for robot programming by demonstration using hidden Markov model and dynamic time warping. IEEE Trans. Syst. Man Cybern. B 42:4 (2012), 1039–1052, 10.1109/TSMCB.2012.2185694.
Ogino, M., Toichi, H., Yoshikawa, Y., Asada, M., Interaction rule learning with a human partner based on an imitation faculty with a simple visuo-motor mapping. Robot. Auton. Syst. 54:5 (2006), 414–418, 10.1016/j.robot.2006.01.005.
U. Nehmzow, O. Akanyeti, C. Weinrich, T. Kyriacou, S.A. Billings, Robot programming by demonstration through system identification, in: Proceedings of the IEEE International Conference on Intelligent Robots and Systems, 2007, pp. 801–806.
Calinon, S., Billard, A., Teaching a humanoid robot to recognize and reproduce social cues. Proceedings of the 15th IEEE International Symposium on Robot and Human Interactive Communication, 2006, 346–351, 10.1109/ROMAN.2006.314458.
Ogawara, K., Takamatsu, J., Kimura, H., Ikeuchi, K., Generation of a task model by integrating multiple observations of human demonstrations. Proceedings of the IEEE International Conference on Robotics and Automation (Cat. No.02CH37292), 2002, 1545–1550, 10.1109/ROBOT.2002.1014763.
Ijspeert, A., Nakanishi, J., Schaal, S., Trajectory formation for imitation with nonlinear dynamical systems. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems. Expanding the Societal Role of Robotics in the Next Millennium (Cat. No.01CH37180), 2001, 10.1109/IROS.2001.976259.
Li, J., Wang, J., Wang, S., Yang, C., Human–robot skill transmission for mobile robot via learning by demonstration. Neural Comput. Appl., 0123456789, 2021, 10.1007/s00521-021-06449-x.
Chen, Z., Li, J., Wang, S., Wang, J., Ma, L., Flexible gait transition for six wheel-legged robot with unstructured terrains. Robot. Auton. Syst., 150, 2022, 103989, 10.1016/j.robot.2021.103989.
Kaiser, J., Melbaum, S., Tieck, J.C.V., Roennau, A., Butz, M.V., Dillmann, R., Learning to reproduce visually similar movements by minimizing event-based prediction error. Proceedings of the IEEE RAS and EMBS International Conference on Biomedical Robotics and Biomechatronics, Vol. 2018-August, 2018, 260–267, 10.1109/BIOROB.2018.8487959.
Kendon, A., Gesture: Visible Action as Utterance. 2004, Cambridge University Press, 400.
Mosher, J.A., The Essentials of Effective Gesture, for Students of Public Speaking. 1916, The Macmillan company, New York, 214.
Kyrarini, M., Haseeb, M.A., Ristić-Durrant, D., Gräser, A., Robot learning of object manipulation task actions from human demonstrations. Facta Univ. Ser.: Mech. Eng., 15(2), 2017, 217.
Johnson, M., Demiris, Y., Abstraction in recognition to solve the correspondence problem for robot imitation. Taros, 2004, 63–70 URL http://cswww.essex.ac.uk/technical-reports/2004/csm415/Johnson_Demiris.pdf.
Ferreira, M., Costa, P., Rocha, L., Moreira, A., Stereo-based real-time 6-dof work tool tracking for robot programing by demonstraiton.pdf. Int. J. Adv. Manuf. Technol., 2014.
M. Field, D. Stirling, F. Naghdy, Z. Pan, Motion capture in robotics review, in: Proceedings of the IEEE International Conference on Control and Automation, 2009, pp. 1697–1702.
Filippeschi, A., Schmitz, N., Miezal, M., Bleser, G., Ruffaldi, E., Stricker, D., Survey of motion tracking methods based on inertial sensors: A focus on upper limb human motion. Sensors (Switzerland), 17(6), 2017, 1257, 10.3390/s17061257.
Grisbrook, T., Imms, C., Walmsley, C.P., Campbell, A.C., Elliott, C., Williams, S.A., Measurement of upper limb range of motion using wearable sensors: A systematic review. Sports Med. Open 4:1 (2018), 53–75, 10.1186/s40798-018-0167-7.
Wolf, M.T., Assad, C., Vernacchia, M.T., Fromm, J., Jethani, H.L., Gesture-based robot control with variable autonomy from the JPL BioSleeve. Proceedings of the IEEE International Conference on Robotics and Automation, 2013, 1160–1165, 10.1109/ICRA.2013.6630718.
Ijspeert, A., Nakanishi, J., Schaal, S., Nonlinear dynamical systems for imitation with humanoid robots (december). 2003, 1398–1403.
S. Calinon, A. Billard, Incremental learning of gestures by imitation in a humanoid robot, in: Proceeding of the ACM/IEEE International Conference on Human-Robot Interaction - HRI ’07, 2007, p. 255.
Pinto, V.H., Amorim, A., Rocha, L., Moreira, A.P., Enhanced performance real-time industrial robot programming by demonstration using stereoscopic vision and an IMU sensor. IEEE International Conference on Autonomous Robot Systems and Competitions, 2020, 10.1109/ICARSC49921.2020.9096200.
W.-Y. Go, J.-H. Kim, Wireless remote control of robot dual arms and hands using inertial measurement units, in: Proc. International Conference on Robot Intelligence Technology and Applications (RiTA), Beijing, 2014, pp. 759–768.
Chen, X., Human motion analysis with wearable inertial sensors. IEEE Sens. J. 16:22 (2016), 7821–7834, 10.1109/JSEN.2016.2609392.
Miezal, M., Taetz, B., Bleser, G., On inertial body tracking in the presence of model calibration errors. Sensors (Switzerland), 16(7), 2016, 1132, 10.3390/s16071132.
Cheng, P., Oelmann, B., Joint-angle measurement using accelerometers and gyroscopes - a survey. IEEE Trans. Instrum. Meas. 59:2 (2010), 404–414, 10.1109/TIM.2009.2024367.
Gallagher, A., Matsuoka, Y., Wei-Tech Ang, G., An efficient real-time human posture tracking algorithm using low-cost inertial and magnetic sensors. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2004, 10.1109/IROS.2004.1389860.
Luinge, H.J., Veltink, P.H., Measuring orientation of human body segments using miniature gyroscopes and accelerometers. Med. Biol. Eng. Comput. 43:2 (2005), 273–282, 10.1007/bf02345966.
Xiaoping, Y., Bachmann, E.R., Design, implementation, and experimental results of a quaternion-based kalman filter for human body motion tracking. IEEE Trans. Robot. 22:6 (2006), 1216–1227, 10.1109/TRO.2006.886270.
Tian, Y., Tan, J., A fast adaptive-gain orientation filter of inertial/magnetic data for human motion tracking in free-living environments. J. Navig. 71:6 (2018), 1478–1491, 10.1017/S0373463318000231.
Zhou, H., Hu, H., Reducing drifts in the inertial measurements of wrist and elbow positions. IEEE Trans. Instrum. Meas. 59:3 (2010), 575–585, 10.1109/TIM.2009.2025065.
Peppoloni, L., Filippeschi, A., Ruffaldi, E., Avizzano, C.A., A novel 7 degrees of freedom model for upper limb kinematic reconstruction based on wearable sensors. Proceedings of the IEEE 11th International Symposium on Intelligent Systems and Informatics (SISY), 2013, 10.1109/SISY.2013.6662551.
Palermo, E., Rossi, S., Marini, F., Patanè, F., Cappa, P., Experimental evaluation of accuracy and repeatability of a novel body-to-sensor calibration procedure for inertial sensor-based gait analysis. Measurement 52:1 (2014), 145–155, 10.1016/j.measurement.2014.03.004.
Robert-lachaine, X., Mecheri, H., Larue, C., Bellefeuille, S., Single pose calibration with inertial measurement units. Gait Posture 54 (2017), 80–86, 10.1016/j.gaitpost.2017.02.029.
De Vries, W., Veeger, H.E., Cutti, A.G., Baten, C., van der Helm, F.C., Functionally interpretable local coordinate systems for the upper extremity using inertial and magnetic measurement systems. J. Biomech. 43:10 (2010), 1983–1988, 10.1016/j.jbiomech.2010.03.007.
Zabat, M., Ababou, A., Ababou, N., Dumas, R., IMU-based sensor-to-segment multiple calibration for upper limb joint angle measurement—a proof of concept. Med. Biol. Eng. Comput. 57 (2019), 2449–2460, 10.1007/s11517-019-02033-7.
C. Jiang, M. Fahad, Y. Guo, J. Yang, Y. Chen, Robot-assisted human indoor localization using the Kinect sensor and smartphones, in: Proceedings of the IEEE International Conference on Intelligent Robots and Systems, 2014.
S.S. Ghidary, Y. Nakata, T. Takamori, M. Hattori, Localization and approaching to the human by mobile home robot, in: Proceedings of the IEEE International Workshop on Robot and Human Interactive Communication, 2000.
Faessler, M., Mueggler, E., Schwabe, K., Scaramuzza, D., A monocular pose estimation system based on infrared LEDs. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2014, 10.1109/ICRA.2014.6906962.
Gromov, B., Gambardella, L.M., Giusti, A., Robot identification and localization with pointing gestures. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018, 3921–3928, 10.1109/IROS.2018.8594174.
Roetenberg, D., Henk, L., Peter, V., Inertial and magnetic sensing of human movement near ferromagnetic materials. Proceedings of the Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003, 10.1109/ISMAR.2003.1240714.
Ligorio, G., Sabatini, A.M., Dealing with magnetic disturbances in human motion capture: A survey of techniques. Micromachines, 7(3), 2016, 43, 10.3390/mi7030043.
De Vries, W., Veeger, H., Baten, C., van der Helm, F.C., Magnetic distortion in motion labs, implications for validating inertial magnetic sensors. Gait Posture 29:4 (2009), 535–541, 10.1016/j.gaitpost.2008.12.004.
Zihajehzadeh, S., Park, E.J., A novel biomechanical model-aided IMU/UWB fusion for magnetometer-free lower body motion capture. IEEE Trans. Syst. Man Cybern. 47:6 (2017), 927–938, 10.1109/TSMC.2016.2521823.
Kok, M., Hol, J.D., Schön, T.B., Using inertial sensors for position and orientation estimation. Found. Trends® Signal Process. 11:1–2 (2017), 1–153, 10.1561/2000000094.
López-nava, I.H., Muñoz-Meléndez, A., Wearable inertial sensors for human motion analysis : A review. IEEE Sens. J. 16:22 (2016), 7821–7834, 10.1109/JSEN.2016.2609392.
Mahony, R., Hamel, T., Pflimlin, J.-M., Non-linear complementary filters on the special orthogonal group. IEEE Trans. Automat. Control 53:5 (2008), 1203–1218, 10.1109/TAC.2008.923738.
Fourati, H., Heterogeneous data fusion algorithm for pedestrian navigation via foot-mounted inertial measurement unit and complementary filter. IEEE Trans. Instrum. Meas., 64(1), 2015, 221.229, 10.1109/TIM.2014.2335912.
Euston, M., Coote, P., Mahony, R., Kim, J., Hamel, T., A complementary filter for attitude estimation of a fixed-wing UAV. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, 2008, 340–345, 10.1109/IROS.2008.4650766.
Kim, A., Golnaraghi, M., A quaternion-based orientation estimation algorithm using an inertial measurement unit. Proceedings of the Position Location and Navigation Symposium, 2004, 10.1109/PLANS.2004.1309003.
Choukroun, D., Bar-Itzhack, I.Y., Oshman, Y., Novel quaternion Kalman filter. IEEE Trans. Aerosp. Electron. Syst. 42:1 (2006), 174–190, 10.1109/TAES.2006.1603413.
Barrau, A., Bonnabel, S., Invariant Kalman filtering. Annu. Rev. Control Robot. Auton. Syst. 1 (2018), 237–257, 10.1146/annurev-control-060117-105010.
S. Madgwick, An Efficient Orientation Filter for Inertial and Inertial/magnetic Sensor Arrays, Technical Report X-Io, 2010.
Brückner, H.P., Spindeldreier, C., Blume, H., Schoonderwaldt, E., Altenmüller, E., Evaluation of inertial sensor fusion algorithms in grasping tasks using real input data: Comparison of computational costs and root mean square error. Proceedings of the Ninth International Conference on Wearable and Implantable Body Sensor Networks, 2012, 10.1109/BSN.2012.9.
Hu, W., Charry, E., Umer, M., Ronchi, A., Taylor, S., An inertial sensor system for measurements of tibia angle with applications to knee valgus/varus detection. Proceedings of the IEEE Ninth International Conference on Intelligent Sensors, Sensor Networks and Information Processing (ISSNIP), 2014, 10.1109/ISSNIP.2014.6827603.
Bandini, A., Zariffa, J., Analysis of the hands in egocentric vision: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 2020, 1, 10.1109/tpami.2020.2986648 arXiv:1912.10867.
Rodriguez, I., Peinado, M., Boulic, R., Meziat, D., Bringing the human arm reachable space to a virtual environment for its analysis. Proceedings - IEEE International Conference on Multimedia and Expo, Vol. 1, 2003, 229–232, 10.1109/ICME.2003.1220896 (August).
Clément, J., Raison, M., Rouleau, D.M., Reproducibility analysis of upper limbs reachable workspace, and effects of acquisition protocol, sex and hand dominancy. J. Biomech. 68 (2018), 58–64, 10.1016/j.jbiomech.2017.12.010.
Beravs, T., Podobnik, J., Munih, M., Three-axial accelerometer calibration using Kalman filter covariance matrix for online estimation of optimal sensor orientation. IEEE Trans. Instrum. Meas. 61:9 (2012), 2501–20511, 10.1109/TIM.2012.2187360.
Tedaldi, D., Pretto, A., Menegatti, E., A robust and easy to implement method for IMU calibration without external equipments. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2014, 10.1109/ICRA.2014.6907297.
Andrle, M.S., Crassidis, J.L., Geometric integration of quaternions. J. Guid. Control Dyn. 36:6 (2013), 1762–1772, 10.2514/1.58558.
Arun, K.S., Huang, T.S., Blostein, S.D., Least-squares fitting of two 3-d point sets. IEEE Trans. Pattern Anal. Mach. Intell. PAMI-9:5 (1987), 698–700, 10.1109/TPAMI.1987.4767965.
Paulich, M., Schepers, M., Rudigkeit, N., Bellusci, G., Xsens MTw awinda: Miniature wireless inertial-magnetic motion tracker for highly accurate 3D kinematic applications. 2013 Whitepaper.
Fourati, H., Manamanni, N., Afilal, L., Handrich, Y., Complementary observer for body segments motion capturing by inertial and magnetic sensors. IEEE/ASME Trans. Mechatronics 19:1 (2014), 149–157, 10.1109/TMECH.2012.2225151.
Amabile, C., Bull, A.M.J., Kedgley, A.E., The centre of rotation of the shoulder complex and the effect of normalisation. J. Biomech. 49:9 (2016), 1938–1943, 10.1016/j.jbiomech.2016.03.035.
Campbell, A.C., Alderson, J.A., Elliott, B.C., Lloyd, D.G., Effects of different technical coordinate system definitions on the three dimensional representation of the glenohumeral joint centre. Med. Biol. Eng. Comput. 47:5 (2009), 543–550, 10.1007/s11517-009-0467-7.