Non-Verbal Human-Robot Interaction Using Neural Network for The Application of Service Robot

Authors

DOI:

https://doi.org/10.31436/iiumej.v24i1.2577

Keywords:

Attentive Recognition Model (ARM), Long-Short-Term Memory (LSTM), Human Robot Interaction (HRI)

Abstract

Service robots are prevailing in many industries to assist humans in conducting repetitive tasks, which require a natural interaction called Human Robot Interaction (HRI). In particular, nonverbal HRI plays an important role in social interactions, which highlights the need to accurately detect the subject's attention by evaluating the programmed cues. In this paper, a conceptual attentiveness model algorithm called Attentive Recognition Model (ARM) is proposed to recognize a person’s attentiveness, which improves the accuracy of detection and subjective experience during nonverbal HRI using three combined detection models: face tracking, iris tracking and eye blinking. The face tracking model was trained using a Long Short-Term Memory (LSTM) neural network, which is based on deep learning. Meanwhile, the iris tracking and eye blinking use a mathematical model. The eye blinking model uses a random face landmark point to calculate the Eye Aspect Ratio (EAR), which is much more reliable compared to the prior method, which could detect a person blinking at a further distance even if the person was not blinking. The conducted experiments for face and iris tracking were able to detect direction up to 2 meters. Meanwhile, the tested eye blinking model gave an accuracy of 83.33% at up to 2 meters. The overall attentive accuracy of ARM was up to 85.7%. The experiments showed that the service robot was able to understand the programmed cues and hence perform certain tasks, such as approaching the interested person.

ABSTRAK: Robot perkhidmatan lazim dalam banyak industri untuk membantu manusia menjalankan tugas berulang, yang memerlukan interaksi semula jadi yang dipanggil Interaksi Robot Manusia (HRI). Khususnya, HRI bukan lisan memainkan peranan penting dalam interaksi sosial, yang menonjolkan keperluan untuk mengesan perhatian subjek dengan tepat dengan menilai isyarat yang diprogramkan. Dalam makalah ini, algoritma model perhatian konseptual yang dipanggil Model Pengecaman Perhatian (ARM) dicadangkan untuk mengenali perhatian seseorang, yang meningkatkan ketepatan pengesanan dan pengalaman subjektif semasa HRI bukan lisan menggunakan tiga model pengesanan gabungan: pengesanan muka, pengesanan iris dan mata berkedip. . Model penjejakan muka telah dilatih menggunakan rangkaian saraf Memori Jangka Pendek Panjang (LSTM), yang berdasarkan pembelajaran mendalam. Manakala, pengesanan iris dan mata berkelip menggunakan model matematik. Model mata berkelip menggunakan titik mercu tanda muka rawak untuk mengira Nisbah Aspek Mata (EAR), yang jauh lebih dipercayai berbanding kaedah sebelumnya, yang boleh mengesan seseorang berkelip pada jarak yang lebih jauh walaupun orang itu tidak berkelip. Eksperimen yang dijalankan untuk pengesanan muka dan iris dapat mengesan arah sehingga 2 meter. Sementara itu, model berkelip mata yang diuji memberikan ketepatan 83.33% sehingga 2 meter. Ketepatan perhatian keseluruhan ARM adalah sehingga 85.7%. Eksperimen menunjukkan bahawa robot perkhidmatan dapat memahami isyarat yang diprogramkan dan seterusnya melaksanakan tugas tertentu, seperti mendekati orang yang berminat.

Downloads

Download data is not yet available.

Metrics

Metrics Loading ...

References

Haider, MH, Wang Z, Khan AA, Ali H, Zheng H, Usman S, Kumar R, Bhutta MUM, Zhi P. (2022) Robust mobile robot navigation in cluttered environments based on hybrid adaptive neuro-fuzzy inference and sensor fusion. Journal of King Saud University-Computer and Information Sciences. https://doi.org/10.1016/j.jksuci.2022.08.031 DOI: https://doi.org/10.1016/j.jksuci.2022.08.031

Hacene N, Mendil B. (2021) Behavior-based autonomous navigation and formation control of mobile robots in unknown cluttered dynamic environments with dynamic target tracking. International Journal of Automation and Computing, 18: 766-786. https://doi.org/10.1007/s11633-020-1264-x DOI: https://doi.org/10.1007/s11633-020-1264-x

Sigalas M, Pateraki M, Trahanias P. (2015, July) Visual estimation of attentive cues in HRI: the case of torso and head pose. International Conference on Computer Vision Systems, vol (9163): pp 375-388. https://doi.org/10.1007/978-3-319-20904-3_34 DOI: https://doi.org/10.1007/978-3-319-20904-3_34

D’Eusanio A, Simoni A, Pini S, Borghi G, Vezzani R, Cucchiara R. (2020, November) A Transformer-Based Network for Dynamic Hand Gesture Recognition. 2020 International Conference on 3D Vision (3DV): pp. 623-632. https://doi.org/10.1109/3DV50981.2020.00072 DOI: https://doi.org/10.1109/3DV50981.2020.00072

Lombardi M, Maiettini E, De Tommaso D, Wykowska A, Natale L. (2022) Toward an Attentive Robotic Architecture: Learning-Based Mutual Gaze Estimation in Human–Robot Interaction. Frontiers in Robotics and AI, 9, 770165. https://doi.org/10.3389/frobt.2022.770165 DOI: https://doi.org/10.3389/frobt.2022.770165

Hömke P, Holler J, Levinson SC. (2018) Eye blinks are perceived as communicative signals in human face-to-face interaction. PLoS ONE, 13(12): e0208030. https://doi.org/10.1371/journal.pone.0208030 DOI: https://doi.org/10.1371/journal.pone.0208030

Keiling H. (2019, February 28) 9 Types of Nonverbal Communication and How To Understand Them. Indeed Career Guide [https://www.indeed.com/career-advice/career-development/nonverbal-communication-skills#:~:text=Nonverbal%20communication%20is%20important%20because]

Khan ZH, Siddique A, Lee CW. (2020) Robotics Utilization for Healthcare Digitization in Global COVID-19 Management. International Journal of Environmental Research and Public Health, 17(11): 1-23. https://doi.org/10.3390/ijerph17113819 DOI: https://doi.org/10.3390/ijerph17113819

Timm F, Barth E. (2011) Accurate eye centre localisation by means of gradients. VISAPP 2011 - Proceedings of the International Conference on Computer Vision Theory and Application, 125-130.

Li J, Liu R, Kong D, Wang S, Wang L, Yin B, Gao R. (2021) Attentive 3D-Ghost Module for Dynamic Hand Gesture Recognition with Positive Knowledge Transfer. Computational Intelligence and Neuroscience, 2021(5044916): pp 1–12. https://doi.org/10.1155/2021/5044916 DOI: https://doi.org/10.1155/2021/5044916

Attiah AZ, Khairullah EF. (2021) Eye-Blink Detection System for Virtual Keyboard. National Computing Colleges Conference (NCCC), pp 1-6. https://doi.org/10.1109/NCCC49330.2021.9428797. DOI: https://doi.org/10.1109/NCCC49330.2021.9428797

Shen Z, Elibol A, Chong NY. (2020) Understanding nonverbal communication cues of human personality traits in human-robot interaction. IEEE/CAA Journal of Automatica Sinica, 7(6): 1465-1477. https://doi.org/10.1109/JAS.2020.1003201 DOI: https://doi.org/10.1109/JAS.2020.1003201

Palinko O, Rea F, Sandini G, Sciutti A. (2016, October) Robot reading human gaze: Why eye tracking is better than head tracking for human-robot collaboration. International Conference on Intelligent Robots and Systems (IROS), pp. 5048-5054. https://doi.org/ 10.1109/IROS.2016.7759741 DOI: https://doi.org/10.1109/IROS.2016.7759741

Chandra B, Sharon HLU, Vignesh CP, Sriram R. (2020) Eye Blink Controlled Virtual Interface Using Opencv And Dlib. European Journal of Molecular & Clinical Medicine, 7(8), pp 2119–2126. https://ejmcm.com/article_4542.html

Pasternak K, Wu Z, Visser U, Lisetti C. (2021) Let’s be friends! A rapport-building 3D embodied conversational agent for the Human Support Robot. ArXiv preprint ArXiv:2103.04498. https://doi.org/10.48550/arXiv.2103.04498

Saran A, Majumdar S, Short ES, Thomaz A, Niekum S. (2018, October 1) Human Gaze Following for Human-Robot Interaction. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 8615-8621. https://doi.org/10.1109/IROS.2018.8593580 DOI: https://doi.org/10.1109/IROS.2018.8593580

Saunderson S, Nejat G. (2019) How Robots Influence Humans: A Survey of Nonverbal Communication in Social Human–Robot Interaction. International Journal of Social Robotics, 11(4): 575–608. https://doi.org/10.1007/s12369-019-00523-0 DOI: https://doi.org/10.1007/s12369-019-00523-0

Li X. (2021) Design and implementation of human motion recognition information processing system based on LSTM recurrent neural network algorithm. Computational Intelligence and Neuroscience. https://doi.org/10.1155/2021/3669204 DOI: https://doi.org/10.1155/2021/3669204

Laghrissi FE, Douzi S, Douzi K, Hssina B. (2021) Intrusion detection systems using long short-term memory (LSTM). Journal of Big Data, 8(1): 1-16. https://doi.org/10.1186/s40537-021-00448-4 DOI: https://doi.org/10.1186/s40537-021-00448-4

Downloads

Published

2023-01-04

How to Cite

Soomro, Z. A., BIN SHAMSUDIN, A. U., Abdul Rahim, R., Adrianshah, A., & Mohd Hazeli. (2023). Non-Verbal Human-Robot Interaction Using Neural Network for The Application of Service Robot. IIUM Engineering Journal, 24(1), 301–318. https://doi.org/10.31436/iiumej.v24i1.2577

Issue

Section

Mechatronics and Automation Engineering