Design and Implementation of a Deep Learning-Based Hand Gesture Recognition System for Rehabilitation Internet-of-Things (RIoT) Environments Using MediaPipe

Nurul Hanis Mohd Dhuzuki; Ahmad Anwar Zainuddin; Nur Anis Sofea Kamarul Zaman; Alin Nur Maisarah Ahmad Razmi; Wonderful Shammah Kaitane; Asmarani Ahmad Puzi; Mohd Naqiuddin  Johar; Maslina Yazid; Nor Azlin Mohd Nordin; Shahrul Naim Sidek; Hasan Firdaus Mohd Zaki

doi:10.31436/iiumej.v26i1.3455

Authors

Nurul Hanis Mohd Dhuzuki International Islamic University Malaysia https://orcid.org/0009-0000-7177-3535
Ahmad Anwar Zainuddin International Islamic University Malaysia
Nur Anis Sofea Kamarul Zaman International Islamic University Malaysia https://orcid.org/0009-0004-5542-7471
Alin Nur Maisarah Ahmad Razmi International Islamic University Malaysia https://orcid.org/0009-0005-3485-2724
Wonderful Shammah Kaitane MILA University
Asmarani Ahmad Puzi International Islamic University Malaysia https://orcid.org/0000-0001-8796-7289
Mohd Naqiuddin Johar Hospital Putrajaya
Maslina Yazid Hospital Shah Alam
Nor Azlin Mohd Nordin National University of Malaysia https://orcid.org/0000-0002-0038-1670
Shahrul Naim Sidek International Islamic University Malaysia https://orcid.org/0000-0002-3204-1347
Hasan Firdaus Mohd Zaki International Islamic University Malaysia https://orcid.org/0000-0002-7209-4355

DOI:

https://doi.org/10.31436/iiumej.v26i1.3455

Keywords:

Rehabilitation Internet-of-Things (RIOT), MediaPipe, Deep Learning (DL), hand gesture recognition, Artificial Intelligence (AI)

Abstract

Frequent hospital visits for hand rehabilitation exercises, such as strengthening and opposition exercises, present significant challenges, especially for patients in remote areas. This paper addresses this problem by developing a Rehabilitation Internet-of-Things (RIOT) system that utilizes MediaPipe with its pre-trained Deep Learning (DL) to deliver real-time feedback during hand rehabilitation exercises alongside Web Assembly (WASM) for efficient processing. The system's objective is to provide precise, real-time tracking of hand movements, enabling patients to perform exercises at home by maintaining an optimal distance between the camera and hand placement, ensuring ideal room lighting conditions across IoT devices such as mobile phones' front cameras and webcams, while healthcare professionals remotely monitor their progress. The methodology involves the integration of MediaPipe for detecting hand landmarks and adaptive sensitivity algorithms to ensure reliable recognition across different environments, such as varying lighting and hand positions. Future work could incorporate additional deep-learning models like CNNs and RNNs to enhance gesture classification accuracy. Several limitations, including latency and distance sensitivity, are addressed in this system with edge computing alongside adaptive algorithms. The key contributions of this research are as follows: First, developing a real-time and cost-effective solution for remote stroke rehabilitation. Second, accuracy is improved by integrating MediaPipe with deep learning techniques. Lastly, latency issues and accuracy challenges at extended distances are alleviated by employing innovative calibration methods and adaptive adjustments. Initial trials demonstrate promising results, though further testing is required under real-world conditions to validate the system's effectiveness fully.

ABSTRAK: Perjalanan yang kerap ke hospital untuk latihan pemulihan tangan, seperti latihan rawatan fisioterapi telah memberikan cabaran yang besar bagi pesakit yang tinggal di pedalaman. Sistem Pemulihan Internet Benda (RIOT) menggunakan MediaPipe bersama Deep Learning (DL) yang telah dilatih untuk memberikan maklum balas masa nyata semasa latihan pemulihan tangan, serta Web Assembly (WASM) untuk pemprosesan yang cekap, sebagai penyelesaian. Tujuan sistem ini adalah untuk menyediakan penjejakan pergerakan tangan yang tepat dalam masa nyata, yang mampu dijalankan latihan di rumah dengan pemantauan pegawai perubatan untuk meneliti kemajuan mereka dari jarak jauh. Metodologi melibatkan penyatuan MediaPipe untuk mengesan titik penting pada tangan dan algoritma kepekaan suaian untuk memastikan pengiktirafan yang boleh dipercayai dalam pelbagai persekitaran, seperti pencahayaan dan kedudukan tangan. Lonjakan bagi kajian in adalah dapat menggabungkan model DL seperti CNNs dan RNNs untuk meningkatkan ketepatan dan penyusunan isyarat. Sistem ini juga dapat mengurangkan masalah masa pendam dan perubahab jara dengan melaksanakan edge computing dan penyesuaian algoritma. Sumbangan utama kajian ini termasuklah sistem masa nyata yang kos efektif untuk pemulihan strok jarak jauh, peningkatan ketepatan melalui gabungan MediaPipe dan model DL, dan pengurangan masalah masa pendam dan ketepatan jarak yang lebih jauh melalui tentuukur dan suaian algoritma. Percubaan awal telah menunjukkan hasil yang bagus. Walau bagaimanapun, ujian lanjut masih perlu dibuat dalam dunia sebenar untuk menjamin keberkesanan sistem secara keseluruhan.

Downloads

Download data is not yet available.

Metrics

Metrics Loading ...

References

Y. J. Choo and M. C. Chang, “Use of Machine Learning in Stroke Rehabilitation: A Narrative Review,” Brain Neurorehabilitation, vol. 15, no. 3, Nov. 2022, doi: 10.12786/bn.2022.15.e26. DOI: https://doi.org/10.12786/bn.2022.15.e26

W. Zhang, C. Su, and C. He, “Rehabilitation Exercise Recognition and Evaluation Based on Smart Sensors With Deep Learning Framework,” IEEE Access, vol. 8, pp. 77561–77571, 2020, doi: 10.1109/ACCESS.2020.2989128. DOI: https://doi.org/10.1109/ACCESS.2020.2989128

X. Li and J. Zhong, “Upper Limb Rehabilitation Robot System Based on Internet of Things Remote Control,” IEEE Access, vol. 8, pp. 154461–154470, 2020, doi: 10.1109/ACCESS.2020.3014378. DOI: https://doi.org/10.1109/ACCESS.2020.3014378

S. H. Chae, Y. Kim, K.-S. Lee, and H.-S. Park, “Development and Clinical Evaluation of a Web-Based Upper Limb Home Rehabilitation System Using a Smartwatch and Machine Learning Model for Chronic Stroke Survivors: Prospective Comparative Study,” JMIR MHealth UHealth, vol. 8, no. 7, p. e17216, Jul. 2020, doi: 10.2196/17216. DOI: https://doi.org/10.2196/17216

on behalf of the Writing Expert Group of Expert Consensus on Clinical Application of IOT Medical Technology in the Rehabilitation of Chronic Obstructive Pulmonary Disease, the Respiratory Disease Rehabilitation Professional Committee of China Medical Education Association et al., “Clinical guidelines on the application of Internet of Things (IOT) medical technology in the rehabilitation of chronic obstructive pulmonary disease,” J. Thorac. Dis., vol. 13, no. 8, pp. 4629–4637, Aug. 2021, doi: 10.21037/jtd-21-670. DOI: https://doi.org/10.21037/jtd-21-670

J. T. Kelly, K. L. Campbell, E. Gong, and P. Scuffham, “The Internet of Things: Impact and Implications for Health Care Delivery,” J. Med. Internet Res., vol. 22, no. 11, p. e20135, Nov. 2020, doi: 10.2196/20135. DOI: https://doi.org/10.2196/20135

A. A. Zainuddin, N. H. M. Dhuzuki, A. A. Puzi, M. N. Johar, and M. Yazid, “Calibrating Hand Gesture Recognition for Stroke Rehabilitation Internet-of-Things (RIOT) Using MediaPipe in Smart Healthcare Systems,” Int. J. Adv. Comput. Sci. Appl., vol. 15, no. 7, 2024, doi: 10.14569/IJACSA.2024.0150756. DOI: https://doi.org/10.14569/IJACSA.2024.0150756

N. H. Mohd Dhuzuki, A. A. Zainuddin, S. I. Kamarudin, D. Handayani, K. Subramaniam, and Mohd. I. Mohd. Tamrin, “Web-Based Medical Information System for Stroke Rehabilitation Intemet-of-Things (RIOT) Patients: A Prototype,” in 2024 IEEE International Conference on Artificial Intelligence in Engineering and Technology (IICAIET), Kota Kinabalu, Malaysia: IEEE, Aug. 2024, pp. 326–330. doi: 10.1109/IICAIET62352.2024.10729958. DOI: https://doi.org/10.1109/IICAIET62352.2024.10729958

“MediaPipe Solutions guide | Google AI Edge,” Google AI for Developers. Accessed: Sep. 20, 2024. [Online]. Available: https://ai.google.dev/edge/mediapipe/solutions/guide

B. Saravi et al., “Clinical and radiomics feature-based outcome analysis in lumbar disc herniation surgery,” BMC Musculoskelet. Disord., vol. 24, no. 1, p. 791, Oct. 2023, doi: 10.1186/s12891-023-06911-y. DOI: https://doi.org/10.1186/s12891-023-06911-y

S. Rajayyan and S. M. M. Mustafa, “Prediction of dementia using machine learning model and performance improvement with cuckoo algorithm,” Int. J. Electr. Comput. Eng. IJECE, vol. 13, no. 4, 2023, doi: 10.11591/ijece.v13i4.pp4623-4632. DOI: https://doi.org/10.11591/ijece.v13i4.pp4623-4632

R. Magar, L. Ghule, J. Li, Y. Zhao, and A. B. Farimani, “FaultNet: A Deep Convolutional Neural Network for Bearing Fault Classification,” IEEE Access, vol. 9, 2021, doi: 10.1109/access.2021.3056944. DOI: https://doi.org/10.1109/ACCESS.2021.3056944

C. Lugaresi et al., “MediaPipe: A Framework for Perceiving and Processing Reality”.

D. Parashar, “A Deep Learning-Based Approach for Hand Sign Recognition Using CNN Architecture,” Rev. Intell. Artif., 2023, doi: 10.18280/ria.370414. DOI: https://doi.org/10.18280/ria.370414

J. A. Díez, A. Blanco, J. M. Catalán, F. J. Badesa, L. D. Lledó, and N. García-Aracil, “Hand exoskeleton for rehabilitation therapies with integrated optical force sensor,” Adv. Mech. Eng., vol. 10, no. 2, p. 168781401775388, Feb. 2018, doi: 10.1177/1687814017753881. DOI: https://doi.org/10.1177/1687814017753881

W. Zhang, T. Zhao, J. Zhang, and Y. Wang, “LST-EMG-Net: Long Short-Term Transformer Feature Fusion Network for sEMG Gesture Recognition,” Front. Neurorobotics, 2023, doi: 10.3389/fnbot.2023.1127338. DOI: https://doi.org/10.3389/fnbot.2023.1127338

Z. Slimane, K. Lakhdari, and D. Souhila Korti, “Enhancing Dynamic Hand Gesture Recognition using Feature Concatenation via Multi-Input Hybrid Model,” Int. J. Electr. Comput. Eng. Syst., vol. 14, no. 5, pp. 535–546, Jun. 2023, doi: 10.32985/ijeces.14.5.5. DOI: https://doi.org/10.32985/ijeces.14.5.5

I.-J. Ding and N. Zheng, “RGB-D Depth-Sensor-Based Hand Gesture Recognition Using Deep Learning of Depth Images With Shadow Effect Removal for Smart Gesture Communication,” Sens. Mater., 2022, doi: 10.18494/sam3557. DOI: https://doi.org/10.18494/SAM3557

Y. Huo, J. Shen, X. Chen, and K. Yu, “A dynamic gesture recognition method based on R(2+1)D-transformer network,” in Third International Conference on Computer Graphics, Image, and Virtualization (ICCGIV 2023), Y. Wang and A. J. Moshayedi, Eds., Nanjing, China: SPIE, Nov. 2023, p. 57. doi: 10.1117/12.3008203. DOI: https://doi.org/10.1117/12.3008203

T. Bao, S. A. R. Zaidi, S. Xie, P. Yang, and Z.-Q. Zhang, “A CNN-LSTM Hybrid Model for Wrist Kinematics Estimation Using Surface Electromyography,” IEEE Trans. Instrum. Meas., vol. 70, pp. 1–9, 2021, doi: 10.1109/TIM.2020.3036654. DOI: https://doi.org/10.1109/TIM.2020.3036654

M. Jaén-Vargas et al., “A Deep Learning Approach to Recognize Human Activity Using Inertial Sensors and Motion Capture Systems,” in Frontiers in Artificial Intelligence and Applications, A. J. Tallón-Ballesteros, Ed., IOS Press, 2021. doi: 10.3233/FAIA210196. DOI: https://doi.org/10.3233/FAIA210196

J.-S. Kim, M.-G. Kim, and S.-B. Pan, “Two-Step Biometrics Using Electromyogram Signal Based on Convolutional Neural Network-Long Short-Term Memory Networks,” Appl. Sci., vol. 11, no. 15, p. 6824, Jul. 2021, doi: 10.3390/app11156824. DOI: https://doi.org/10.3390/app11156824

G. Palanisamy and S. S. Thangaswamy, “An Efficient Hand Gesture Recognition Based on Optimal Deep Embedded Hybrid Convolutional Neural Network?long Short Term Memory Network Model,” Concurr. Comput. Pract. Exp., 2022, doi: 10.1002/cpe.7109. DOI: https://doi.org/10.1002/cpe.7109

D. Copaci, J. Arias, M. Gómez-Tomé, L. Moreno, and D. Blanco, “sEMG-Based Gesture Classifier for a Rehabilitation Glove,” Front. Neurorobotics, vol. 16, May 2022, doi: 10.3389/fnbot.2022.750482. DOI: https://doi.org/10.3389/fnbot.2022.750482

C. Bagath Basha, “Enhancing Healthcare Data Security Using Quantum Cryptography for Efficient and Robust Encryption,” J. Electr. Syst., vol. 20, no. 5s, pp. 2070–2077, Apr. 2024, doi: 10.52783/jes.2544. DOI: https://doi.org/10.52783/jes.2544

A. Cisnal, J. Pérez-Turiel, J. Fraile, D. Sierra, and E. de la Fuente, “RobHand: A Hand Exoskeleton With Real-Time EMG-Driven Embedded Control. Quantifying Hand Gesture Recognition Delays for Bilateral Rehabilitation,” IEEE Access, vol. 9, pp. 137809–137823, 2021, doi: 10.1109/ACCESS.2021.3118281. DOI: https://doi.org/10.1109/ACCESS.2021.3118281

A. K. Panda, R. Chakravarty, and S. Moulik, “Hand Gesture Recognition using Flex Sensor and Machine Learning Algorithms,” 2020 IEEE-EMBS Conf. Biomed. Eng. Sci. IECBES, pp. 449–453, 2021, doi: 10.1109/IECBES48179.2021.9398789. DOI: https://doi.org/10.1109/IECBES48179.2021.9398789

S. Zhang et al., “Real-Time and Accurate Gesture Recognition With Commercial RFID Devices,” IEEE Trans. Mob. Comput., vol. 22, pp. 7327–7342, 2023, doi: 10.1109/TMC.2022.3211324. DOI: https://doi.org/10.1109/TMC.2022.3211324

K. Bell et al., “Verification of a Portable Motion Tracking System for Remote Management of Physical Rehabilitation of the Knee,” Sensors, vol. 19, no. 5, 2019, doi: 10.3390/s19051021. DOI: https://doi.org/10.3390/s19051021

P. Lin, R. Zhuo, S. Wang, Z. Wu, and J. Huangfu, “LED Screen-Based Intelligent Hand Gesture Recognition System,” IEEE Sens. J., vol. 22, pp. 24439–24448, 2022, doi: 10.1109/JSEN.2022.3219645. DOI: https://doi.org/10.1109/JSEN.2022.3219645