LUNA: Bridging communication with sign language translation
Keywords:
Sign language, smart glove, IoT, gesture recognition, inclusive technologyAbstract
This paper presents Learn, Unite, Network and Assist (LUNA) App, an assistive Internet of Things (IoT) system designed to facilitate two-way communication between individuals with hearing and speech impairments and the general public. The LUNA system integrates a smart glove embedded with flex sensors and an MPU6050 accelerometer to recognize hand gestures and translate them into real-time text displayed via a mobile application developed using Flutter. Beyond gesture recognition, LUNA includes a sign language learning module, GPS-based emergency location tracking, and video call functionality via WebRTC, enabling enhanced communication and user safety. Developed using the Scrum methodology, the system combines ESP32 microcontroller-based hardware with Firebase for real-time data handling. Experimental results demonstrate that LUNA achieves high accuracy and responsiveness, with gesture recognition response times ranging from 850 to 1200 milliseconds. This project represents a step toward inclusive communication by promoting mutual understanding and interaction between the disabled and non-disabled communities.
References
R. Ambar, S. Salim, M. H. A. Wahab, M. M. A. Jamil, and T. C. Phing, "Development of a wearable sensor glove for real-time sign language translation," Annals of Emerging Technologies in Computing (AETiC), vol. 7, no. 5, pp. 25–38, Oct. 2023. [Online]. Available: http://aetic.theiaer.org/archive/v7/v7n5/p3.html
M. E. E. Elshareif, N. A. A. Alias, N. Jomhari, and H. Sofian, "Smart glove with mobile application to detect static Arabic Hijaiyah hand code for Quran recitation," in Proc. IEEE, 2024. [Online]. Available: https://doi.org/10.1109/LT60077.2024.10469054
A. J. Abougarair and W. A. Arebi, "Smart glove for sign language translation," Int. Robot. Autom. J., vol. 8, no. 3, pp. 109–117, Dec. 2022.
M. S. Amin, M. T. Amin, M. Y. Latif, A. A. Jathol, N. Ahmed, and M. I. N. Tarar, "Alphabetical gesture recognition of American sign language using E-Voice smart glove," in Proc. IEEE INMIC, 2021. [Online]. Available:
https://doi.org/10.1109/INMIC50486.2020.9318185
N. S. Mohamad and J. Lias, "Smart glove Malaysian Sign Language translator," Emerging Electrical and Electronic Engineering, vol. 2, no. 2, Oct. 2021. [Online]. Available:
https://doi.org/10.30880/eeee.2021.02.02.007
Oxford University Press, "Methodology," Oxford Learner’s Dictionaries. [Online]. Available:
https://www.oxfordlearnersdictionaries.com/definition/english/methodology. [Accessed: Dec. 13, 2024].
Global Union, "SDG 10: Reduce inequality within and among countries." [Online]. Available:
https://www.globalgoals.org/goals/10-reduced-inequalities/
A. M. M. Ibrahim and H. H. Kamel, "Social welfare services as a mechanism to reduce the social exclusion of the deaf and mute," Int. J. Soc. Sci., vol. 14, no. 2, Jun. 2022.
M. Abbas, F. Hayat, A. U. Rehman, K. S. Arif, and K. Wahab, "The influence of Agile development (Scrum) on software project management," in Proc. IEEE, 2019. [Online]. Available: https://ieeexplore.ieee.org
H. K. Kondaveeti, N. K. Kumaravelu, S. D. Vanambathina, S. E. Mathe, and S. Vappangi, "A systematic literature review on prototyping with Arduino: Applications, challenges, advantages and limitations," Comput. Sci. Rev., vol. 40, May 2021. [Online]. Available: https://www.sciencedirect.com/science/article/abs/pii/S1574013721000046
D. Hercog, T. Lerher, M. Truntic, and O. Tezak, "Design and implementation of ESP32-based IoT devices," Sensors, vol. 23, no. 15, Art. no. 6739, Jul. 2023. [Online]. Available: https://www.mdpi.com/1424-8220/23/15/6739
