Automatic sign language translation system using neural network technologies and 3D animation




automation of sign speech, animated character, body position tracking, people with hearing impairments, sign language, neural networks, gesture recognition, ukrainian sign language, sign language translation, reduce inequality


Implementation of automatic sign language translation software in the process of social inclusion of people with hearing impairment is an important task. Social inclusion for people with hearing disabilities is an acute problem that must be solved in the context of the development of IT technologies and legislative initiatives that ensure the rights of people with disabilities and their equal opportunities. This substantiates the relevance of the research of assistive technologies, in the context of software tools, such as the process of social inclusion of people with severe hearing impairment in society. The subject of research is methods of automated sign language translation using intelligent technologies. The purpose of the work is the development and research of sign language automation methods to improve the quality of life of people with hearing impairments in accordance with the "Goals of Sustainable Development of Ukraine" (in the "Reduction of Inequality" part). The main tasks of the research are the development and testing of methods of converting sign language into text, converting text into sign language, as well as automating translation from one sign language to another sign language using modern intelligent technologies. Neural network modeling and 3D animation methods were used to solve these problems. The following results were obtained in the work: the main problems and tasks of social inclusion for people with hearing impairments were identified; a comparative analysis of modern methods and software platforms of automatic sign language translation was carried out; a system combining the SL-to-Text method is proposed and investigated; the Text-to-SL method using 3D animation to generate sign language concepts; the method of generating a 3D-animated gesture from video recordings; method of implementing the Sign Language1 to Sign Language2 technology. For gesture recognition, a convolutional neural network model is used, which is trained using imported and system-generated datasets of video gestures. The trained model has a high recognition accuracy (98.52%). The creation of a 3D model for displaying the gesture on the screen and its processing took place in the Unity 3D environment. The structure of the project, executive and auxiliary files used to build 3D animation for the generation of sign language concepts includes: event handler files; display results according to which they carry information about the position of the tracked points of the body; files that store the characteristics of materials that have been added to certain body mapping points. Conclusions: the proposed methods of automated translation have practical significance, which is confirmed by the demo versions of the software applications "Sign Language to Text" and "Text to Sign Language". A promising direction for continuing research on the topic of the work is the improvement of SL1-to-SL2 methods, the creation of open datasets of video gestures, the joining of scientists and developers to fill dictionaries with concepts of various sign languages.

Author Biographies

Yevhenii Shovkovyi, Kharkiv National University of Radio Electronics

student of Artificial Intelligence Department

Olena Grinyova, Kharkiv National University of Radio Electronics

Lecturer of Artificial Intelligence Department

Serhii Udovenko, Simon Kuznets Kharkiv National University of Economics

doctor of technical sciences, professor, Head of Department of Informatics and Computer Engineering

Larysa Chala, Kharkiv National University of Radio Electronics

candidat of technical sciences, docent, Associate Professor of Artificial Intelligence Department



Chumachenko, I.V. "Methods of human resources management in the formation World Health Organization. Deafness and hearing loss". 2023. URL: (date of application: 10.04.2023).

"Web Accessibility Initiative". URL: (date of application: 10.04.2023).

Decree of the President of Ukraine №722/2019 "On the Sustainable Development Goals of Ukraine for the period up to 2030" dated September 19, 2019. URL: (date of application: 10.04.2023).

Kruglyk, O.P., Horlachev, O.S. (2023), "The importance of auditory perception in the process of sign language translation for people with hearing impairment" [Znachennya slukhovoho spryymannya v protsesi zdiysnennya surdoperekladu dlya osib z porushennyam slukhu], Scientific journal. Series 19 - Correctional pedagogy and special psychology, No. 43, P. 39–48. DOI

Gobhinath, S., Vignesh, T., Pavankumar, R., et al. (2020), "A Study of Hand Gesture Segmentation Techniques for Sign Languages", Journal of Computational and Theoretical Nanoscience, Vol. 17, No. 4, P. 1764-1769. DOI:

Ghosh, P., Dutta, A., Topno, S. (2022), "Sign Language Hand Glove", American Journal of Electronics & Communication, Vol. 3, No. 1, P. 14-16. DOI:

Chakraborty, S., Prayosi, P., Sarkar, S., Chakraborty A. (2023), "Sign Language Recognition Using Landmark Detection, GRU and LSTM", American Journal of Electronics & Communication, Vol. 3, No. 3, P. 20-26. DOI:

Mahesh, R., Kumar, T., Kavin, R, Karthikeyan S. (2020), "Manipulation of Web Using Gestures", Journal of Computational and Theoretical Nanoscience, Vol. 17, No.8, P. 3782-3785. DOI:

Khetavath, S., Sendhilkumar, N., Mukunthan P., et al. (2023), "An Intelligent Heuristic Manta-Ray Foraging Optimization and Adaptive Extreme Learning Machine for Hand Gesture Image Recognition", Big Data Mining and Analytics, No. 6(3), P. 321-335. DOI:

Zhou, H., Wang, W., Liu G., et al. (2022), "PointGAT: Graph attention networks for 3D object detection", Intelligent and Converged Networks, No. 3(2), P. 204-216. DOI:

Wuyan, L., Xiaolong, X., Fu, X. (2022), "Human gesture recognition of dynamic skeleton using graph convolutional networks", Journal of Electronic Imaging, Vol. 32, Issue 2, Р.1-21. DOI:

John, J., Deshpande, S. (2023),"Hand Gesture Identification Using Deep Learning and Artificial Neural Networks: A Review", Computational Intelligence for Engineering and Management Applications. Lecture Notes in Electrical Engineering, Vol 984, Springer, Singapore. Р. 389-403. DOI:

Tan, Y., Lim, K., Lee, C. (2021), "Hand gesture recognition via enhanced densely connected convolutional neural network", Expert Syst. Appl., Vol. 175, No. 114797. Р. 28569–28587 DOI: 10.1016/j.eswa.2021.114797

Gadekallu, T., Alazab, M., Kaluri, R., et al. (2021), "Hand gesture classification using a novel CNN-crow search algorithm", Complex & Intell. Syst., Vol. 7, P. 1855–1868. DOI:

Zhang, Y., Huang, Y., Sun, X., et al. (2020), "Static and dynamic human arm/hand gesture capturing and recognition via multiinformation fusion of _exible strain sensors", IEEE Sensors Journal, Vol. 20, No. 12. P. 6450-6459. DOI: 10.1109/JSEN.2020.2965580.

Rahim, M., Miah, A., Sayeed A. and Shin J. (2020), "Hand Gesture Recognition Based on Optimal Segmentation in Human-Computer Interaction", Proceedings of the 3rd IEEE International Conference on Knowledge Innovation and Invention (ICKII), Kaohsiung, Taiwan, P. 163-166. DOI: 10.1109/ICKII50300.2020.9318870

Bhavana, D., Kumar, K., Bipin Chandra, M., et al. (2021), "Hand Sign Recognition using CNN", International Journal of Performance Analysis in Sport, Vol. 17(3), P. 314–321. DOI: 10.23940/ijpe.21.03.p7.314321




How to Cite

Shovkovyi, Y., Grinyova, O., Udovenko, S., & Chala, L. (2023). Automatic sign language translation system using neural network technologies and 3D animation. INNOVATIVE TECHNOLOGIES AND SCIENTIFIC SOLUTIONS FOR INDUSTRIES, (4(26), 108–121.