Effectiveness of the use of algorithms and methods of artificial technologies for sign language recognition for people with disabilities
DOI:
https://doi.org/10.15587/1729-4061.2022.262509Keywords:
neural network model, convolutional neural network, LSTM module, convolution, sign languageAbstract
According to WHO, the number of people with disabilities in the world has exceeded 1 billion. At the same time, 80 percent of all people with disabilities live in developing countries. In this regard, the demand for the use of applications for people with disabilities is growing every day. The paper deals with neural network methods like MediaPipe Holistic and the LSTM module for determining the sign language of people with disabilities. MediaPipe has demonstrated unprecedented low latency and high tracking accuracy in real-world scenarios thanks to built-in monitoring solutions. Therefore, MediaPipe Holistic was used in this work, which combines pose, hand, and face control with detailed levels.
The main purpose of this paper is to show the effectiveness of the HAR algorithm for recognizing human actions, based on the architecture of in-depth learning for classifying actions into seven different classes.
The main problem of this paper is the high level of recognition of the sign language of people with disabilities when implementing their work in cross-platform applications, web applications, social networks that facilitate the daily life of people with disabilities and interact with society. To solve this problem, an algorithm was used that combines the architecture of a convolutional neural network (CNN) and long short-term memory (LSTM) to study spatial and temporal capabilities from three-dimensional skeletal data taken only from a Microsoft Kinect camera. This combination allows you to take advantage of LSTM when modeling temporal data and CNN when modeling spatial data.
The results obtained based on calculations carried out by adding a new layer to the existing model showed higher accuracy than calculations carried out on the existing model.
References
- Rastgoo, R., Kiani, K., Escalera, S. (2021). Sign Language Recognition: A Deep Survey. Expert Systems with Applications, 164, 113794. doi: https://doi.org/10.1016/j.eswa.2020.113794
- Chuikov, A. V., Vulfin, A. M. (2017). Gesture recognition system. Vestnik UGATU, 21 (3 (77)), 113–122. Available at: https://cyberleninka.ru/article/n/sistema-raspoznavaniya-zhestov-na-osnove-neyrosetevyh-tehnologiy
- Wang, M., Lyu, X.-Q., Li, Y.-J., Zhang, F.-L. (2020). VR content creation and exploration with deep learning: A survey. Computational Visual Media, 6 (1), 3–28. doi: https://doi.org/10.1007/s41095-020-0162-z
- Murlin, A. G., Piotrovskiy, D. L., Rudenko, E. A., Yanaeva, M. V. (2014). Algorithms and methods for detection and recognition of hand gestures on video in real time. Politematicheskiy setevoy elektronniy nauchnyy zhurnal Kubanskogo gosudarstvennogo agrarnogo universiteta, 97, 626–635. Available at: https://www.elibrary.ru/item.asp?id=21527334
- Rautaray, S. S., Agrawal, A. (2012). Vision based hand gesture recognition for human computer interaction: a survey. Artificial Intelligence Review, 43 (1), 1–54. doi: https://doi.org/10.1007/s10462-012-9356-9
- Bae, S. H., Choi, I. K., Kim, N. S. (2016). Acoustic Scene Classification Using Parallel Combination of LSTM and CNN. Detection and Classification of Acoustic Scenes and Events. Available at: https://dcase.community/documents/workshop2016/proceedings/Bae-DCASE2016workshop.pdf
- Lee, D.-H., Hong, K.-S. (2010). A Hand gesture recognition system based on difference image entropy. In 2010 6th International Conference on Advanced Information Management and Service (IMS), 410–413.
- Chen, Y., Luo, B., Chen, Y.-L., Liang, G., Wu, X. (2015). A real-time dynamic hand gesture recognition system using kinect sensor. 2015 IEEE International Conference on Robotics and Biomimetics (ROBIO). doi: https://doi.org/10.1109/robio.2015.7419071
- Korchenko, A., Tereykovskiy, I., Karpinskiy, N., Tynymbaev, S. (2016). Neyrosetevye modeli, metody i sredstva otsenki parametrov bezopasnosti internet-orientirovannykh informatsionnykh sistem. Kyiv: TOV "Nash Format".
- Top 10 Deep Learning Algorithms You Should Know in 2022. Available at: https://www.simplilearn.com/tutorials/deep-learning-tutorial/deep-learning-algorithm
- Liu, N., Lovell, B. C. (2003). Gesture classification using hidden markov models and viterbi path counting. In VII-th Digital image computing: techniques and Applications. Available at: https://www.researchgate.net/publication/37616560_Gesture_Classification_Using_Hidden_Markov_Models_and_Viterbi_Path_Counting
- Phan, N. H., Bui, T. T. T., Spitsyn, V. G. (2013). Real-time hand gesture recognition base on Viola-Jones method, algorithm CAMShift, wavelet transform and principal component analysis. Upravlenie, vychislitel'naya tekhnika i informatika, 2 (23), 102–111. Available at: https://cyberleninka.ru/article/n/raspoznavanie-zhestov-na-videoposledovatelnosti-v-rezhime-realnogo-vremeni-na-osnove-primeneniya-metoda-violy-dzhonsa-algoritma
- Tkhang, N. T., Spitsyn, V. G. (2012). Algoritmicheskoe i programmnoe obespechenie dlya raspoznavaniya formy ruki v real'nom vremeni s ispol'zovaniem surfcdeskriptorov i neyronnoy seti. Izvestiya Tomskogo politekhnicheskogo universiteta. Inzhiniring georesursov, 320 (5), 48–54.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Aigulim Bayegizova, Gulden Murzabekova, Aisulu Ismailova, Ulzada Aitimova, Ayagoz Mukhanova, Zhanar Beldeubayeva, Aliya Ainagulova, Akgul Naizagarayeva
This work is licensed under a Creative Commons Attribution 4.0 International License.
The consolidation and conditions for the transfer of copyright (identification of authorship) is carried out in the License Agreement. In particular, the authors reserve the right to the authorship of their manuscript and transfer the first publication of this work to the journal under the terms of the Creative Commons CC BY license. At the same time, they have the right to conclude on their own additional agreements concerning the non-exclusive distribution of the work in the form in which it was published by this journal, but provided that the link to the first publication of the article in this journal is preserved.
A license agreement is a document in which the author warrants that he/she owns all copyright for the work (manuscript, article, etc.).
The authors, signing the License Agreement with TECHNOLOGY CENTER PC, have all rights to the further use of their work, provided that they link to our edition in which the work was published.
According to the terms of the License Agreement, the Publisher TECHNOLOGY CENTER PC does not take away your copyrights and receives permission from the authors to use and dissemination of the publication through the world's scientific resources (own electronic resources, scientometric databases, repositories, libraries, etc.).
In the absence of a signed License Agreement or in the absence of this agreement of identifiers allowing to identify the identity of the author, the editors have no right to work with the manuscript.
It is important to remember that there is another type of agreement between authors and publishers – when copyright is transferred from the authors to the publisher. In this case, the authors lose ownership of their work and may not use it in any way.
