Determining the psycho-emotional state of the observed based on the analysis of video observations

Authors

DOI:

https://doi.org/10.15587/1729-4061.2024.296500

Keywords:

computer vision, physical aggression, emotional reaction, bullying, model training, neural network

Abstract

This paper develops a system for determining the psycho-emotional state of the observed people based on the analysis of video surveillance with the application of artificial intelligence technology using hardware and software tools such as PoseNet, PyTorch, SQLite, FastAPI and Flask. In many areas of human endeavor, there is an urgent need for a surveillance system that can reliably function and detect suspicious activities. To solve this problem, this paper proposes a novel framework for a real-time surveillance system that automatically detects abnormal human activities.

The system has been tested and validated in real environments. The results of testing artificial intelligence program models showed the best results (f1 score with values of 0.98–0.99). The weighted average value of the f1-score metric was 0.96, which is quite a high value. The use of PoseNet implemented with PyTorch allowed to accurately determine the pose of the person in the video and extract information about the position of different body parts. The peculiarity of this work lies in the development of artificial intelligence models for automatic detection of possible physical aggression in videos, in the methods of forming an optimal set of features for the development of AI models that identify the aggressor and the victim of bullying.

The developed system has the potential to be a useful tool in various fields such as psychology, medicine, security and others where it is important to analyze the emotional state of people based on their physical manifestations. The obtained applied results can be used in educational institutions and in spheres where video analysis is necessary

Author Biographies

Yedilkhan Amirgaliyev, Institute of Information and Computational Technologies

Doctor of Technical Sciences, Professor, Chief Researcher, Head of Laboratory

Laboratory of Artificial Intelligence and Robotics

Iurii Krak, Taras Shevchenko National University of Kyiv

Doctor of Physical and Mathematical Sciences, Professor, Head of Department

Department of Theoretical Cybernetics

Indira Bukenova, Almaty Technological University

Master of Technical Sciences, Lecturer

Department of Information Systems

Bayan Kazangapova, Almaty Technological University

Associate Professor

Department of Technology

Gani Bukenov, Almaty Technological University

Master of Mathematics, Lecturer

Department of Information Systems

References

  1. Bauman, S. (2016). Do We Need More Measures of Bullying? Journal of Adolescent Health, 59 (5), 487–488. https://doi.org/10.1016/j.jadohealth.2016.08.021
  2. Al-Nawashi, M., Al-Hazaimeh, O. M., Saraee, M. (2016). A novel framework for intelligent surveillance system based on abnormal human activity detection in academic environments. Neural Computing and Applications, 28 (S1), 565–572. https://doi.org/10.1007/s00521-016-2363-z
  3. Seldin, M., Yanez, C. (2019). Student Reports of Bullying: Results from the 2017 School Crime Supplement to the National Crime Victimization Survey. Web Tables. NCES 2019–054. National Center for Education Statistics. Available at: https://nces.ed.gov/pubs2019/2019054.pdf
  4. McCarthy, R. J., Elson, M. (2018). A Conceptual Review of Lab-Based Aggression Paradigms. Collabra: Psychology, 4 (1). https://doi.org/10.1525/collabra.104
  5. Parrott, D. J., Zeichner, A. (2002). Effects of alcohol and trait anger on physical aggression in men. Journal of Studies on Alcohol, 63 (2), 196–204. https://doi.org/10.15288/jsa.2002.63.196
  6. Allen, J. J., Anderson, C. A. (2017). Aggression and Violence: Definitions and Distinctions. The Wiley Handbook of Violence and Aggression, 1–14. https://doi.org/10.1002/9781119057574.whbva001
  7. Zhou, P., Ding, Q., Luo, H., Hou, X. (2018). Violence detection in surveillance video using low-level features. PLOS ONE, 13 (10), e0203668. https://doi.org/10.1371/journal.pone.0203668
  8. Hassner, T., Itcher, Y., Kliper-Gross, O. (2012). Violent flows: Real-time detection of violent crowd behavior. 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. https://doi.org/10.1109/cvprw.2012.6239348
  9. Ullah, F. U. M., Ullah, A., Muhammad, K., Haq, I. U., Baik, S. W. (2019). Violence Detection Using Spatiotemporal Features with 3D Convolutional Neural Network. Sensors, 19 (11), 2472. https://doi.org/10.3390/s19112472
  10. Amirgaliyev, Y. N., Bukenova, I. N., Bukenov, G. S., Kenshimov, C. A. (2023). Software solutions for the recognition violent movements by video. Bulletin of East Kazakhstan Technical University, 2, 31–42.
  11. Peixoto, B. M., Avila, S., Dias, Z., Rocha, A. (2018). Breaking down violence. Proceedings of the 13th International Conference on Availability, Reliability and Security. https://doi.org/10.1145/3230833.3232809
  12. Song, D., Kim, C., Park, S.-K. (2018). A multi-temporal framework for high-level activity analysis: Violent event detection in visual surveillance. Information Sciences, 447, 83–103. https://doi.org/10.1016/j.ins.2018.02.065
  13. Carneiro, S. A., da Silva, G. P., Guimaraes, S. J. F., Pedrini, H. (2019). Fight Detection in Video Sequences Based on Multi-Stream Convolutional Neural Networks. 2019 32nd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). https://doi.org/10.1109/sibgrapi.2019.00010
  14. Febin, I. P., Jayasree, K., Joy, P. T. (2019). Violence detection in videos for an intelligent surveillance system using MoBSIFT and movement filtering algorithm. Pattern Analysis and Applications, 23 (2), 611–623. https://doi.org/10.1007/s10044-019-00821-3
  15. Marinoiu, E., Zanfir, M., Olaru, V., Sminchisescu, C. (2018). 3D Human Sensing, Action and Emotion Recognition in Robot Assisted Therapy of Children with Autism. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/cvpr.2018.00230
  16. Koppula, H. S., Gupta, R., Saxena, A. (2013). Learning human activities and object affordances from RGB-D videos. The International Journal of Robotics Research, 32 (8), 951–970. https://doi.org/10.1177/0278364913478446
  17. Bermejo Nievas, E., Deniz Suarez, O., Bueno García, G., Sukthankar, R. (2011). Violence Detection in Video Using Computer Vision Techniques. Lecture Notes in Computer Science, 332–339. https://doi.org/10.1007/978-3-642-23678-5_39
  18. Zhou, P., Ding, Q., Luo, H., Hou, X. (2017). Violent Interaction Detection in Video Based on Deep Learning. Journal of Physics: Conference Series, 844, 012044. https://doi.org/10.1088/1742-6596/844/1/012044
  19. Pawar, K., Attar, V. (2018). Deep learning approaches for video-based anomalous activity detection. World Wide Web, 22 (2), 571–601. https://doi.org/10.1007/s11280-018-0582-1
  20. Zhao, H., Torralba, A., Torresani, L., Yan, Z. (2019). HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization. 2019 IEEE/CVF International Conference on Computer Vision (ICCV). https://doi.org/10.1109/iccv.2019.00876
  21. Olweus, D. (1978). Aggression in the schools: Bullies and whipping boys. Hemisphere.
  22. Solberg, M. E., Olweus, D. (2003). Prevalence estimation of school bullying with the Olweus Bully/Victim Questionnaire. Aggressive Behavior, 29 (3), 239–268. https://doi.org/10.1002/ab.10047
  23. Shetgiri, R. (2013). Bullying and Victimization Among Children. Advances in Pediatrics, 60 (1), 33–51. https://doi.org/10.1016/j.yapd.2013.04.004
  24. Fung, A. L. C. (2019). Adolescent Reactive and Proactive Aggression, and Bullying in Hong Kong: Prevalence, Psychosocial Correlates, and Prevention. Journal of Adolescent Health, 64 (6), S65–S72. https://doi.org/10.1016/j.jadohealth.2018.09.018
  25. Lereya, S. T., Copeland, W. E., Costello, E. J., Wolke, D. (2015). Adult mental health consequences of peer bullying and maltreatment in childhood: two cohorts in two countries. The Lancet Psychiatry, 2 (6), 524–531. https://doi.org/10.1016/s2215-0366(15)00165-0
  26. Buch-Frohlich, A., Paradis, A., Hébert, M., Cyr, M., Frappier, J.-Y. (2019). Bullying and sexual harassment as predictors of suicidality in sexually abused adolescent girls. International Journal of Victimology, 35, 63–73.
  27. Yang, T., Guo, L., Hong, F., Wang, Z., Yu, Y., Lu, C. (2020). Association Between Bullying and Suicidal Behavior Among Chinese Adolescents: An Analysis of Gender Differences. Psychology Research and Behavior Management, Volume 13, 89–96. https://doi.org/10.2147/prbm.s228007
  28. Lloyd, K., Rosin, P. L., Marshall, D., Moore, S. C. (2017). Detecting violent and abnormal crowd activity using temporal analysis of grey level co-occurrence matrix (GLCM)-based texture measures. Machine Vision and Applications, 28 (3-4), 361–371. https://doi.org/10.1007/s00138-017-0830-x
  29. Bilinski, P., Bremond, F. (2016). Human violence recognition and detection in surveillance videos. 2016 13th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). https://doi.org/10.1109/avss.2016.7738019
Determining the psycho-emotional state of the observed based on the analysis of video observations

Downloads

Published

2024-02-28

How to Cite

Amirgaliyev, Y., Krak, I., Bukenova, I., Kazangapova, B., & Bukenov, G. (2024). Determining the psycho-emotional state of the observed based on the analysis of video observations. Eastern-European Journal of Enterprise Technologies, 1(2 (127), 45–53. https://doi.org/10.15587/1729-4061.2024.296500

Issue

Vol. 1 No. 2 (127) (2024): Information technology. Industry control systems

Section

Information technology

License

Copyright (c) 2024 Yedilkhan Amirgaliyev, Iurii Krak, Indira Bukenova, Bayan Kazangapova, Gani Bukenov

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

The consolidation and conditions for the transfer of copyright (identification of authorship) is carried out in the License Agreement. In particular, the authors reserve the right to the authorship of their manuscript and transfer the first publication of this work to the journal under the terms of the Creative Commons CC BY license. At the same time, they have the right to conclude on their own additional agreements concerning the non-exclusive distribution of the work in the form in which it was published by this journal, but provided that the link to the first publication of the article in this journal is preserved.

A license agreement is a document in which the author warrants that he/she owns all copyright for the work (manuscript, article, etc.).
The authors, signing the License Agreement with TECHNOLOGY CENTER PC, have all rights to the further use of their work, provided that they link to our edition in which the work was published.
According to the terms of the License Agreement, the Publisher TECHNOLOGY CENTER PC does not take away your copyrights and receives permission from the authors to use and dissemination of the publication through the world's scientific resources (own electronic resources, scientometric databases, repositories, libraries, etc.).
In the absence of a signed License Agreement or in the absence of this agreement of identifiers allowing to identify the identity of the author, the editors have no right to work with the manuscript.
It is important to remember that there is another type of agreement between authors and publishers – when copyright is transferred from the authors to the publisher. In this case, the authors lose ownership of their work and may not use it in any way.