Methods for detecting anomalies in microservices using statistical analysis
DOI:
https://doi.org/10.30837/2522-9818.2024.2.121Keywords:
anomaly detection; microservices; statistical analysis; regression analysis; clustering.Abstract
The subject of the study is methods of detecting anomalies in microservices using statistical analysis. Microservices is a popular software development architecture that allows for flexible and scalable systems. However, due to their complexity, such systems can be vulnerable to various types of anomalies that can affect their performance and reliability. The goal of the work is an analytical review of existing methods of detecting anomalies in microservice systems using statistical analysis methods. Detection of anomalies is critical to ensure stable system operation and quick response to possible problems. To achieve the purpose, the following tasks are defined: review of methods for detecting anomalies in microservices; description of the principles of regression analysis, cluster analysis and the method of principal components; comparison of methods according to the criteria of efficiency, computational complexity, resistance to noise and adaptability; recommendations for choosing a method and the possibility of combining them; summary of results and identification of directions for future research. A method for detecting anomalies in microservices is considered, which includes regression analysis, cluster analysis, and the method of principal components (PCA). The results of the study confirmed that each method has its advantages and limitations. Regression analysis is effective in systems with clear trends, but less effective in complex and dynamic systems. Cluster analysis has proven to be robust to noise and capable of detecting both individual anomalies and groups of anomalous events but requires significant computational resources. The method of principal components (PCA) is a powerful tool for the analysis of high-dimensional data, but it has limitations in the high complexity of calculations and interpretation of results. Each of the considered methods has its pros and cons, so the study proposed a new method that would consist in combining them. The conclusions emphasize the importance of statistical analysis for monitoring microservice systems. Well-chosen data analysis techniques facilitate the detection of anomalies in complex environments such as microservices. The use of regression analysis, cluster analysis and the method of principal components allows you to get a deep insight into the operation of the system. However, for best results, it is recommended to combine different methods and analyze their results in the context of a specific system. This approach provides greater resistance to anomalies and faster response to them in microservice architectures.
References
Список літератури
Ghani A., Ahmad S., Khan M. A., Khalid S., Zohaib S. Огляд технік виявлення відхилень для мікросервісів. IEEE Access. №9. 2021. Р. 122766–122805. DOI: https://ieeexplore.ieee.org/document/10479425
Andrzejak M., Moniruzzaman M., Schumann D., Winkler S., Biffl C. Виявлення аномалій у даних веб-трафіку за допомогою статистичного навчання. Міжнародна конференція IEEE з видобутку даних. IEEE, 2009. Р. 423–430. DOI: https://ieeexplore.ieee.org/document/6211951
Singh R., Paul A. Виявлення аномалій у мікросервісах за допомогою статистичного навчання. Журнал хмарних обчислень, 2021. 9(1), 21 р.
Carvalho F. D., Gama J., Rocha R. Моніторинг потоків мережевого трафіку для виявлення аномалій за допомогою однокласових опорних векторних машин. Міжнародна конференція IEEE з видобутку даних. IEEE, 2010. Р. 77–84. DOI: https://ieeexplore.ieee.org/document/9751518
Alarcar J., Nguyen H. Q., Nguyen S. V., Gaber M. Виявлення аномалій в архітектурах хмарних мікросервісів. ACM Computing Surveys. №52. 2019. Р. 1–38. DOI: https://www.sciencedirect.com/science/article/pii/S0164121223003126
Jiang X., Chen C., Zhou Y., Zhang J., Liu X. Виявлення аномалій для мікросервісів на основі глибокого навчання. Міжнародна конференція IEEE з веб-сервісів (ICWS). IEEE, 2021. Р. 574–581. DOI: https://ieeexplore.ieee.org/document/9461271
Li Z., Zhang W., Li Z., Zhao M. Виявлення аномалій для мікросервісів. ACM Computing Surveys. №53. 2020. Р. 1–42. DOI: https://dl.acm.org/doi/10.1145/3639478.3643535
He S., Jin H., Dai W., Hu X. Виявлення аномалій в системах на основі мікросервісів. IEEE Access. №6. 2018. Р. 8459–8469. DOI: https://doi.org/10.1109/ACCESS.2018.2805848
Liu F. T., Ting K. M., Zhou Z. H. Outlier Detection and Classification: A Review. In: Aggarwal, C. C., Zhai, Y. (Eds.), Outlier Analysis. Springer, Berlin, Heidelberg, 2012. Р. 1–21. DOI: https://www.sciencedirect.com/science/article/pii/S2772662223000048
Luo J., Wu J., Zhang Y., Sun L., Liu Y. Виявлення аномалій для мікросервісів на основі федеративного навчання. Міжнародна конференція IEEE з веб-сервісів (ICWS). IEEE, 2022. Р. 582–589. DOI: https://ieeexplore.ieee.org/document/9461272
He P., Ranjan R., Nogueira J., Veiga L. M., Zhao W. Огляд виявлення аномалій для мікросервісів в хмарному обчисленні. Журнал системного та програмного забезпечення. № 182. 2021. 111318 р. DOI: https://journalofcloudcomputing.springeropen.com/articles/10.1186/s13677-022-00296-4
Pimentel A., Clifton L., Clifton L., Lee Y., Kang A. (2014), Огляд технік виявлення аномалій у даних часових рядів. Сигнальна обробка. №92. 2014. Р. 67–81. DOI: https://link.springer.com/chapter/10.1007/978-3-030-73100-7_60
Jiang Y., Tan K. M. C., Lam S. W., Chen P. Виявлення аномалій за допомогою однокласових опорних векторних машин у високорозмірних просторах. Міжнародна конференція IEEE з видобутку даних. 2017. Р. 622–631. DOI: https://doi.org/10.110
Xu W., Huang C., Li W., Dong Z., Xu D. Виявлення аномалій в системах хмарного обчислення: огляд. Міжнародна конференція IEEE з хмарного обчислення. IEEE, 2014. Р. 986–993. DOI: https://ieeexplore.ieee.org/document/5331755
Hawkins D. M. Викиди: виявлення аномалій у даних. Чам: Springer International Publishing. 2016. DOI: https://link.springer.com/10.1007/978-1-4899-7502-7_912-1
Patcha A., Park J.-M. Огляд технік виявлення аномалій: існуючі рішення та останні технологічні тенденції. Комп'ютерні мережі. №51. 2007. Р. 3448–3470. DOI: https://doi.org/10.1016/j.comnet.2007.02.001
Thompson R., Williams L. Прикладні моделі лінійної регресії. Sage Publications. 2019. DOI: https://doi.org/10.4135/9781412993882
Kim Y., Lee H. Аналіз кластерів: концепції та практика. Springer. 2017.
Stewart D., DeCoster J. Аналіз головних компонент і пов'язані техніки. Routledge. 2020.
Yu S., Zhao L., Zhang Y. Anomaly Detection in Microservices Using Principal Component Analysis (PCA). Visual Studio Magazine. 2019. DOI: http://dx.doi.org/10.1109/ACCESS.2020.3044610
References
Ghani, A., Ahmad, S., Khan, M. A., Khalid, S., Zohaib, S. (2021), "A Survey on Anomaly Detection Techniques for Microservices". IEEE Access. №9. Р. 122766–122805. DOI: https://ieeexplore.ieee.org/document/10479425
Andrzejak, M., Moniruzzaman, M., Schumann, D., Winkler, S., Biffl, C. (2009), "Detecting Anomalies in Web Traffic Data Using Statistical Learning". International Conference on Data Mining. IEEE, Р. 423–430. DOI: https://ieeexplore.ieee.org/document/6211951
Singh, R., Paul, A. (2020), "Anomaly Detection in Microservices Using Statistical Learning". Journal of Cloud Computing, 9(1), 21.
Carvalho, F. D., Gama, J., Rocha, R. (2010), "Monitoring Streams of Network Traffic for Anomaly Detection Using One-Class Support Vector Machines". In: 2010 IEEE International Conference on Data Mining. IEEE, Р. 77–84. DOI: https://ieeexplore.ieee.org/document/9751518
Alarcar, J., Nguyen, H. Q., Nguyen, S. V., Gaber, M. (2019), "Anomaly Detection in Cloud-Based Microservice Architectures: A Survey". ACM Computing Surveys. №52. Р. 1–38. DOI: https://www.sciencedirect.com/science/article/pii/S0164121223003126
Jiang, X., Chen, C., Zhou, Y., Zhang, J., Liu, X. (2021), "Anomaly Detection for Microservices Based on Deep Learning". In: 2021 IEEE International Conference on Web Services (ICWS). IEEE, Р. 574–581. DOI: https://ieeexplore.ieee.org/document/9461271
Li, Z., Zhang, W., Li, Z., Zhao, M. (2020), "Anomaly Detection for Microservices: A Survey". ACM Computing Surveys. №53. Р. 1–42. DOI: https://dl.acm.org/doi/10.1145/3639478.3643535
He, S., Jin, H., Dai, W., Hu, X. (2018), "Anomaly Detection in Microservices-Based Systems: A Survey". IEEE Access. №6. Р. 8459–8469. DOI: https://doi.org/10.1109/ACCESS.2018.2805848
Liu, F. T., Ting, K. M., Zhou, Z. H. (2012), "Outlier Detection and Classification: A Review". In: Aggarwal, C. C., Zhai, Y. (Eds.), Outlier Analysis. Springer, Berlin, Heidelberg, P. 1–21. DOI: https://www.sciencedirect.com/science/article/pii/S2772662223000048
Luo, J., Wu, J., Zhang, Y., Sun, L., Liu, Y. (2022), "Anomaly Detection for Microservices Based on Federated Learning". In: 2022 IEEE International Conference on Web Services (ICWS). IEEE, Р. 582–589. DOI: https://ieeexplore.ieee.org/document/9461272
He, P., Ranjan, R., Nogueira, J., Veiga, L. M., Zhao, W. (2021), "A Survey on Anomaly Detection for Microservices in Cloud Computing". Journal of Systems and Software. №182. 111318 р. DOI: https://journalofcloudcomputing.springeropen.com/articles/10.1186/s13677-022-00296-4
Pimentel, A., Clifton, L., Clifton, L., Lee, Y., Kang, A. (2014), "A Review of Techniques for Detecting Anomalies in Time Series Data". Signal Processing. №92. Р. 67–81. DOI: https://link.springer.com/chapter/10.1007/978-3-030-73100-7_60
Jiang, Y., Tan, K. M. C., Lam, S. W., Chen, P. (2017), "Anomaly Detection Using One-Class SVM in High-Dimensional Spaces". IEEE International Conference on Data Mining. Р. 622–631. DOI: https://doi.org/10.1109/ICDM.2017.103
Xu, W., Huang, C., Li, W., Dong, Z., Xu, D. (2014), "Anomaly Detection in Cloud Computing Systems: A Survey". In: 2014 IEEE International Conference on Cloud Computing. IEEE, Р. 986–993. DOI: https://ieeexplore.ieee.org/document/5331755
Hawkins, D. M. (2016), "The Outliers: Detecting Anomalies in Data". Cham: Springer International Publishing. DOI: https://link.springer.com/10.1007/978-1-4899-7502-7_912-1
Patcha, A., Park, J.-M. (2007), "An Overview of Anomaly Detection Techniques: Existing Solutions and Latest Technological Trends". Computer Networks. № 51. Р. 3448–3470. DOI: https://doi.org/10.1016/j.comnet.2007.02.001
Thompson, R., Williams, L. (2019), "Applied Linear Regression Models". Sage Publications. DOI: https://doi.org/10.4135/9781412993882
Kim, Y., Lee, H. (2017), "Cluster Analysis: Concepts and Practice". Springer.
Stewart, D., DeCoster, J. (2020), "Principal Components Analysis and Related Techniques". Routledge.
Yu, S., Zhao, L., Zhang, Y. (2019), "Anomaly Detection in Microservices Using Principal Component Analysis (PCA)". Visual Studio Magazine. DOI: http://dx.doi.org/10.1109/ACCESS.2020.3044610
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Our journal abides by the Creative Commons copyright rights and permissions for open access journals.
Authors who publish with this journal agree to the following terms:
Authors hold the copyright without restrictions and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0) that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
Authors are able to enter into separate, additional contractual arrangements for the non-commercial and non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
Authors are permitted and encouraged to post their published work online (e.g., in institutional repositories or on their website) as it can lead to productive exchanges, as well as earlier and greater citation of published work.
