Development and verification of an orchestration architecture of AI agents for automated API testing with a unified representation of requirements

Authors

DOI:

https://doi.org/10.15587/2706-5448.2026.360928

Keywords:

API, testing, orchestration, API requirements, LLM, DevOps, CI/CD, architecture, automation, determinism

Abstract

The object of this research is the process of automated testing of application programming interfaces (API) in systems developed following Agile and DevOps approaches, where requirements are heterogeneous, frequently changing, and represented in various formats: from formal OpenAPI specifications to textual documentation (Confluence). The problem addressed is the absence of an architectural mechanism for systematic integration of heterogeneous requirements sources and for decoupling requirements interpretation from test generation. Specification-oriented approaches fail to incorporate business rules from textual sources, covering only 70–80% of specification content. LLM-based approaches are unstable: repeated runs with identical prompts yield test sets differing by 20–40%, with coverage standard deviation reaching ±12%.

AI-driven orchestration architecture is proposed, comprising a coordination layer O, requirements-processing agents A, and a protocol-independent unified requirements representation R. The test generation process is formalized as T = G(O(A(S))), where S denotes the set of requirements sources and G the deterministic test generation algorithm. The key property of the architecture is isolation of the stochastic LLM component at the agent level, guaranteeing reproducibility of the test set T for any fixed R.

Verification was conducted through a comparative experiment on a REST API service with 5 endpoints and 12 business rules. API coverage: 88–92% vs. 72–78% (specification-based) and 55–82% (LLM-based). Standard deviation: ±2% vs. ±3% and ±12%. Reproducibility: 0.97 vs. 0.95 and 0.62. Maintenance: 15–20% modified tests vs. 60–70% and 40–55%.

The proposed architecture targets software development teams practising API-First Development and Documentation-Driven Development. Results are applicable to Agile/DevOps environments with frequently changing requirements.

Author Biography

Maksym Moskalenko, Educational and Scientific Institute “Ukrainian State University of Chemical Technology”

PhD Student

Department of Information Systems

References

  1. Harman, M., Mansouri, S. A., Zhang, Y. (2012). Search-based software engineering. ACM Computing Surveys, 45 (1), 1–61. https://doi.org/10.1145/2379776.2379787
  2. McIntosh, S., Kamei, Y., Adams, B., Hassan, A. E. (2015). An empirical study of the impact of modern code review practices on software quality. Empirical Software Engineering, 21 (5), 2146–2189. https://doi.org/10.1007/s10664-015-9381-9
  3. Ammann, P., Offutt, J. (2016). Introduction to Software Testing. Cambridge University Press, 768. Available at: https://www.cambridge.org/highereducation/books/introduction-to-software-testing/95E57CCADEA697EC8594F03729F47311
  4. Amalfitano, D., Faralli, S., Hauck, J. C. R., Matalonga, S., Distante, D. (2023). Artificial Intelligence Applied to Software Testing: A Tertiary Study. ACM Computing Surveys, 56 (3), 1–38. https://doi.org/10.1145/3616372
  5. Richardson, L., Amundsen, M. (2013). RESTful Web APIs. O’Reilly Media, 406. Available at: https://www.oreilly.com/library/view/restful-web-apis/9781449359713/
  6. Cerny, T., Donahoo, M. J., Trnka, M. (2018). Contextual understanding of microservice architecture. ACM SIGAPP Applied Computing Review, 17 (4), 29–45. https://doi.org/10.1145/3183628.3183631
  7. ISTQB Certified Tester Foundation Level Syllabus. Version 4.0 (2023). International Software Testing Qualifications Board (ISTQB). Available at: https://istqb.org/istqb-releases-certified-tester-foundation-level-v4-0-ctfl/
  8. Garousi, V., Felderer, M., Hacaloğlu, T. (2017). Software test maturity assessment and test process improvement: A multivocal literature review. Information and Software Technology, 85, 16–42. https://doi.org/10.1016/j.infsof.2017.01.001
  9. Kim, M., Xin, Q., Sinha, S., Orso, A. (2022). Automated test generation for REST APIs: no time to rest yet. Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, 289–301. https://doi.org/10.1145/3533767.3534401
  10. Corradini, D., Zampieri, A., Pasqua, M., Viglianisi, E., Dallago, M., Ceccato, M. (2022). Automated black‐box testing of nominal and error scenarios in RESTful APIs. Software Testing, Verification and Reliability, 32 (5). https://doi.org/10.1002/stvr.1808
  11. Utting, M., Pretschner, A., Legeard, B. (2011). A taxonomy of model‐based testing approaches. Software Testing, Verification and Reliability, 22 (5), 297–312. https://doi.org/10.1002/stvr.456
  12. Utting, M., Legeard, B. (2010). Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann, 424. Available at: https://www.sciencedirect.com/book/9780123725011/practical-model-based-testing
  13. Arcuri, A., Briand, L. (2012). Formal Analysis of the Probability of Interaction Fault Detection Using Random Testing. IEEE Transactions on Software Engineering, 38 (5), 1088–1099. https://doi.org/10.1109/tse.2011.85
  14. Arcuri, A., Briand, L. (2011). Adaptive random testing. Proceedings of the 2011 International Symposium on Software Testing and Analysis, 265–275. https://doi.org/10.1145/2001420.2001452
  15. Golmohammadi, A., Zhang, M., Arcuri, A. (2023). Testing RESTful APIs: A Survey. ACM Transactions on Software Engineering and Methodology, 33 (1), 1–41. https://doi.org/10.1145/3617175
  16. Chen, Y., Hu, Z., Zhi, C., Han, J., Deng, S., Yin, J. (2024). ChatUniTest: A Framework for LLM-Based Test Generation. Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 572–576. https://doi.org/10.1145/3663529.3663801
  17. Tang, Y., Liu, Z., Zhou, Z., Luo, X. (2024). ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generation. IEEE Transactions on Software Engineering, 50 (6), 1340–1359. https://doi.org/10.1109/tse.2024.3382365
  18. Kim, M., Corradini, D., Sinha, S., Orso, A., Pasqua, M., Tzoref-Brill, R., Ceccato, M. (2023). Enhancing REST API Testing with NLP Techniques. Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, 1232–1243. https://doi.org/10.1145/3597926.3598131
  19. Atlidakis, V., Godefroid, P., Polishchuk, M. (2019). RESTler: Stateful REST API Fuzzing. 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), 748–758. https://doi.org/10.1109/icse.2019.00083
  20. Arcuri, A. (2019). RESTful API Automated Test Case Generation with EvoMaster. ACM Transactions on Software Engineering and Methodology, 28 (1), 1–37. https://doi.org/10.1145/3293455
Development and verification of an orchestration architecture of AI agents for automated API testing with a unified representation of requirements

Downloads

Published

2026-05-29

How to Cite

Moskalenko, M. (2026). Development and verification of an orchestration architecture of AI agents for automated API testing with a unified representation of requirements. Technology Audit and Production Reserves, 3(2(89), 31–40. https://doi.org/10.15587/2706-5448.2026.360928

Issue

Vol. 3 No. 2(89) (2026): Information and control systems

Section

Information Technologies

License

Copyright (c) 2026 Maksym Moskalenko

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

The consolidation and conditions for the transfer of copyright (identification of authorship) is carried out in the License Agreement. In particular, the authors reserve the right to the authorship of their manuscript and transfer the first publication of this work to the journal under the terms of the Creative Commons CC BY license. At the same time, they have the right to conclude on their own additional agreements concerning the non-exclusive distribution of the work in the form in which it was published by this journal, but provided that the link to the first publication of the article in this journal is preserved.