Development of adaptive reconfiguration method for stream data processing systems using system metrics

Abstract

The object of research is the process of adaptive configuration changes for stream processing applications which is focused on improving specific performance properties. The absence of the generalized automated approach for dynamic reconfiguration of state-store in limited hardware environment is the research problem addressed in this paper. The proposed solution helps to avoid a need for manual application reconfiguration from engineers. The implementation is based on Kafka Streams but designed to be portable across other frameworks that use RocksDB as a state store. Static configuration of modern stream processing systems limits efficiency under variable workloads. In this study, an adaptive module is proposed that monitors system metrics in real-time and automatically updates state-store configurations. The module performs deterministic check to derive new configuration based on predefined thresholds or utilizes a fine-tuned Large Language Model (LLM) to select new configuration values when decisions are vague. The method dynamically applies updates to the affected instance. High-load experimental results reveal the fact that adaptive executions eliminated write stalls, increased memtable hit ratio from 2% to 40% and block-cache hit ratio from 15% to 80%, reduced disk I/O by approximately 50%, and improved throughput by around 5%, at the cost of higher memory consumption. To avoid redundant adaptive updates and outlier-based bias a 10-minute observation frequency was selected. The approach is suitable for systems with fixed resources, state-intensive workloads with high key cardinality. Additionally, if covers the need for safe configuration change under operational constraints. The architecture is framework agnostic for the RocksDB-based based stream processing with state stores.