Optimized adaptive machine learning for dynamic data streams

Abstract

The object of the study is the adaptive machine learning systems that are able to process large amounts of rapidly changing streaming data in real time. The problem of maintaining prediction accuracy and computational efficiency in the presence of concept drift is treated. Concept drift refers to the overweighting of static models when stationary models are tried, and the nature of the underlying distributions changes. The adaptive architecture includes revision divergence-oriented concept drift detection, incremental model updating via hyper-dimensional statistical clustering of segments. Results from experiments using simulated and real-world datasets demonstrate that the adaptive architecture maintains predictive accuracy above 0.83 across abrupt, gradual, recurrent, and continuous drift scenarios. Compared with non-adaptive models, adaptation latency is reduced by approximately 2.6×, while unnecessary retraining operations are decreased by up to 40%. These results are possible due to the fact that proposed framework is able to retrain solutions if, and only if, distributional changes are determined to be statistically significant and meaningful to the model. This leads to the avoidance of processors being given redundant computations and providing a steady-state model during non-drift conditions. A principal contribution is that feature engineering is accomplished in a drift-aware manner, thresholding is made adaptive to the distributions indicated, and update mechanisms are employed which efficiently utilize resources in a unified high-performance streaming pipeline. The architecture performs well under abrupt, gradual, recurrent, and continuous drift and effective for real-time applications which include smart-city analytics, cyber security monitoring, roadways system works, and IoT for industrial systems