Development of deep learning framework for complex pattern recognition in big data

Abstract

The object of this study is the process and analysis of intelligent recognition and classification of spatiotemporal patterns in large arrays of streaming data. The problem to be solved is the absence of a deep learning framework that can guarantee adaptability to rapidly changing concepts, efficient computation for continuous data streams, and the transparency of the prediction process when working with heterogeneous and dynamically changing sources of big data used to support decision making.

The developed programing framework applies convolutional neural network- long short – term memory networks with an attention-gating mechanism that enables detection of spatiotemporal dependencies and exhibits model interpretation of decisions. Extensive evaluation of the implemented system using multivariate flow-based data demonstrated the performance capabilities of the system with a classification accuracy of 0.98, F1 score of 0.97, area under the receiver operating characteristic curve of 0.99 and Harmonic Score of 0.90. The interpretation of the results is summarized by the interaction of multilevel feature extraction followed by an optimization process through Kullback-Leibler divergence that ensures reliable online drift detection and automatic models re-training. Additional contributions of the systematic use of the framework included interpretable decisions using Shapley Additive explanations and gradient-weighted class activation mapping visualizations. It has a strong evidence of sustained reliable model performance in non-stationarity data conditions and streaming data. The outcome of this study embodies significant practical implications toward the creation of real-time decision-support systems in domains. Finally, the structured framework can also be utilized for future investigations into the development of highly scalable, explainable, and trust-worthy artificial intelligence architectures.