THE MECHANISM FOR CREATION OF EVENT-DRIVEN APPLICATIONS BASED ON WOLFRAM MATHEMATICA AND APACHE KAFKA
DOI:
https://doi.org/10.30837/ITSSI.2021.15.053Keywords:
event-driven applications, queue manager, mathematical processor, saving resources and funds, Kafka, MathematicaAbstract
The article is devoted to the research and development of the mechanism of interaction between Wolfram Mathematica programs and Apache Kafka queue to provide the ability to build event-driven applications based on it. The subject of the research is the practical principles of building a mechanism for interaction between Wolfram Mathematica and Apache Kafka. The purpose of the article is to develop and substantiate practical recommendations regarding the formation of a mechanism for publishing messages to the Apache Kafka queue and reading messages from it for programs of the mathematical processor Wolfram Mathematica, which will make it possible to build event-driven applications. Tasks: to determine the mechanism of such interaction, prove the choice of tools for its implementation, create and test the obtained results. The research used the following tools: Apache Kafka, Kafkacat, the method of developing the Wolfram Mathematica package. The results of the research: the mechanism of interaction between Wolfram Mathematica and Apache Kafka was determined and the corresponding toolkit was created on its basis in the form of two Mathematica packages, which are built on using Apache Kafka as a queue client and third-party Kafkacat software, respectively. It is shown that the first option is less reliable and consumes much more computer resources during operation. It has been demonstrated that the Mathematica processor is currently not suitable in its pure form for real-time data analysis. Recommendations are given regarding the use of built-in compilation functions to increase the speed of such processing. Conclusions. Practical recommendations have been developed and substantiated regarding the formation of the mechanism of interaction between the Wolfram Mathematica mathematical processor and the Apache Kafka queue manager for the possibility of working in two directions with the queue: publishing messages and reading them. A toolkit for such interaction in the form of Mathematica packages has been created, their capabilities have been demonstrated, as well as comparison with each other. The economic benefit of using the described tools is shown. Future ways of its improvement are given.
References
Ed-daoudy, A., Maalmi, K. (2019), "A new Internet of Things architecture for real-time prediction of various diseases using machine learning on big data environment", Journal of Big Data, Vol. 6, No. 104. DOI: https://doi.org/10.1186/s40537-019-0271-7
Mahapatra, T. (2020), "Composing high-level stream processing pipelines", Journal of Big Data, Vol. 7, No. 81. DOI: https://doi.org/10.1186/s40537-020-00353-2
Nasiri, H., Nasehi, S., Goudarzi, M. (2019), "Evaluation of distributed stream processing frameworks for IoT applications in Smart Cities", Journal of Big Data, Vol. 6, No. 52. DOI: https://doi.org/10.1186/s40537-019-0215-2
Jung, S., Kim, Y., Hwang, E. (2018), "Real-time car tracking system based on surveillance videos", EURASIP Journal on Image and Video Processing, Vol. 2018, No. 133. DOI: https://doi.org/10.1186/s13640-018-0374-7
Ismail, A., Truong, H. L., Kastner, W. (2019), "Manufacturing process data analysis pipelines: a requirements analysis and survey", Journal of Big Data, Vol. 6, No. 1. DOI: https://doi.org/10.1186/s40537-018-0162-3
Kim, Y. K., Kim, Y., Jeong, C. S. (2018), "RIDE: real-time massive image processing platform on distributed environment", EURASIP Journal on Image and Video Processing, Vol. 2018, No. 39. DOI: https://doi.org/10.1186/s13640-018-0279-5
Kolajo, T., Daramola, O., Adebiyi, A. (2019), "Big data stream analysis: a systematic literature review", Journal of Big Data, Vol. 6, No. 47. DOI: https://doi.org/10.1186/s40537-019-0210-7
GitHub (2020), "Mathworks-ref-arch/matlab-apache-kafka: MATLAB Interface for Apache Kafka", available at : https://github.com/mathworks-ref-arch/matlab-apache-kafka (last accessed 10 December 2020).
Rehman, S., Idrees, M., Shah, R. A. et al. (2019), "Suction/injection effects on an unsteady MHD Casson thin film flow with slip and uniform thickness over a stretching sheet along variable flow properties", Boundary Value Problems, Vol. 2019, No. 26. DOI: https://doi.org/10.1186/s13661-019-1133-0
Ghorbani, M. A., Singh, V. P., Sivakumar, B. et al. (2017), "Probability distribution functions for unit hydrographs with optimization using genetic algorithm", Applied Water Science, Vol. 7, P. 663–676. DOI: https://doi.org/10.1007/s13201-015-0278-y
DeCanio, S. J. (2020), "Can an AI learn political theory?", AI Perspectives, Vol. 2, Article 3. DOI: https://doi.org/10.1186/s42467-020-00007-2
You, X., Chen, D. R. (2018), "A new sequence convergent to Euler–Mascheroni constant", Journal of Inequalities and Applications, Vol. 2018, Article 75. DOI: https://doi.org/10.1186/s13660-018-1670-6
Mathematica Stack Exchange (2020), "Networking - Connect Mathematica to message broker - Kafka, NATS or mosquitto", available at: https://mathematica.stackexchange.com/questions/199848/connect-mathematica-to-message-broker-kafka-nats-or-mosquitto (last accessed 10 December 2020).
Mathematica Stack Exchange (2020), "Networking - MQTTLink TopicSubscribe[] cannot receive messages", available at: https://mathematica.stackexchange.com/questions/211940/mqttlink-topicsubscribe-cannot-receive-messages (last accessed 10 December 2020).
Apache Kafka (2020), "Apache Kafka", available at: https://kafka.apache.org/documentation/#os (last accessed 10 December 2020).
GitHub (2020), "Edenhill/kafkacat: Generic command line non-JVM Apache Kafka producer and consumer", available at: https://github.com/edenhill/kafkacat (last accessed 10 December 2020).
Microsoft Docs (2020), "An overview on the Windows Subsystem for Linux", available at: https://docs.microsoft.com/en-us/windows/wsl/ (last accessed 10 December 2020).
Stack Overflow (2020), "Kafka bootstrap-servers vs zookeeper in kafka-console-consumer", available at: https://stackoverflow.com/questions/41774446/kafka-bootstrap-servers-vs-zookeeper-in-kafka-console-consumer (last accessed 10 December 2020).
Stack Overflow (2020), "Apache kafka - bootstrap-server vs zookeeper params in consumer console", available at: https://stackoverflow.com/questions/53954877/bootstrap-server-vs-zookeeper-params-in-consumer-console (last accessed 10 December 2020).
Zolotariov, D. (2020), "The distributed system of automated computing based on cloud infrastructure", Innovative Technologies and Scientific Solutions for Industries, No. 4 (14), P. 47–55. DOI: https://doi.org/10.30837/ITSSI.2020.14.047
Zolotariov, D. A. (2020), "Automation and optimization of scientific and engineering calculations in Wolfram Mathematica", Kharkiv : FOP Panov A. M. ISBN: 978-617-7859-36-8 [In Ukrainian].
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Our journal abides by the Creative Commons copyright rights and permissions for open access journals.
Authors who publish with this journal agree to the following terms:
Authors hold the copyright without restrictions and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0) that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
Authors are able to enter into separate, additional contractual arrangements for the non-commercial and non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
Authors are permitted and encouraged to post their published work online (e.g., in institutional repositories or on their website) as it can lead to productive exchanges, as well as earlier and greater citation of published work.
