Design of hybrid neural networks of the ensemble structure

Authors

DOI:

https://doi.org/10.15587/1729-4061.2021.225301

Keywords:

hybrid convolutional neural network, genetic algorithm, ensemble, structural-parametric synthesis

Abstract

This paper considers the structural-parametric synthesis (SPS) of neural networks (NNs) of deep learning, in particular convolutional neural networks (CNNs), which are used in image processing. It has been shown that modern neural networks may possess a variety of topologies. That is ensured by using unique blocks that determine their essential features, namely, the compression and excitation unit, the attention module convolution unit, the channel attention module, the spatial attention module, the residual unit, the ResNeXt block. This, first of all, is due to the need to increase their efficiency in the processing of images. Due to the large architectural space of parameters, including the type of unique block, the location in the structure of the convolutional neural network, its connections with other blocks, layers, computing costs grow nonlinearly. To minimize computational costs while maintaining the specified accuracy this work set tasks of both the generation of possible topology and structural-parametric synthesis of convolutional neural networks. To resolve them, the use of a genetic algorithm (GA) has been proposed. Parameter configuration was implemented using a genetic algorithm and modern gradient methods (GM). For example, stochastic gradient descent with momentum, accelerated Nesterov gradient, adaptive gradient algorithm, distribution of the root of the mean square of the gradient, assessment of adaptive momentum, adaptive Nesterov momentum. It is assumed to use such networks in the intelligent medical diagnostic system (IMDS), for determining the activity of tuberculosis. To improve the accuracy of solving the classification problem in the processing of images, the ensemble structure of hybrid convolutional neural networks (HCNNs) has been proposed in the current work. The parallel structure of the ensemble with the merged layer was used. Algorithms of optimal choice and integration of features in the construction of the ensemble have been developed

Author Biographies

Victor Sineglazov , National Aviation University

Doctor of Technical Sciences, Professor, Head of Department

Department of Aviation Computer-Integrated Complexes

Anatoly Kot, National Technical University of Ukraine "Igor Sikorsky Kyiv Polytechnic Institute"

Postgraduate Student

Department of Computer-Aided Management and Data Processing Systems

References

  1. Al-Marzouqi, H. (2009). Data Clustering Using a Modified Kuwahara Filter. 2009 International Joint Conference on Neural Networks. doi: https://doi.org/10.1109/ijcnn.2009.5178658
  2. Chumachenko, O. I. (2017). Deep Learning Classifier Based on NEFCLASS and NEFPROX Neural Networks. Materialy mizhnarodnoi naukovo-praktychnoi konferentsiyi: Informatsiyni tekhnolohiyi ta kompiuterne modeliuvannia. Ivano-Frankivsk. Івано-Франківськ, 278–281. Available at: http://itcm.comp-sc.if.ua/2017/Chumachenko.pdf
  3. Zgurovsky, M., Sineglazov, V., Chumachenko, E. (2021). Artificial Intelligence Systems Based on Hybrid Neural Networks. Studies in Computational Intelligence. doi: https://doi.org/10.1007/978-3-030-48453-8
  4. Xie, L., Yuille, A. (2017). Genetic CNN. arxiv.org. Available at: https://arxiv.org/pdf/1703.01513.pdf
  5. Real, E., Aggarwal, A., Huang, Y., Le, Q. V. (2019). Regularized Evolution for Image Classifier Architecture Search. Proceedings of the AAAI Conference on Artificial Intelligence, 33, 4780–4789. doi: https://doi.org/10.1609/aaai.v33i01.33014780
  6. Saxena, S., Verbeek, J. (2016). Convolutional Neural Fabrics. Advances in Neural Information Processing Systems (NIPS). Barcelona.
  7. Baker, B., Gupta, O., Naik, N., Raskar, R. (2016). Designing Neural Network Architectures using Reinforcement Learning. arXiv.org. Available at: https://arxiv.org/pdf/1611.02167.pdf
  8. Brock, A., Lim, T., Ritchie, J. M., Weston, N. J. (2018). SmaSH: One-shot model architecture search through hypernetworks. Paper presented at 6th International Conference on Learning Representations 2018. Vancouver. Available at: https://arxiv.org/pdf/1708.05344.pdf
  9. Zoph, B., Vasudevan, V., Shlens, J., Le, Q. V. (2018). Learning Transferable Architectures for Scalable Image Recognition. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. doi: https://doi.org/10.1109/cvpr.2018.00907
  10. Liu, H., Simonyan, K., Vinyals, O., Fernando, C., Kavukcuoglu, K. (2018). Hierarchical representations for efficient architecture search. arXiv.org. Available at: https://arxiv.org/pdf/1711.00436.pdf
  11. Zhang, W., Li, R., Deng, H., Wang, L., Lin, W., Ji, S., Shen, D. (2015). Deep convolutional neural networks for multi-modality isointense infant brain image segmentation. NeuroImage, 108, 214–224. doi: https://doi.org/10.1016/j.neuroimage.2014.12.061
  12. Yap, M. H., Pons, G., Marti, J., Ganau, S., Sentis, M., Zwiggelaar, R. et. al. (2018). Automated Breast Ultrasound Lesions Detection Using Convolutional Neural Networks. IEEE Journal of Biomedical and Health Informatics, 22 (4), 1218–1226. doi: https://doi.org/10.1109/jbhi.2017.2731873
  13. Ronneberger, O., Fischer, P., Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, 234–241. doi: https://doi.org/10.1007/978-3-319-24574-4_28
  14. Yu, L., Chen, H., Dou, Q., Qin, J., Heng, P.-A. (2017). Automated Melanoma Recognition in Dermoscopy Images via Very Deep Residual Networks. IEEE Transactions on Medical Imaging, 36 (4), 994–1004. doi: https://doi.org/10.1109/tmi.2016.2642839
  15. Bi, L., Kim, J., Ahn, E., Kumar, A., Fulham, M., Feng, D. (2017). Dermoscopic Image Segmentation via Multistage Fully Convolutional Networks. IEEE Transactions on Biomedical Engineering, 64 (9), 2065–2074. doi: https://doi.org/10.1109/tbme.2017.2712771
  16. Yuan, Y., Chao, M., Lo, Y.-C. (2017). Automatic Skin Lesion Segmentation Using Deep Fully Convolutional Networks With Jaccard Distance. IEEE Transactions on Medical Imaging, 36 (9), 1876–1886. doi: https://doi.org/10.1109/tmi.2017.2695227
  17. Goyal, M., Yap, M., Hassanpour, S. (2020). Multi-class Semantic Segmentation of Skin Lesions via Fully Convolutional Networks. Proceedings of the 13th International Joint Conference on Biomedical Engineering Systems and Technologies. doi: https://doi.org/10.5220/0009380302900295
  18. Vesal, S., Malakarjun Patil, S., Ravikumar, N., Maier, A. K. (2018). A Multi-task Framework for Skin Lesion Detection and Segmentation. OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis, 285–293. doi: https://doi.org/10.1007/978-3-030-01201-4_31
  19. Yap, M. H., Goyal, M., Ng, J., Oakley, A. (2019). Skin lesion boundary segmentation with fully automated deep extreme cut methods. Medical Imaging 2019: Biomedical Applications in Molecular, Structural, and Functional Imaging. doi: https://doi.org/10.1117/12.2513015
  20. Soudani, A., Barhoumi, W. (2019). An image-based segmentation recommender using crowdsourcing and transfer learning for skin lesion extraction. Expert Systems with Applications, 118, 400–410. doi: https://doi.org/10.1016/j.eswa.2018.10.029
  21. De Brebisson, A., Montana, G. (2015). Deep neural networks for anatomical brain segmentation. 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). doi: https://doi.org/10.1109/cvprw.2015.7301312
  22. LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., Jackel, L. D. (1989). Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation, 1 (4), 541–551. doi: https://doi.org/10.1162/neco.1989.1.4.541
  23. Lee, H., Grosse, R., Ranganath, R., Ng, A. Y. (2009). Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Proceedings of the 26th Annual International Conference on Machine Learning - ICML ’09. doi: https://doi.org/10.1145/1553374.1553453
  24. Khan, A., Sohail, A., Zahoora, U., Qureshi, A. S. (2020). A survey of the recent architectures of deep convolutional neural networks. Artificial Intelligence Review, 53 (8), 5455–5516. doi: https://doi.org/10.1007/s10462-020-09825-6
  25. Chumachenko, H. I., Levitsky, O. Y. (2011). Development of image processing algorithm for diagnostics. Electronics and Control Systems, 1 (27), 57–65. doi: https://doi.org/10.18372/1990-5548.27.258
  26. Chollet, F. (2017). Xception: Deep Learning with Depthwise Separable Convolutions. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). doi: https://doi.org/10.1109/cvpr.2017.195
  27. Xie, S., Girshick, R., Dollar, P., Tu, Z., He, K. (2017). Aggregated Residual Transformations for Deep Neural Networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). doi: https://doi.org/10.1109/cvpr.2017.634
  28. Zhang, X., Li, Z., Loy, C. C., Lin, D. (2017). PolyNet: A Pursuit of Structural Diversity in Very Deep Networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). doi: https://doi.org/10.1109/cvpr.2017.415
  29. Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H. et. al. (2017). Residual Attention Network for Image Classification. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). doi: https://doi.org/10.1109/cvpr.2017.683
  30. Woo, S., Park, J., Lee, J.-Y., Kweon, I. S. (2018). CBAM: Convolutional Block Attention Module. Lecture Notes in Computer Science, 3–19. doi: https://doi.org/10.1007/978-3-030-01234-2_1
  31. Hu, J., Shen, L., Sun, G. (2018). Squeeze-and-Excitation Networks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. doi: https://doi.org/10.1109/cvpr.2018.00745
  32. Ioffe, S., Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. ICML'15: Proceedings of the 32nd International Conference on International Conference on Machine Learning, 37, 448–456.
  33. He, K., Zhang, X., Ren, S., Sun, J. (2016). Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). doi: https://doi.org/10.1109/cvpr.2016.90
  34. Simonyan, K., Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition. Published as a conference paper at ICLR 2015. Available at: https://arxiv.org/pdf/1409.1556.pdf
  35. Bodyanskiy, E. V., Rudenko, O. G. (2004). Iskusstvennye neyronnye seti: arhitektury, obuchenie, primeneniya. Kharkiv: Teletekh, 369.
  36. Golovko, V. A. (2001). Neyronnye seti. Obuchenie, organizatsiya i primenenie. Kniga 4. Moscow: IPRZHR, 256.
  37. Kohonen Self-Organizing Maps. Available at: http://mnemstudio.org/neural-networks-kohonen-self-organizing-maps.htm
  38. Martinez-Munoz, G., Hernandez-Lobato, D., Suarez, A. (2009). An Analysis of Ensemble Pruning Techniques Based on Ordered Aggregation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31 (2), 245–259. doi: https://doi.org/10.1109/tpami.2008.78
  39. Goyal, M., Oakley, A., Bansal, P., Dancey, D., Yap, M. H. (2020). Skin Lesion Segmentation in Dermoscopic Images With Ensemble Deep Learning Methods. IEEE Access, 8, 4171–4181. doi: https://doi.org/10.1109/access.2019.2960504
  40. Powers, D. M. W. (2011). Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation. International Journal of Machine Learning Technology, 2 (1), 37–63.

Downloads

Published

2021-02-26

How to Cite

Sineglazov , V., & Kot, A. (2021). Design of hybrid neural networks of the ensemble structure. Eastern-European Journal of Enterprise Technologies, 1(4 (109), 31–45. https://doi.org/10.15587/1729-4061.2021.225301

Issue

Vol. 1 No. 4 (109) (2021): Mathematics and Cybernetics - applied aspects

Section

Mathematics and Cybernetics - applied aspects

License

Copyright (c) 2021 Виктор Михайлович Синеглазов, Анатолий Тарасович Кот

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

The consolidation and conditions for the transfer of copyright (identification of authorship) is carried out in the License Agreement. In particular, the authors reserve the right to the authorship of their manuscript and transfer the first publication of this work to the journal under the terms of the Creative Commons CC BY license. At the same time, they have the right to conclude on their own additional agreements concerning the non-exclusive distribution of the work in the form in which it was published by this journal, but provided that the link to the first publication of the article in this journal is preserved.

A license agreement is a document in which the author warrants that he/she owns all copyright for the work (manuscript, article, etc.).
The authors, signing the License Agreement with TECHNOLOGY CENTER PC, have all rights to the further use of their work, provided that they link to our edition in which the work was published.
According to the terms of the License Agreement, the Publisher TECHNOLOGY CENTER PC does not take away your copyrights and receives permission from the authors to use and dissemination of the publication through the world's scientific resources (own electronic resources, scientometric databases, repositories, libraries, etc.).
In the absence of a signed License Agreement or in the absence of this agreement of identifiers allowing to identify the identity of the author, the editors have no right to work with the manuscript.
It is important to remember that there is another type of agreement between authors and publishers – when copyright is transferred from the authors to the publisher. In this case, the authors lose ownership of their work and may not use it in any way.