Review of statistical analysis methods of high-dimensional data

Authors

  • Makrufa Sharif Hajirahimova Institut Information Technology of ANAS 9 B. Vahabzade str., Baku, Republic of Azerbaijan, AZ1141, Azerbaijan
  • Aybeniz Salman Aliyeva Institut Information Technology of ANAS 9 B. Vahabzade str., Baku, Republic of Azerbaijan, AZ1141, Azerbaijan

DOI:

https://doi.org/10.15587/1729-4061.2015.51603

Keywords:

big data, statistical analysis, statistical analysis methods, bootstrap, resampling

Abstract

We live in the era of "big data". Big data opens up new opportunities for modern society, has become the "raw material" for production, a new source for immense economic and social value. At the same time, big data has set new computational and statistical tasks before the researcher. In order to study the status of these tasks, the paper describes the main applications of big data, investigates the statistical computing problems, associated with a large volume, diversity and high speed, affecting a paradigm shift of statistical and computational methods. A review of existing statistical methods, algorithms and research in recent years is presented. The research results show that several factors require the development of new, more effective statistical methods and algorithms: firstly, traditional statistical methods are not justified in terms of statistical significance with respect to big data; secondly, in terms of computational efficiency; the third factor is relevant to the specific features inherent in big data: heterogeneity, the accumulation of noise, spurious correlations, etc. It appears that this area will continue to be the subject of research.

Author Biographies

Makrufa Sharif Hajirahimova, Institut Information Technology of ANAS 9 B. Vahabzade str., Baku, Republic of Azerbaijan, AZ1141

PhD

Aybeniz Salman Aliyeva, Institut Information Technology of ANAS 9 B. Vahabzade str., Baku, Republic of Azerbaijan, AZ1141

Researcher

References

  1. Alguliyev, R. M., Hajirahimova, M. S. (2014). "Big Data" phenomenon: Challenges and Opportunities. Problems of Information Technology, 2, 3–16.
  2. Philip Chen, C. L., Zhang, C.-Y. (2014). Data-intensive applications, challenges, techniques and technologies: A survey on Big Data. Information Sciences, 275, 314–347. doi: 10.1016/j.ins.2014.01.015
  3. Soares, S. (2012). Big Data Governance, An Emerging Imperative. MC Press Online, LLC. 1st edition, 368.
  4. Kambatla, K., Kollias, G., Kumar, V., Grama, A. (2014). Trends in big data analytics. Journal of Parallel and Distributed Computing, 74 (7), 2561–2573. doi: 10.1016/j.jpdc.2014.01.003
  5. Fan, J., Han, F., Liu, H. (2014). Challenges of Big Data analysis. National Science Review, 1 (2), 293–314. doi: 10.1093/nsr/nwt032
  6. Jordan J. M., Lin D. J. (2014). Statistics for big data: Are statisticians ready for big data? International Chinese Statistical Association Bulletin, 26, 59–66.
  7. Yu, B. (2014). Let us own data science. IMS Bulletin Online, 43 (7).
  8. Chun,W. C., Chen, M. H., Schifano, E., Wu, J., Yan, J. (2015). A Survey of Statistical Methods and Computing for Big Data. Available at: http://de.arxiv.org/abs/1502.07989v1
  9. Lin, N., Xi, R. (2011). Aggregated estimating equation estimation. Statistics and Its Interface, 4 (1), 73–83. doi: 10.4310/sii.2011.v4.n1.a8
  10. Chen, M. H., Craiu, R., Liang, F., Liu, C. Statistical and Computational Theory and Methodology for Big Data Analysis. Available at: https://www.birs.ca/workshops/2014/14w5086
  11. Liang, F., Cheng, Y., Song, Q., Park, J., Yang, P. (2013). A Resampling-Based Stochastic Approximation Method for Analysis of Large Geostatistical Data. Journal of the American Statistical Association, 108 (501), 325–339. doi: 10.1080/01621459.2012.746061
  12. Liang, F., Kim, J. (2013). A bootstrap Metropolis–Hastings algorithm for Bayesian analysis of big data. Tech. rep., Department of Statistics, Texas A & M University.
  13. Liang, F., Liu, C., Carroll, R. J. (2010). Advanced Markov chain Monte Carlo methods: learning from past samples. Wiley, New York, 378.
  14. Hastie, T., Tibshirani, R., Friedman, J. (2009). The Elements of Statistical, Learning: Data Mining Inference and Prediction. Second edition. Springer. doi: 10.1007/978-0-387-84858-7
  15. Bennett, J., Grout, R., Pebay, P., Roe, D., Thompson, D. (2009). Numerically stable, single-pass, parallel statistics algorithms. Proceedings of the IEEE International Conference on Cluster Computing and Workshops, 1–8. doi: 10.1109/clustr.2009.5289161
  16. Sysoev, O., Burdakov, O., Grimvall, A. (2011). A segmentation-based algorithm for large-scale partially ordered monotonic regression. Computational Statistics & Data Analysis, 55 (8), 2463–2476. doi: 10.1016/j.csda.2011.03.001
  17. Pébay, P., Thompson, D., Bennett, J., Mascarenhas, A. (2011). Design and performance of a scalable, parallel statistics toolkit /Proceedings of the International Symposium on Parallel and Distributed Processing Workshops and Phd Forum, 1475–1484. doi: 10.1109/ipdps.2011.293
  18. Lizhe, W., Hui, Z., Ranjan, R., Zomaya, A., Peng Liu. (2014). Estimating the Statistical Characteristics of Remote Sensing Big Data in the Wavelet Transform Domain. IEEE Transactions on Emerging Topics in Computing, 2 (3), 324–337. doi: 10.1109/tetc.2014.2356499
  19. Jin, X., Wah, B. W., Cheng, X., Wang, Y. (2015). Significance and Challenges of Big Data Research. Big Data Research, 2 (2), 59–64. doi: 10.1016/j.bdr.2015.01.006
  20. ITU-T Technology Watch. Big data: Big today, normal tomorrow, 2013. Available at: http://unstats.un.org/unsd/trade/events/2014/beijing/documents/other/ITU.pdf
  21. Big Data, Big Impact: New Possibilities for International Development. Available at: http://www.weforum.org/reports/big-data-big-impact-new-possibilities-international-development
  22. 8 top challenges big data brings to statisticians. Available at: http://www.fiercebigdata.com/story/8-top-challenges-big-data-brings-statisticians/2014-07-14
  23. 2013 International Year of Statistics. Available at: http://www.isi-web.org/recent-pages/490-2013-international-year-of-statistics
  24. Kleiner, A., Talwalkar, A., Sarkar, P., Jordan, M. I. (2014). A scalable bootstrap for massive data. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 76 (4), 795–816. doi: 10.1111/rssb.12050
  25. Basiri, S., Ollila, E. (2015). Robust, scalable and fast bootstrap method for analyzing large scale data. Available at: http://arxiv.org/pdf/1504.02382‎
  26. Ma, P., Sun, X. (2014). Leveraging for big data regression. WIREs Comput Stat, 7 (1), 70–76. doi: 10.1002/wics.1324
  27. Schifano, E. D., Wu, J., Wang, C., Yan, J., Chen, M.-H. (2014). Online updating of statistical inference in the big data setting. Tech. Rep. University of Connecticut, Storrs, Connecticut.
  28. Di Ciaccio, A., Coli, M., Angulo Ibanez, J. M. (Eds.) (2012). Advanced Statistical Methods for the Analysis of Large Data-Sets. Springer. doi: 10.1007/978-3-642-21037-2
  29. Computational statistics. Available at: https://en.wikipedia.org/wiki/Computational_statistics
  30. Wilkinson, L. (2008). The Future of Statistical Computing. Technometrics, 50 (4), 418–435. doi: 10.1198/004017008000000460
  31. James, G., Witten, D., Hastie, T., Tibshirani, R. An Introduction to Statistical Learning with Applications in R. Available at: http://www-bcf.usc.edu/
  32. Schbidberger, M., Morgan, M., Eddelbuettel, D. et. al. (2009). State of the art in parallel computing with R. Journal of Statistical Software, 31 (1), 1–27.

Downloads

Published

2015-10-20

How to Cite

Hajirahimova, M. S., & Aliyeva, A. S. (2015). Review of statistical analysis methods of high-dimensional data. Eastern-European Journal of Enterprise Technologies, 5(3(77), 23–30. https://doi.org/10.15587/1729-4061.2015.51603

Issue

Vol. 5 No. 3(77) (2015): Control processes

Section

Control processes

License

Copyright (c) 2015 Makrufa Sharif Hajirahimova, Aybeniz Salman Aliyeva

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

The consolidation and conditions for the transfer of copyright (identification of authorship) is carried out in the License Agreement. In particular, the authors reserve the right to the authorship of their manuscript and transfer the first publication of this work to the journal under the terms of the Creative Commons CC BY license. At the same time, they have the right to conclude on their own additional agreements concerning the non-exclusive distribution of the work in the form in which it was published by this journal, but provided that the link to the first publication of the article in this journal is preserved.

A license agreement is a document in which the author warrants that he/she owns all copyright for the work (manuscript, article, etc.).
The authors, signing the License Agreement with TECHNOLOGY CENTER PC, have all rights to the further use of their work, provided that they link to our edition in which the work was published.
According to the terms of the License Agreement, the Publisher TECHNOLOGY CENTER PC does not take away your copyrights and receives permission from the authors to use and dissemination of the publication through the world's scientific resources (own electronic resources, scientometric databases, repositories, libraries, etc.).
In the absence of a signed License Agreement or in the absence of this agreement of identifiers allowing to identify the identity of the author, the editors have no right to work with the manuscript.
It is important to remember that there is another type of agreement between authors and publishers – when copyright is transferred from the authors to the publisher. In this case, the authors lose ownership of their work and may not use it in any way.