Latent semantic method of extraction information from the internet resources

Authors

  • Александр Африканович Стенин National Technical University of Ukraine “Kyiv Polytechnic Institute” Peremogy 37, Kyiv, Ukraine, 03056, Ukraine
  • Юрий Афанасиевич Тимошин National Technical University of Ukraine “Kyiv Polytechnic Institute” Peremogy 37, Kyiv, Ukraine, 03056, Ukraine https://orcid.org/0000-0001-9332-3228
  • Екатерина Юрьевна Мелкумян National Technical University of Ukraine “Kyiv Polytechnic Institute” Peremogy 37, Kyiv, Ukraine, 03056, Ukraine
  • В. В. Курбанов National Technical University of Ukraine “Kyiv Polytechnic Institute” Peremogy 37, Kyiv, Ukraine, 03056, Ukraine

DOI:

https://doi.org/10.15587/1729-4061.2013.16387

Keywords:

internet resources, information retrieval, intelligent agents, descriptors, Zipf’s law

Abstract

Unlike traditional information retrieval systems (IRS) Internet has the following features: as the information warehouse it lacked the search function,thus it was decentralized; the network is social, heterogeneous, combines both modern and previous systems versions; access time to various parts is unequal; the information volume exceeds the largest IRS volume. The main task of IRS in the internet is providing methods and ways of semantic analysis of the text in natural language, which entails the ability of information extraction from the specified HTML documents in the form of certain pieces of information.

The paper suggests the latent semantic method of weighed descriptors, allowingto extract the most meaningful documents the that are close to the subject area of the search, as well as the search algorithm. The method assumes that the conceptual descriptors, based on the Zipf’s law, in sentences have the downstream «latent» meaning obscured by the use of different words. Interpretation of the Zipf’s law is based on the correlation properties of additive Markov chains with a memory step function.

Also, the latent semantic analysis (LSA) is disclosed, which is the method of processing of information in natural language and analyzes the relationship between the documents collection and terms. The LSA can be compared to the simple version of a neural network consisting of three layers

Author Biographies

Александр Африканович Стенин, National Technical University of Ukraine “Kyiv Polytechnic Institute” Peremogy 37, Kyiv, Ukraine, 03056

Professor

Department of Technical Cybernetic

Юрий Афанасиевич Тимошин, National Technical University of Ukraine “Kyiv Polytechnic Institute” Peremogy 37, Kyiv, Ukraine, 03056

Docent

Department of Technical Cybernetic

Екатерина Юрьевна Мелкумян, National Technical University of Ukraine “Kyiv Polytechnic Institute” Peremogy 37, Kyiv, Ukraine, 03056

Ph.D.

Department of Technical Cybernetic  

В. В. Курбанов, National Technical University of Ukraine “Kyiv Polytechnic Institute” Peremogy 37, Kyiv, Ukraine, 03056

Postgraduate Student

Department of Technical Cybernetic

References

  1. Козлов, Д. Д. ИПС в Интернет: текущее состояние и пути развития [Текст] / Д. Д. Козлов. – М.:МГУ. – 2000. – 28 с.
  2. Ландэ, Д. В. Поиск знаний в Internet [Текст] / Д. В. Ландэ. – М.: Диалектика. – 2005. – 28 с.
  3. Мидоу, Ч. Ч. Анализ информационно-поисковых систем [Текст] / Ч. Ч. Мидоу. – М.:Мир. – 1970.
  4. Lawrence, S. Accessibility of Information on the Web [Текст] / S. Lawrence, C. Giles // Nature. – 1999. – vol. 400 – С. 107-109
  5. Hermans, B. Intelligent Software Agents on the Internet [Електронний ресурс] / B. Hermans. – 1996. – 89 с. – Режим доступу: www/ URL: http://www.hermans.org/agents
  6. Bergman, K. The Deep Web: Surfacing Hidden Value, BrightPlanet.com LLC [Електронний ресурс] / K. Bergman. – Режим доступу: www/ URL: http://www.completeplanet.com/Tutorials/DeepWeb/index.asp
  7. Inktomi Corp., Web Surpasses One Billion Documents, press release issued January 18, 2000 [Електронний ресурс]. – Режим доступу: www/ URL: http://www.inktomi.com/new/press/billion.html
  8. Методы и средства извлечения слабоструктурированных схем из документов в HTML и конвертирования HTML документов в их XMLпредставление [Електронний ресурс]. – Режим доступу: www/ URL: http://synthesis.ipi.ac.ru/syntesis/projects/XMLBIS/html2xml.html
  9. Некрестьянов, И. Обнаружение структурного подобия HTML-документов [Електронний ресурс]/ И. Некрестьянов, Е. Павлова. – СпбГУ, 2002. – C. 38-54. – Режим доступу: www/ URL: http://meta.math.spbu.ru
  10. Gerdt, V. P. Computer Algebra and Constrained Dynamics [Текст] / V. P. Gerdt // Problem of Modern Physics. – 2000. – JINR D2-99-263. – C. 164-171
  11. Kechedzhy, K. E. Rank distributions of words in additive many-step Markov chains and the Zipf Law [Текст] / K. E. Kechedzhy, O. V. Ustenko, V. A. Yampol’ski // Arxiv LANL. – 2004. – Phys.Rev.E. – 2005. – V 72. – pp. 1-6
  12. Wentain, Li. Random Texts Exibition Zipf’s Law – Like Word Frequency Distribution. [Текст] / Li. Wentain // Santa Fe institute. NM 87501. – 1992. – V. 38-№6. – C. 1842-1845
  13. Голуб, Дж. Матричные исчисления [Текст] / Дж. Голуб, И. Ван Лоун. М.: Мир. – 1999.
  14. Kozlov, D. (2000). Internet ISS: current status and development. Moskow, MGU, 28.
  15. Landje, D. (2005). Internet search of knowledge. M.: Dialektika, 28.
  16. Midou, Ch. (1970). The analysis of ISS. M.: Mir.
  17. Lawrence, S., Giles, C. (1999). Accessibility of Information on the Web. Nature vol. 400 pp., 107-109.
  18. Hermans, B. (1996). Intelligent Software Agents on the Internet. Available: http://www.hermans.org/agents
  19. Bergman, K. The Deep Web: Surfacing Hidden Value, BrightPlanet.com LLC. Available: http://www.completeplanet.com/Tutorials/DeepWeb/index.asp.
  20. Inktomi Corp. (2000). Web Surpasses One Billion Documents, press release issued January 18. Available: http://www.inktomi.com/new/press/billion.html.
  21. Metody i sredstva izvlechenija slabostrukturirovannyh shem iz
  22. dokumentov v HTML i konvertirovanija HTML dokumentov v ih XML predstavlenie. Available: http://synthesis.ipi.ac.ru/syntesis/projects/XMLBIS/html2xml.html.
  23. Nekrest'janov I., Pavlova E. (2002). Obnaruzhenie strukturnogo podobija HTML-dokumentov. SpbGU, 38-54. Available: http://meta.math.spbu.ru
  24. Gerdt, V. (2000). Computer Algebra and Constrained Dynamics. «Problem of Modern Physics», JINR D2-99-263, 164-171.
  25. Kechedzhy, K., Ustenko O., Yampol’ski V. (2004). Rank distributions of words in additive many-step Markov chains and the Zipf Law. Arxiv LANL. Phys.Rev.E., 1-6.
  26. Wentain, Li. (1992). Random Texts Exibition Zipf’s Law – Like Word Frequency Distribution. Santa Fe institute. NM 87501. V. 38-№6, 1842-1845.
  27. Golub, Dzh. Van Loun, I. (1999). Matrichnye ischislenija. M.: Mir.

Downloads

Published

2013-08-15

How to Cite

Стенин, А. А., Тимошин, Ю. А., Мелкумян, Е. Ю., & Курбанов, В. В. (2013). Latent semantic method of extraction information from the internet resources. Eastern-European Journal of Enterprise Technologies, 4(9(64), 19–22. https://doi.org/10.15587/1729-4061.2013.16387

Issue

Vol. 4 No. 9(64) (2013): Information and control systems

Section

Information and controlling system

License

Copyright (c) 2014 Александр Африканович Стенин, Юрий Афанасиевич Тимошин, Екатерина Юрьевна Мелкумян, В. В. Курбанов

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

The consolidation and conditions for the transfer of copyright (identification of authorship) is carried out in the License Agreement. In particular, the authors reserve the right to the authorship of their manuscript and transfer the first publication of this work to the journal under the terms of the Creative Commons CC BY license. At the same time, they have the right to conclude on their own additional agreements concerning the non-exclusive distribution of the work in the form in which it was published by this journal, but provided that the link to the first publication of the article in this journal is preserved.

A license agreement is a document in which the author warrants that he/she owns all copyright for the work (manuscript, article, etc.).
The authors, signing the License Agreement with TECHNOLOGY CENTER PC, have all rights to the further use of their work, provided that they link to our edition in which the work was published.
According to the terms of the License Agreement, the Publisher TECHNOLOGY CENTER PC does not take away your copyrights and receives permission from the authors to use and dissemination of the publication through the world's scientific resources (own electronic resources, scientometric databases, repositories, libraries, etc.).
In the absence of a signed License Agreement or in the absence of this agreement of identifiers allowing to identify the identity of the author, the editors have no right to work with the manuscript.
It is important to remember that there is another type of agreement between authors and publishers – when copyright is transferred from the authors to the publisher. In this case, the authors lose ownership of their work and may not use it in any way.