AUTOMATION OF TEST ITEMS QUALITY ANALYSIS AND IT STUDENTS TEST RESULTS ASSESSMENT
DOI:
https://doi.org/10.30837/2522-9818.2020.11.019Keywords:
statistic methods, IRT models, Rasch model, test quality, test items, knowledge testing, validity, reliability, discriminant abilityAbstract
The subject of this article is statistical models and methods for test items quality assessment, as well as processes and information support technologies for test results and test quality evaluating. The goal is to improve the quality of test items and the accuracy of the evaluation of test results in the context of constant changes in the content of study programs and the content of test tasks, and to reduce the time for test results processing through automation, and developed algorithms and an information system usage. The following tasks was solved: analysis of the test preparation and testing process; an analytical review of statistical methods for the test items quality assessment and the test items quality indicators; generalized process of the test preparation and conduct of testing development; and automation of IT students test results assessment. To solve these problems, the following methods was used: statistical analysis methods, the Rasch IRT (Item Response Theory) model, and software application development technologies were used. The following results are obtained. The process of the test preparation and testing was analyzed; its modification is proposed taking into account the peculiarities of IT students testing. The test task quality indicators, models and methods for it calculation were selected, in particular, the Rasch IRT model. The interpretation of the test item quality indicators values, and the calculation of the coefficients of the anomalies in the test item quality, what are the basis for the generation of an advisory solution to improve the test quality, are given. A prototype of an information system for support of processes of test items quality indicators evaluating, test results assessment and decision making about test modification has been developed. The system was tested during the module control of third-year students results evaluating. Conclusions: automation and information support of the test items quality indicators calculating and analyzing, and test results assessment will improve the accuracy of the assessment and reduce the time for test results evaluating.
References
Avanesov, V. S. (1994), Scientific problems of test knowledge measurement [Nauchnye problemy testovogo kontrolya znanii], Moscow, ITsPKPS, 86 p.
Avanesov, V. S. (2006), Form of the tests [Forma testovykh zadanii], Moscow, Tsentr testirovaniya Publ., 156 p.
Omarov, М., Muradova, V. (2018), "Infological support of the information technology for managemant of distance education", Innovative Technologies and Scientific Solutions for Industries, No. 4 (6), P. 146–153. DOI: https://doi.org/10.30837/2522-9818.2018.6.146
Kovalʹčuk, Ju. O. (2012), Educational Measurement Theory [Teorija osvitnix vymirjuvanʹ], Nizhyn, Publisher PP Lysenko M. M., 200 p.
Lisova, T. V. (2012), Models and methods of the modern test theory [Modeli ta metody suchasnoyi teoriyi testiv], Nizhyn, Publisher PP Lysenko M. M. 112 p.
An, X. and Yung, Y. (2014), "Item Response Theory: What it is and how you can use the IRT procedure to apply it", Paper SAS364-2014 – Cary, NC, USA : SAS Institute Inc., 14 p., available at : http://support.sas.com/resources/papers/proceeding (last accessed 11.02.2020).
Razak, N. A., Razak, A. Z. K. and Lei, M.T. (2012), "Examining Quality of Mathemtics Test Items Using Rasch Model: Preminarily Analysis", Procedia - Social and Behavioral Sciences, Vol. 69, P. 2205–2214. DOI: https://doi.org/10.1016/j.sbspro.2012.12.187
Maiorov, A. N. (2002), Theory and practice of tests creating for the education system [Teoriya i praktika sozdaniya testov dlya sistemy obrazovaniya], Moscow, Intellekt-Tsentr Publ., 267 p.
Malieieva, Ju., Kosenko, V., Malyeyeva, O., Svetlichnyj, D. (2019), "Creation of collaborative development in the system of distance learning", Innovative Technologies and Scientific Solutions for Industries, No. 2 (8), P. 62–71. DOI: https://doi.org/10.30837/2522-9818.2019.8.062
Croker, L., Algina, J. (2010), Introduction to Classical and Modern Test Theory. [Vvedenie v klassicheskuyu i sovremennuyu teoriyu testov], Moscow, Logos Publ., 668 p.
Rasch, G. (1980), Probabilistic Models for Some Intelligence and Attainment Tests. Copenhagen, Danish Institute of Educational Research. (Expanded edition), 324 p.
Kim, V. S. (2005), "Test results analysis during Rasch measurement" ["Analiz rezul'tatov testirova-niya v protsesse Rasch measurement"], Pedagogicheskie izmereniya – Pedagogical Measurements, Vol. 4, P. 39–45.
Rusch, T., Lowry, P. B., Mair, P., Treiblmaier, H. (2017), "Breaking free from the limitations of classical test theory: Developing and measuring information systems scales using item response theory", Information & Management, Vol. 54, Issue 2, P. 189–203. DOI: https://doi.org/10.1016/j.im.2016.06.005
Wilson, M. (2005), Constructing Measures: An Item Response Modeling Approach, NJ : Lawrence Erlbaum Associates, 228 p. DOI: https://doi.org/10.4324/9781410611697
Mikheev, O. V. (2004), "The mathematical model of pedagogical measurements" ["Matematicheskie modeli pedagogicheskikh izmerenii"], Pedagogicheskie izmereniya – Pedagogical Measurements, Vol. 2, P. 75–88.
Dykhovichnyi, A. A., Dudko, A. F. (2013), "Automated system for computer testing in higher mathematics results analysis" ["Avtomatizirovannaya sistema analiza rezul'tatov komp'yuternogo testirovaniya po vysshei matematike"], Naukovі pratsі DonNTU. Serіya: "Pedagogіka, psikhologіya і sotsіologіya", Vol. 2 (14), P. 103–110.
Suparman, U. (2016), "Analyzing the quality of english test items of daily, mid semester and final school examinations in bandar lampung: (assessment and evaluation in language teaching)", International Conference On Educational Research & Evaluation, Yogyakarta, 14 p. URL: http://repository.lppm.unila.ac.id/id/eprint/3121 (last accessed 11.02.2020).
Quaigrain, K., Arhin, A. K. (2017), "Using reliability and item analysis to evaluate a teacher-developed test in educational measurement and evaluation", Cogent Education, Vol. 4 (1). P. 1–11. DOI: https://doi.org/10.1080/2331186X.2017.1301013.
Downloads
How to Cite
Issue
Section
License
Copyright (c) 2020 Leonid Smidovych, Olesia Artiukh, Nataliia Kosenko
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Our journal abides by the Creative Commons copyright rights and permissions for open access journals.
Authors who publish with this journal agree to the following terms:
Authors hold the copyright without restrictions and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0) that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
Authors are able to enter into separate, additional contractual arrangements for the non-commercial and non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
Authors are permitted and encouraged to post their published work online (e.g., in institutional repositories or on their website) as it can lead to productive exchanges, as well as earlier and greater citation of published work.
