Development of a Writing Rating Scale for Culture- and Context-Specific Educational Setting in Ukraine
DOI:
https://doi.org/10.17721/2663-0303.2022.1.02Keywords:
writing scale, summative writing test, context-specific settings of educationAbstract
In the Ukrainian university context, the assessment of writing in language programs has traditionally been performed by the individual teacher, with no institutionally mandated scales or criteria to rely on. The use of rating scales from external proficiency tests cannot meet the requirement of total alignment with the curricula taught in context-specific settings. This is obvious in terms of frequent mismatches of course objectives and test construct as well as task types employed on the external and local tests. To address he tissue, this paper reports on efforts to introduce a rating scale for collective use by teachers across a department teaching English to students majoring in Linguistics. Although teacher raters can be trained to employ external rating scales consistently, we believe that it is both more valuable in terms of professional development and more likely to result in acceptance of the scales if they engage in scale development. The evidence we collected suggests that the collaborative approach adopted in this study can offer a sustainable template for improving the poor standards of tests and examinations in our country (and elsewhere), which often result from a lack of testing and assessment expertise among those who prepare assessment materials.
References
Aryadoust, V. (2010). Investigating Writing Sub-skills in Testing English as a Foreign Language: A Structural Equation Modeling Study. TESL-EJ, 13(4). https://www.tesl-
Banerjee, J., Franceschina, F., & Smith, A. M. (2007). Documenting features of written language production typical at different IELTS band score levels [IELTS Research Report No. 7, the British Council/ University of Cambridge Local Examinations Syndicate]. https://www.ielts.org/research/research-reports/volume-07-report-5
Becker, A. (2018). Not to scale? An argument-based inquiry into the validity of an L2 writing rating scale. Assessing Writing, 37, 1–12. https://doi.org/10.1016/j.asw.2018.01.001
Black, P., Harrison, C., Lee, C., Marshall, B., & Wiliam, D. (2004). Working inside the black box: Assessment for learning in the classroom. Phi Delta Kappan, 86(1), 8–21. https://doi.org/10.1177/003172170408600105
Bolitho. R., & West, R. (2017). The internationalisation of Ukrainian universities: The English language dimension [British Council Ukraine report on English for Universities Project]. https://www.teachingenglish.org.uk/article/internationalisation-ukrainian-universities-english-language-dimension
Cho, D. (2008). Investigating EFL writing assessment in a classroom setting: Features of composition and rater behaviors. The Journal of Asia TEFL, 5(4), 49–84. https://www.earticle.net/Article/A182195
Clandfield et al., (2010). Global Upper Intermediate Student's Book. Macmillan ELT.
Crusan, D., Plakans, L., & Gebril, A. (2016). Writing assessment literacy: Surveying second language teachers’ knowledge, beliefs, and practices. Assessing Writing, 28, 43–56. https://doi.org/10.1016/j.asw.2016.03.001
Ducasse, A. M., & Hill, K. (2015). Development of a Spanish generic writing skills scale for the Colombian graduate skills assessment (SaberPro). Papers in Language Testing and Assessment, 4(2), 18–33. https://arts.unimelb.edu.au/__data/assets/pdf_file/0009/1770606/2.Ducasse-And-Hill.pdf
Fulcher, G. (2003). Testing second language speaking. Pearson Longman.
Gamaroff, R. (2000). Rater reliability in language assessment: The bug of all bears. System, 28(1), 31–53. https://doi.org/10.1016/S0346-251X(99)00059-7
Green, A. (2014). Exploring language assessment and testing: Language in action. Routledge. https://doi.org/10.4324/9781315889627
Hamp-Lyons, L. (1990). Second language writing: Assessment issues. In B. Kroll (Ed.), Second language writing assessment issues and options. Macmillan. https://doi.org/10.1017/CBO9781139524551.009
Hamp-Lyons, L. (1995). Rating nonnative writing: The trouble with holistic scoring. TESOL Quarterly, 29, 759-762.Hawkey, R., & Barker, F. (2004). Developing a common scale for the assessment of writing. Assessing Writing, 9, 122–159. https://doi.org/10.2307/3588173
Heaton, J.B. (1991). Writing English language tests. Longman.
Hill, K., & Ducasse, A.M. (2020). Advancing written feedback practice through a teacher-researcher collaboration in a university Spanish program. In M. Poehner, O. Inbar-Lourie (Eds.), Toward a reconceptualization of second language classroom assessment. Educational Linguistics (vol. 41). Springer, Cham. https://doi.org/10.1007/978-3-030-35081-9_8
Jeong, H. (2015). Rubrics in the classroom: do teachers really follow them? Language Testing in Asia, 5(6). https://doi.org/10.1186/s40468-015-0013-5
Kkese, E. T. (2018). Assessing L2 Writing in the Absence of Scoring Procedures: Construction of Rating Scales in a Cypriot Greek EFL in Class Context. Journal of English Education, 3(2), 46–56. https://doi.org/10.31327/jee.v3i2.809
Knoch, U. (2007). ‘Little coherence, considerable strain for reader’: A comparison between two rating scales for the assessment of coherence. Assessing Writing, 12, 108–128. https://doi.org/10.1016/j.asw.2007.07.002
Knoch, U. (2009). Diagnostic assessment of writing: A comparison of two rating scales. Language Testing, 26(2), 275–304. https://doi.org/10.1177/0265532208101008
Knoch, U. (2011). Rating scales for diagnostic assessment of writing: What should they look like and where should the criteria come from? Assessing Writing, 16, 81–96. https://doi.org/10.1016/j.asw.2011.02.003
Kvasova, O. & Kavytska, T. (2014). The assessment competence of university foreign language teachers: A Ukrainian perspective. CerleS, 4(1), 159–177. https://doi.org/10.1515/cercles-2014-0010
Kvasova, O., Kavytska, T. & Osidak, V. (2019). Investigation of writing assessment literacy of Ukrainian University teachers. Ars linguodidacticae, 4(2), 10–19. https://doi.org/10.17721/2663-0303.2019.4.02
Lim, G. S. (2011). The development and maintenance of rating quality in performance writing. assessment: A longitudinal study of new and experienced raters. Language Testing, 28, 543–560. https://doi.org/10.1177/0265532211406422
Lumley, T. (2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19(3), 246–276. https://doi.org/10.1191/0265532202lt230oa
Mai, D. (2019). A review of theories and research into second language writing assessment criteria. VNU Journal of Foreign Studies, 35(3). https://doi.org/10.25073/2525-2445/vnufs.4371
Mellati, M., & Khademi, M. (2018). Exploring teachers’ assessment literacy: Impact on learners’ writing achievements and implications for teacher development. Australian Journal of Teacher Education, 43(6). http://dx.doi.org/10.14221/ajte.2018v43n6.1.
Mendoza, A., & Knoch, U. (2017). Examining the validity of an analytic rating scale for a Spanish test for academic purposes. Assessing writing, 35, 41–55. https://doi.org/10.1016/j.asw.2017.12.003
North, B., & Schneider, G. (1998). Scaling descriptors for language proficiency scales. Language Testing, 15(2), 217–263. https://doi.org/10.1177/026553229801500204
Plakans, L. (2013). Writing scale development and use within a language program. TESOL Journal, 4(1). https://doi.org/10.1002/tesj.66
Skar, G. B., & Jølle, L. J. (2017). Teachers as raters: An investigation of a long-term writing assessment program. L1-Educational Studies in Language and Literature, 17, 1–30. https://doi.org/10.17239/L1ESLL-2017.17.01.06
Turner, C. E., & Upshur, J. A. (2002). Rating scales derived from student samples: Effects of the scale maker and the student sample on scale content and student scores. TESOL Quarterly, 36(1), 49–70. https://doi.org/10.2307/3588360
Vaughan, C. (1991). Holistic assessment: What goes on in the raters’ mind? In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 111–126). Norwood, NJ: Ablex.
Vogt, K. & Tsagari, D. (2014). Assessment literacy of foreign language teachers: Findings of a European study. Language Assessment Quarterly, 11(4), 374–402. http://dx.doi.org/10.1080/15434303.2014.960046
Weigle, S. C. (2002). Assessing writing. Cambridge University Press. https://doi.org/10.1017/CBO9780511732997
Weigle, S.C. (2007). Teaching writing teachers about assessment. Journal of Second Language Writing, 16(3), 194–209. https://doi.org/10.1016/j.jslw.2007.07.004
Yamanishi, H., Ono, M., & Hijikata, Y. (2019). Developing a scoring rubric for L2 summary writing: A hybrid approach combining analytic and holistic assessment. Language Testing in Asia, 9(13). https://doi.org/10.1186/s40468-019-0087-6
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright policy according to the terms of the license: Creative Commons "Attribution-NonCommercial" 4.0 International (CC BY-NC 4.0).
Authors who publish their articles in "Ars Linguodidacticae" (Open Access Journal) retain the following rights:
- The authors retain the copyright of their article and grant the Ars Linguodidacticae journal the right to first publish the manuscript of their article under the Creative Commons (CC BY-NC 4.0) Attribution License, which allows others to freely distribute the published work with mandatory reference to the author of the original work and first original publication in the Ars Linguodidacticae journal. An indication of the retention of the copyright of the work is provided on the title page of the article.
- The authors reserve the right to enter into separate contracts for the non-exclusive distribution of their article as published in Ars Linguodidacticae (e.g., placing the article in electronic libraries, archives and catalogs or publishing it as part of institute collections and monographs), provided that a full reference to the first original publication in Ars Linguodidacticae is given.
- The policy of the "Ars Linguodidacticae" journal allows and encourages authors to post a manuscript both before and during editorial processing, as this promotes productive scientific discussion and has a positive effect on the speed and dynamics of citing the article.
The editorial board reserves publishing rights to:
- the collated original articles and to the entire issue of the journal.
- the design of the journal and original illustrative and supplementary materials.
- the reprint reprints of the Journal in printed and electronic form.
The copyright policy is carried out according to the terms of the license: Creative Commons "Attribution-NonCommercial" 4.0 International (CC BY-NC 4.0).
For more information, please read the full text of the CC BY-NC 4.0 Public License.
Creative Commons Attribution-NonCommercial 4.0 International License.
