Rater behavior and background factors in writing assessment : a mixed-method study of how Indonesian university lecturers assess EFL essays

Abstract

This dissertation examines lecturers' rating behavior in assessing EFL essays in Indonesian higher education context. Specifically, it explores their rating behavior in terms of severity, consistency, and bias/interaction effects between lecturers' background factors and specific aspects of rubric criteria such as Content, Organization, Vocabulary, Language Use, and Mechanics. This dissertation also explores how lecturers describe and reflect on their rating behavior, including how they perceive fairness, how they prioritize rubric criteria, and how they maintain assessment consistency. The research employed an explanatory sequential mixed method design, beginning with quantitative phase to identify patterns in rating behavior. Participants in this research are thirty-six university lecturers who rated forty-five EFL student essays using the ESL Composition Profile rubric. Prior to the rating sessions, rater trainings were conducted to establish shared understanding of the rubric. The rating data were analyzed using Many Facet Rasch Measurement model to estimate rater severity, consistency, and bias/interaction effects among raters, their background variables, and rubric criteria. Afterwards, qualitative phase was conducted in the form of semi-structured interviews to selected lecturers based on their estimated rating behavior. Lastly, integration analysis was conducted by connecting qualitative themes to quantitative patterns in order to explain observed differences in rating behavior. From the quantitative results, it can be seen that there are substantial differences in rating severity among the lecturers. They are also differed in terms of consistency, where some rated consistently, some rated too predictable, and some others rated inconsistently with fluctuating patterns. Subsequently, bias/interaction analysis reveals that lecturer background factors contribute differently to rating severity. Then, the qualitative findings show that rating behavior is shaped by teaching experience, emotional condition, workload, and institutional context. They also prioritized different writing components and employed various strategies to maintain consistency. Findings conclude that severity is affected by lecturers' linguistic priorities, professional experience, work habits, and their belief about fairness in assessment. This dissertation contributes to understanding rater behavior in writing assessment by showing how lecturer background and professional experience shape their rating behavior beyond the rubrics. It also extends research on rater-mediated assessment by providing evidence from under explored contexts in Indonesian higher education. This research suggests institutional support to improve and to provide rater training to address workload management, cognitive load, and lecturers' well-being during the assessment process, in addition to rubric-related training, to support more accurate, consistent, and fair writing assessment practices. Future research could apply extended rubric, more essay genres, as well as new demographic aspects not covered in this research. This research can also be expanded to more participation from wider range of higher education institutions in Indonesia. Additionally, a longitudinal study would offer more insights into severity that can grow due to experience, exposure to different students, or curriculum changes.