This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Kondo-Brown, K. Search for Related Content Social Bookmarking                   What's this?

A FACETS analysis of rater bias in measuring Japanese second language writing performance

University of Hawaii at Manoa, kondo{at}hawaii.edu

The purpose of this study is twofold. First, using FACETS (Linacre, 1996), it investigates how judgements of trained teacher raters are biased towards certain types of candidates and certain criteria in assessing Japanese second language (L2) writing. Previous studies that identified significantly biased rater-candidate interactions did not discuss who the candidates were, but this study examines rater-candidate interactions in much more detail. Secondly, since there is no established rating scale for assessing Japanese L2 writing, this study explores the potential for using a modified version of Jacobs et al.’s (1981) rating scale for norm-referenced decisions about Japanese L2 writing ability. The participants in the study comprised 234 university candidates and three trained teacher raters. The raters produced highly correlated scores and were self-consistent, but significant differences in overall severity surfaced. The raters scored certain candidates and criteria more leniently or harshly, and every rater’s bias pattern was different. The highest percentage of significantly biased rater-candidate interactions was found among the candidates whose ability was extremely high or low. This study suggests that the modified version of Jacobs et al.’s scale can be a reliable tool in assessing Japanese L2 writing in norm-referenced settings, but multiple ratings are still necessary.

Language Testing, Vol. 19, No. 1, 3-31 (2002)
DOI: 10.1191/0265532202lt218oa


CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

This article has been cited by other articles:

				S. Matsuno Self-, peer-, and teacher-assessments in Japanese university EFL writing classrooms Language Testing, January 1, 2009; 26(1): 075 - 100. [Abstract] [PDF]

				E. Schaefer Rater bias patterns in an EFL writing assessment Language Testing, October 1, 2008; 25(4): 465 - 493. [Abstract] [PDF]

				Y. Sawaki Construct validation of analytic rating scales in a speaking assessment: Reporting a score profile and a composite Language Testing, July 1, 2007; 24(3): 355 - 390. [Abstract] [PDF]