This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Brown, A. Search for Related Content Social Bookmarking                         What's this?

The effect of rater variables in the development of an occupation-specific language performance test

University of Melbourne

There has been little research into the effect of raters' background on assessments made in language tests. This article explores the effect that the rater's occupational and linguistic background has on assessments made in an occupation-specific oral language test, the Japanese Language Test for Tour Guides. Assessments of 51 test candidates made by 33 such assessors, including native and near-native speakers of Japanese with backgrounds either in teaching Japanese as a foreign language or in tour guiding in Japanese, were compared in order to determine what effect background has on assessments made on both linguistic and 'real-world' criteria. Multifaceted Rasch analysis was used, as this allows a range of facets to be modelled; for the purposes of this research we were interested in the facet 'raters', and in particular how the scale was applied for each of the assessment categories. These were then compared across the different types of rater. While differences were found in the assessments according to rater type, these were minor and did not point to the unsuitability of any group as a whole; there were no significant differences between the different types of rater in terms of the overall grade awarded. However, there were significant differences in ratings awarded for some individual criteria, and the fact that these reveal different perceptions of what constitutes good performances has obvious implications not only for the selection and training of raters for occupation- specific language performance tests but also for the development of assess ment procedures.

Language Testing, Vol. 12, No. 1, 1-15 (1995)
DOI: 10.1177/026553229501200101


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This article has been cited by other articles:

				J. S. Johnson and G. S. Lim The influence of rater language background on writing performance assessment Language Testing, October 1, 2009; 26(4): 485 - 505. [Abstract] [PDF]

				Y.-H. Kim An investigation into native and non-native teachers' judgments of oral English performance: A mixed methods approach Language Testing, April 1, 2009; 26(2): 187 - 217. [Abstract] [PDF]

				C. W. Stansfield Lecture: `Where we have been and where we should go' Language Testing, July 1, 2008; 25(3): 311 - 326. [PDF]

				R. Schoonen, M. Vergeer, and M. Eiting The assessment of writing ability: expert readers versus lay readers Language Testing, July 1, 1997; 14(2): 157 - 184. [Abstract] [PDF]

				M. Chalhoub-Deville Theoretical models, assessment frameworks and test construction Language Testing, March 1, 1997; 14(1): 3 - 22. [Abstract] [PDF]

				T. Lumley and T.F. McNamara Rater characteristics and rater bias: implications for training Language Testing, March 1, 1995; 12(1): 54 - 71. [Abstract] [PDF]