Language Testing current issue

The influence of rater language background on writing performance assessment

Johnson, J. S., Lim, G. S. — Mon, 05 Oct 2009 07:20:41 PDT

Language performance assessments typically require human raters, introducing possible error. In international examinations of English proficiency, rater language background is an especially salient factor that needs to be considered. The existence of rater language background-related bias in writing performance assessment is the object of this study. Data for this study are ratings assigned by Michigan English Language Assessment Battery (MELAB) raters to compositions written by examinees of various language backgrounds. While most of the raters are native speakers of English, four have first languages other than English: two Spanish, one Korean, and one bilingual speaker of Filipino and Chinese (Amoy). Examinees were divided into 21 language groups. The IRT application FACETS was used to estimate and control for rater severity when calculating the amount of bias reflected by each rater’s set of ratings for each language/language group. Results show that the magnitude of bias terms for all raters for all language groups was minimal, thus having little effect on examinee scores, and that there is no pattern of language-related bias in the ratings.

Score generalizability of academic writing tasks: Does one test method fit it all?

Gebril, A. — Mon, 05 Oct 2009 07:20:41 PDT

Generalizability of writing scores has always been a longstanding concern in L2 writing assessment. A number of studies have been conducted to investigate this topic during the last two decades. However, with the introduction of new test methods, such as reading-to-write tasks, generalizability studies need to focus on the score accuracy of different task types. The current study attempts to examine how reading-to-write tasks are compara ble to independent tasks in terms of score generalizability. To look into this issue, 115 Egyptian university students were asked to write on two independent tasks and two reading-to-write tasks. The essays were holistica lly scored by three raters, who received training prior to scoring. The study design followed a fully crossed univariate design (p x t x r), and consequently GENOVA (Crock & Brennan, 1983) was used to analyze the results. Results showed that the reading-to-write tasks yielded as reliable scores as independent tasks. In addition, the result indicated that score generalizability is very low when using one writing task due to the large (pt) variance component. Implications and limitations of the study are presented.

Investigating differences in the writing performance of international and Generation 1.5 students

di Gennaro, K. — Mon, 05 Oct 2009 07:20:41 PDT

Practitioners working closely with second language (L2) writers in the US recognize at least two types of L2 students: international (IL2) and Generation 1.5 (G1.5) students. Some argue that specific differences in each group’s writing performance are evident (cf. Harklau, 2003; Reid, 2006); however, investigations into observable and measurable differences have been minimal. Using a Rasch measurement model, this study offers empiri cal evidence of how these students may differ in their writing performance, specifically with regard to grammatical, rhetorical, cohesive, sociolinguistic, and content c ontrol, in addition to essay length. Two separate statistical analyses were conducted: the first examined differences between IL2 and G1.5 students when modeled together as a single group; the second looked at how the groups compared when modeled separately. Results from the whole-group analysis showed the two groups differed only with regard to rhetorical control; the separate-group analysis revealed differences in length and in the difficulty rankings of the scoring components, with content control ranked at opposite extremes for the two groups. In neither analysis did the groups d iffer with regard to grammatical, cohesive, or sociolinguistic control as defined in this study. Results are interpreted in terms of students’ strengths and weaknesses relevant to placement decisions and instruction.

Discourse synthesis in integrated second language writing assessment

Plakans, L. — Mon, 05 Oct 2009 07:20:41 PDT

As integrated tasks become more common in assessing writing for academic purposes, it is necessary to investigate how test takers approach these tasks. The present study explores the processes of test takers undertaking reading-to-write tasks developed for a university English placement exam. Think-aloud protocols and interviews of university-level non-native writers of English were collected to capture the writers’ composing processes. These data were analysed to determine if the tasks elicited an academic writing process of discourse synthesis and to identify what language issues were present. The results revealed some writers use discourse synthesis subprocesses in composing responses to the reading-to-write tasks, with apparent d ifferences across writers. Language difficulties that emerged for writers were vocabulary knowledge and use, as well as stylistic concerns. The implications of the study suggest that reading-to-write tasks may elicit a discourse synthesis process that could be used for validity evidence in some contexts and lead to more appropriate placement into academic writing courses.

Appropriateness of the TOEIC(R) Bridge test for students in three countries of South America

Mon, 05 Oct 2009 07:20:41 PDT

In order to facilitate the interpretation of test scores from the TOEIC^® Bridge as a measure of English language proficiency, one form of the test was administered to more than 6000 test takers in three South American c ountries — Colombia, Chile and Ecuador. The appropriateness of the TOEIC Bridge test as a measure of English language skill was evaluated using several approaches, such as factor analysis and computation of correla tions between the TOEIC Bridge scores and other measures like local English test scores, student self-assessment scores, and teacher ratings. The evidence collected was generally consistent with the interpretation of TOEIC Bridge scores as indicators of English language competencies for the students examined.

Test review: Test of English as a Foreign LanguageTM: Internet-based Test (TOEFL iBT(R))

Alderson, J. C. — Mon, 05 Oct 2009 07:20:41 PDT

List of reviewers

Mon, 05 Oct 2009 07:20:41 PDT