Language Testing recent issues

The influence of rater language background on writing performance assessment

Johnson, J. S., Lim, G. S. — Mon, 05 Oct 2009 07:20:41 PDT

Language performance assessments typically require human raters, introducing possible error. In international examinations of English proficiency, rater language background is an especially salient factor that needs to be considered. The existence of rater language background-related bias in writing performance assessment is the object of this study. Data for this study are ratings assigned by Michigan English Language Assessment Battery (MELAB) raters to compositions written by examinees of various language backgrounds. While most of the raters are native speakers of English, four have first languages other than English: two Spanish, one Korean, and one bilingual speaker of Filipino and Chinese (Amoy). Examinees were divided into 21 language groups. The IRT application FACETS was used to estimate and control for rater severity when calculating the amount of bias reflected by each rater’s set of ratings for each language/language group. Results show that the magnitude of bias terms for all raters for all language groups was minimal, thus having little effect on examinee scores, and that there is no pattern of language-related bias in the ratings.

Score generalizability of academic writing tasks: Does one test method fit it all?

Gebril, A. — Mon, 05 Oct 2009 07:20:41 PDT

Generalizability of writing scores has always been a longstanding concern in L2 writing assessment. A number of studies have been conducted to investigate this topic during the last two decades. However, with the introduction of new test methods, such as reading-to-write tasks, generalizability studies need to focus on the score accuracy of different task types. The current study attempts to examine how reading-to-write tasks are compara ble to independent tasks in terms of score generalizability. To look into this issue, 115 Egyptian university students were asked to write on two independent tasks and two reading-to-write tasks. The essays were holistica lly scored by three raters, who received training prior to scoring. The study design followed a fully crossed univariate design (p x t x r), and consequently GENOVA (Crock & Brennan, 1983) was used to analyze the results. Results showed that the reading-to-write tasks yielded as reliable scores as independent tasks. In addition, the result indicated that score generalizability is very low when using one writing task due to the large (pt) variance component. Implications and limitations of the study are presented.

Investigating differences in the writing performance of international and Generation 1.5 students

di Gennaro, K. — Mon, 05 Oct 2009 07:20:41 PDT

Practitioners working closely with second language (L2) writers in the US recognize at least two types of L2 students: international (IL2) and Generation 1.5 (G1.5) students. Some argue that specific differences in each group’s writing performance are evident (cf. Harklau, 2003; Reid, 2006); however, investigations into observable and measurable differences have been minimal. Using a Rasch measurement model, this study offers empiri cal evidence of how these students may differ in their writing performance, specifically with regard to grammatical, rhetorical, cohesive, sociolinguistic, and content c ontrol, in addition to essay length. Two separate statistical analyses were conducted: the first examined differences between IL2 and G1.5 students when modeled together as a single group; the second looked at how the groups compared when modeled separately. Results from the whole-group analysis showed the two groups differed only with regard to rhetorical control; the separate-group analysis revealed differences in length and in the difficulty rankings of the scoring components, with content control ranked at opposite extremes for the two groups. In neither analysis did the groups d iffer with regard to grammatical, cohesive, or sociolinguistic control as defined in this study. Results are interpreted in terms of students’ strengths and weaknesses relevant to placement decisions and instruction.

Discourse synthesis in integrated second language writing assessment

Plakans, L. — Mon, 05 Oct 2009 07:20:41 PDT

As integrated tasks become more common in assessing writing for academic purposes, it is necessary to investigate how test takers approach these tasks. The present study explores the processes of test takers undertaking reading-to-write tasks developed for a university English placement exam. Think-aloud protocols and interviews of university-level non-native writers of English were collected to capture the writers’ composing processes. These data were analysed to determine if the tasks elicited an academic writing process of discourse synthesis and to identify what language issues were present. The results revealed some writers use discourse synthesis subprocesses in composing responses to the reading-to-write tasks, with apparent d ifferences across writers. Language difficulties that emerged for writers were vocabulary knowledge and use, as well as stylistic concerns. The implications of the study suggest that reading-to-write tasks may elicit a discourse synthesis process that could be used for validity evidence in some contexts and lead to more appropriate placement into academic writing courses.

Appropriateness of the TOEIC(R) Bridge test for students in three countries of South America

Mon, 05 Oct 2009 07:20:41 PDT

In order to facilitate the interpretation of test scores from the TOEIC^® Bridge as a measure of English language proficiency, one form of the test was administered to more than 6000 test takers in three South American c ountries — Colombia, Chile and Ecuador. The appropriateness of the TOEIC Bridge test as a measure of English language skill was evaluated using several approaches, such as factor analysis and computation of correla tions between the TOEIC Bridge scores and other measures like local English test scores, student self-assessment scores, and teacher ratings. The evidence collected was generally consistent with the interpretation of TOEIC Bridge scores as indicators of English language competencies for the students examined.

Test review: Test of English as a Foreign LanguageTM: Internet-based Test (TOEFL iBT(R))

Alderson, J. C. — Mon, 05 Oct 2009 07:20:41 PDT

List of reviewers

Mon, 05 Oct 2009 07:20:41 PDT

Are two heads better than one? Pair work in L2 assessment contexts

Taylor, L., Wigglesworth, G. — Wed, 24 Jun 2009 07:35:45 PDT

Interacting in pairs in a test of oral proficiency: Co-constructing a better performance

Brooks, L. — Wed, 24 Jun 2009 07:35:45 PDT

This study, framed within sociocultural theory, examines the interaction of adult ESL test-takers in two tests of oral proficiency: one in which they interacted with an examiner (the individual format) and one in which they interacted with another student (the paired format). The data for the eight pairs in this study were drawn from a larger study comparing the two test formats in the context of high-stakes exit testing from an Academic Preparation Program at a large Canadian university. All of the test-takers participated in both test formats involving a discussion with comparable speaking prompts. The findings from the quantitative analyses show that overall the test-takers performed better in the paired format in that their scores were on average higher than when they interacted with an examiner. Qualitative analysis of the test-takers' speaking indicates that the differences in performance in the two test formats were more marked than the scores suggest. When test-takers interacted with other students in the paired test, the interaction was much more complex and revealed the co-construction of a more linguistically demanding performance than did the interaction between examiners and students. The paired testing format resulted in more interaction, negotiation of meaning, consideration of the interlocutor and more complex output. Among the implications for test theory and practice is the need to account for the joint construction of performance in a speaking test in both construct definitions and rating scales.

The influence of interlocutor proficiency in a paired oral assessment

Davis, L. — Wed, 24 Jun 2009 07:35:45 PDT

The use of pair work in speaking assessment may encourage collaboration in the classroom and has other advantages (Saville & Hargreaves, 1999; Taylor, 2000) but from a measurement perspective, the paired oral format may be problematic because a partner may unfairly influence an examinee's performance or otherwise bias scores. In this study, the influence of interlocutor proficiency on speaking performance was examined in a group of 20 first-year students at a Chinese university. The students were divided into groups of relatively high and low English proficiency and tested once with a partner of similar proficiency and once with a partner of higher or lower proficiency. Interlocutor proficiency level had no observable effect on Rasch analysis ability measures, but lower-level examinees produced more language (words) when working with a higher-level partner. The majority of dyads produced collaborative interactions (Galaczi, 2008), unless an examinee was paired with a much lower-level partner, in which case the interaction tended to be asymmetric. Overall, these data suggest that proficiency differences among examinees need not preclude use of the paired oral test format.

Co-constructed interaction in a paired speaking test: The rater's perspective

May, L. — Wed, 24 Jun 2009 07:35:45 PDT

The definition and operationalization of interactional competence in speaking tests that entail co-construction of discourse is an area of language testing requiring further research. This article explores the reactions of four trained raters to paired candidates who oriented to asymmetric patterns of interaction in a discussion task. Through an analysis of candidate discourse combined with rater notes, stimulated verbal recalls, rater discussions and scores awarded for interactional effectiveness, the article examines the extent to which raters compensate or penalize candidates for their role in co-constructing asymmetric interactional patterns. The article argues that key features of the interaction are perceived by the raters as mutual achievements, and it further suggests that the awarding of shared scores for interactional competence is one way of acknowledging the inherently co-constructed nature of interaction in a paired speaking test.

Assessing paired orals: Raters' orientation to interaction

Ducasse, A. M., Brown, A. — Wed, 24 Jun 2009 07:35:45 PDT

Speaking tasks involving peer-to-peer candidate interaction are increasingly being incorporated into language proficiency assessments, in both large-scale international testing contexts, and in smaller-scale, for example course-related, ones. This growth in the popularity and use of paired and group orals has stimulated research, particularly into the types of discourse produced and the possible impact of candidate background factors on performance. However, despite the fact that the strongest argument for the validity of peer-to-peer assessment lies in the claim that such tasks allow for the assessment of a broader range of interactional skills than the more traditional interview-format tests do, there is surprisingly little research into the judgments that are made of such performances. The fact that raters, and rating criteria, are in a crucial mediating position between output and outcomes, warrants investigation into how raters construe the interaction in these tasks. Such investigations have the potential to inform the development of interaction-based rating scales and ensure that validity claims are moved beyond the content level to the construct level. This paper reports the findings of a verbal protocol study of teacher-raters viewing the paired test discourse of 17 beginner dyads in a university-based Spanish as a foreign language course. The findings indicate that the raters identified three interaction parameters: non-verbal interpersonal communication, interactive listening, and interactional management. The findings have implications for our understanding of the construct of effective interaction in paired candidate speaking tests, and for the development of appropriate rating scales.

Pair versus individual writing: Effects on fluency, complexity and accuracy

Wigglesworth, G., Storch, N. — Wed, 24 Jun 2009 07:35:45 PDT

The assessment of oral language is now quite commonly done in pairs or groups, and there is a growing body of research which investigates the related issues (e.g. May, 2007). Writing generally tends to be thought of as an individual activity, although a small number of studies have documented the advantages of collaboration in writing in the second language classroom (e.g. DiCamilla & Anton, 1997; Storch, 2005; Swain & Lapkin, 1998). Particularly in university contexts, group or pair assignments are widely used in many disciplines. In addition, collaborative writing could be used in second language classroom assessment contexts as formative assessment. However, research which compares texts produced by learners collaboratively to texts produced individually, and the implications of this for assessment practices, is rare.

This study is a first step in the investigation of using collaborative writing in second language contexts and comparing the performance of two groups of second language learners: one group worked individually, and the other group worked in pairs. When writing in pairs, each pair produced a single text. All participants completed one writing task: an argumentative essay. The performances of the individuals (N = 48) and the pairs (N = 48) were compared on detailed discourse analytic measures of fluency, complexity and accuracy. This comparison revealed that collaboration impacted positively on accuracy, but did not affect fluency and complexity. A detailed analysis of the pair transcripts recorded during the writing activity provides insights into the ways in which pairs work together, and the foci of their endeavour. The implications of these findings for in-class assessment of second language writing are discussed.

Test review: Review of the Certificate of Proficiency in English (CPE) Speaking Test

Macqueen, S., Harding, L. — Wed, 24 Jun 2009 07:35:45 PDT

Book review: Young, R. F. (2008). Language and interaction: An advanced resource book (Routledge Applied Linguistics). London and New York: Routledge. 330 pp. ISBN 10: 0415385520 (hardback)

Kauper, N. — Wed, 24 Jun 2009 07:35:45 PDT

SAGE/ILTA Award for Best Book on Language Testing

Wed, 24 Jun 2009 07:35:45 PDT

The effects of group members' personalities on a test taker's L2 group oral discussion test scores

Ockey, G. J. — Thu, 26 Mar 2009 08:48:23 PDT

The second language group oral is a test of second language speaking proficiency, in which a group of three or more English language learners discuss an assigned topic without interaction with interlocutors. Concerns expressed about the extent to which test takers' personal characteristics affect the scores of others in the group have limited its attractiveness. This study investigates the degree to which assertive and non-assertive test takers' scores are affected by the levels of assertiveness of their group members. The sample of test takers was Japanese first year university students who were studying English in Japan. The students took the revised NEO-PI-R (Costa & McCrae, 1992; Shimanoka et al., 2002), a group oral test, and PhonePass SET-10 (Ordinate, 2004). Two separate MANCOVA analyses were conducted, one designed to determine the extent to which assertive test takers' scores are affected by the levels of assertiveness of group members (N = 112), and one designed to determine the extent to which non-assertive test takers' scores are affected by the levels of assertiveness of group members (N = 113). The analyses indicated that assertive test takers were assigned higher scores than expected when grouped with only non-assertive test takers and lower scores than expected when grouped with only assertive test takers, while the study failed to find an effect for grouping based on assertiveness for non-assertive test takers' scores. The findings of the study suggest that when the group oral is used, rater-training sessions should include guidance on how to evaluate a test taker in the context of the group in which the test taker is assessed and assign scores that are not based on a comparison of proficiencies of group members.

An investigation into native and non-native teachers' judgments of oral English performance: A mixed methods approach

Kim, Y.-H. — Thu, 26 Mar 2009 08:48:23 PDT

This study used a mixed methods research approach to examine how native English-speaking (NS) and non-native English-speaking (NNS) teachers assess students' oral English performance. The evaluation behaviors of two groups of teachers (12 Canadian NS teachers and 12 Korean NNS teachers) were compared with regard to internal consistency, severity, and evaluation criteria. Results of a Many-faceted Rasch Measurement analysis showed that most of the NS and NNS teachers maintained acceptable levels of internal consistency, with only one or two inconsistent raters in each group. The two groups of teachers also exhibited similar severity patterns across different tasks. However, substantial dissimilarities emerged in the evaluation criteria teachers used to assess students' performance. A qualitative analysis demonstrated that the judgments of the NS teachers were more detailed and elaborate than those of the NNS teachers in the areas of pronunciation, specific grammar use, and the accuracy of transferred information. These findings are used as the basis for a discussion of NS versus NNS teachers as language assessors on the one hand and the usefulness of mixed methods inquiries on the other.

A meta-analysis of test format effects on reading and listening test performance: Focus on multiple-choice and open-ended formats

In'nami, Y., Koizumi, R. — Thu, 26 Mar 2009 08:48:23 PDT

A meta-analysis was conducted on the effects of multiple-choice and open-ended formats on L1 reading, L2 reading, and L2 listening test performance. Fifty-six data sources located in an extensive search of the literature were the basis for the estimates of the mean effect sizes of test format effects. The results using the mixed effects model of meta-analysis indicate that multiple-choice formats are easier than open-ended formats in L1 reading and L2 listening, with the degree of format effect ranging from small to large in L1 reading and medium to large in L2 listening. Overall, format effects in L2 reading are not found, although multiple-choice formats are found to be easier than open-ended formats when any one of the following four conditions is met: the studies involve between-subjects designs, random assignment, stem-equivalent items, or learners with a high L2 proficiency level. Format effects favoring multiple-choice formats across the three domains are consistently observed when studies employ between-subjects designs, random assignment, or stem-equivalent items.

The development and validation of a Korean C-Test using Rasch Analysis

Lee-Ellis, S. — Thu, 26 Mar 2009 08:48:23 PDT

Despite the importance of having a reliable and valid measure of Second Language (L2) proficiency, L2 researchers of less commonly taught languages rarely have such a tool. Existing proficiency measures (e.g., DLPT, OPI) are often costly, labor-intensive, time-consuming, or unavailable to the public. With the intent to provide a practical and reliable measure of Korean L2 proficiency, this study attempted to develop and validate a 30-minute C-Test (Klein-Braley, 1981). This Korean C-Test was developed with the specifics of Korean language structure in mind, and Interagency Language Roundtable (ILR) skill-level descriptions were utilized in passage selection in order to test a wide range of participant proficiency levels. The resulting test and a self-assessment questionnaire (Kondo-Brown, 2005) were administered to 37 learners of Korean. Rasch analysis (Bond & Fox, 2007) was used to examine the reliability and concurrent validity of the measure, and Rasch measurement statistics such as separation reliability, difficulty measures, and model fit statistics were used to suggest further improvement to the C-Test. The developed test demonstrated excellent reliability and validity indices, and the results reveal the potential of this C-Test as a quick proficiency indicator.

Diagnostic assessment of writing: A comparison of two rating scales

Knoch, U. — Thu, 26 Mar 2009 08:48:23 PDT

Alderson (2005) suggests that diagnostic tests should identify strengths and weaknesses in learners' use of language and focus on specific elements rather than global abilities. However, rating scales used in performance assessment have been repeatedly criticized for being imprecise and therefore often resulting in holistic marking by raters (Weigle, 2002). The aim of this study is to compare two rating scales for writing in an EAP context; one `a priori' developed scale with less specific descriptors of the kind commonly used in proficiency tests and one empirically developed scale with detailed level descriptors. The validation process involved 10 trained raters applying both sets of descriptors to the rating of 100 writing scripts yielded from a large-scale diagnostic assessment administered to both native and non-native speakers of English at a large university. A quantitative comparison of rater behaviour was undertaken using FACETS. Questionnaires and interviews were administered to elicit the raters' perceptions of the efficacy of the two types of scales. The results indicate that rater reliability was substantially higher and that raters were able to better distinguish between different aspects of writing when the more detailed descriptors were used. Rater feedback also showed a preference for the more detailed scale. The findings are discussed in terms of their implications for rater training and rating scale development.

Test review: BEST Plus spoken language test

Van Moere, A. — Thu, 26 Mar 2009 08:48:23 PDT

Test review: The Versant SpanishTM Test

Fox, J., Fraser, W. — Thu, 26 Mar 2009 08:48:23 PDT

Factor structure of the TOEFL Internet-based test

Sawaki, Y., Stricker, L. J., Oranje, A. H. — Thu, 11 Dec 2008 02:46:25 PST

This construct validation study investigated the factor structure of the Test of English as a Foreign Language^TM Internet-based test (TOEFL® iBT). An item-level confirmatory factor analysis was conducted for a test form completed by participants in a field study. A higher-order factor model was identified, with a higher-order general factor (ESL/EFL ability) and four first-order factors for reading, listening, speaking and writing. Integrated Speaking and Writing tasks, which require language processing in multiple modalities, defined the target modalities (speaking and writing). These results broadly supported the current practice of reporting a total score and four scores corresponding to the modalities for the test, as well as the test design that permits the integrated tasks to contribute only to the scores of the target modalities.

Cognitive diagnostic assessment of L2 reading comprehension ability: Validity arguments for Fusion Model application to LanguEdge assessment

Jang, E. E. — Thu, 11 Dec 2008 02:46:25 PST

With recent statistical advances in cognitive diagnostic assessment (CDA), the CDA approach has been increasingly applied to non-diagnostic tests partly to meet accountability demands for student achievement. The study aimed to evaluate critically the validity of the CDA application to an existing non-diagnostic L2 reading comprehension test and to provide information about challenges and conditions for the CDA approach. Based on Jang's study (2005), this paper focuses on the dependability of the Fusion Model's skill profiling, the characteristics of resulting L2 skill profiles, and the diagnostic capacity of LanguEdge^TM test items. In addition, the paper examines the validity arguments from the users' perspective by focusing on the usefulness of the diagnostic feedback. The results suggest that the CDA approach can provide more fine-grained diagnostic information about the level of competency in reading skills than traditional aggregated-test scoring can. While various empirical evidence supported the dependability of the skill profiling process, the results also raised some concerns about the application of the CDA approach to a test developed for non-diagnostic purposes, most significantly, a lack of diagnostic capacity of some of the test items with extremely easy or difficult levels. The results offer useful information about the potential challenges and conditions for future application of cognitive diagnostic assessment.

Self-, peer-, and teacher-assessments in Japanese university EFL writing classrooms

Matsuno, S. — Thu, 11 Dec 2008 02:46:25 PST

Multifaceted Rasch measurement was used in the present study with 91 student and 4 teacher raters to investigate how self- and peer-assessments work in comparison with teacher assessments in actual university writing classes.

The results indicated that many self-raters assessed their own writing lower than predicted. This was particularly true for high-achieving students. Peer-raters were the most lenient raters; however, they rated high-achieving writers lower and low-achieving writers higher. This tendency was independent of their own writing abilities and therefore offered no support for the hypothesis that high-achieving writers rated severely and low-achieving writers rated leniently. On the other hand, most peer-raters were internally consistent and produced fewer bias interactions than self- and teacher-raters. Each of the four teachers was internally consistent; however, each displayed a unique bias pattern. Self-, peer-, and teacher-raters assessed Grammar severely and Spelling leniently. The analysis also revealed that teacher-raters assessed Spelling, Format, and Punctuation differently from the other criteria.

It was concluded that self-assessment was somewhat idiosyncratic and therefore of limited utility as a part of formal assessment. Peer-assessors on the other hand were shown to be internally consistent and their rating patterns were not dependent on their own writing performance. They also produced relatively few bias interactions. These results suggest that in at least some contexts, peer-assessments can play a useful role in writing classes. By using multifaceted Rasch measurement, teachers can inform peer-raters of their bias patterns and help them develop better quality assessment criteria, two steps that might lead to better quality peer-assessment.

Lexical patterns in L2 textual gist identification assessment

Yamada, K. — Thu, 11 Dec 2008 02:46:25 PST

Gist identification or coherent situation model construction performance is an important criterion not only in L1 and L2 reading comprehension assessment but also in every aspect of discourse processing. Whereas most previous L2 research has investigated schematic knowledge of readers about relations within a text, more recent studies have used models of comprehension that also pay attention to external reference and world knowledge. The present study proposes a way to assess gist identification by L2 readers using Hoey's (1991) theory of networks of repetition of lexical items including morphologically similar words and paraphrases of keywords containing the gist of a text. Based on Hoey's tax-onomy, two models were tested. One, following Kintsch (e.g. 1998), assessed the ability to identify the gist of a text using minimal lookbacks. The other encouraged multiple lookbacks. The second model, which involved more reader involvement in finding referential relationships within a text, resulted in more detailed understanding of gist.

Test architecture, test retrofit

Fulcher, G., Davidson, F. — Thu, 11 Dec 2008 02:46:25 PST

Just like buildings, tests are designed and built for specific purposes, people, and uses. However, both buildings and tests grow and change over time as the needs of their users change. Sometimes, they are also both used for purposes other than those intended in the original designs. This paper explores architecture as a metaphor for language test development. Firstly, it describes test purpose and use, and how this affects test design. Secondly, it describes and illustrates the layers of test architecture and design. Thirdly, it discusses the concept of test retrofit, which is the process of altering the test after it has been put into operational use. We argue that there are two types of test retrofit: an upgrade and a change. Each type of retrofit implies changes to layers of the test architecture which must be articulated for a validity argument to be constructed and evaluated. As is true in architecture, we argue that a failure to be explicit about retrofit seriously limits validity claims and clouds issues surrounding the intended effect of the test upon users.

Book review: Penny McKay (2006) Assessing young language learners. Cambridge: Cambridge University Press. 388 pp. (main text). ISBN 13 978--0--521--84138--2 (hardback). ISBN 13 978--0--521--60123--8 (paperback)

Leung, C. — Thu, 11 Dec 2008 02:46:25 PST

Book review: Chapelle, C. A., and Douglas, D. (2006). Assessing language through computer technology. Cambridge: Cambridge University Press. 138 pp. ISBN 0 521 54949 3 (paperback)

Oskoz, A., Chalhoub-Deville, M. — Thu, 11 Dec 2008 02:46:25 PST