Language Testing current issue

The effects of self-assessment among young learners of English

Goto Butler, Y., Lee, J. — Wed, 27 Jan 2010 07:12:38 PST

This study examined the effectiveness of self-assessment among 254 young learners of English as a foreign language. This study looked at 6th grade students in South Korea, who were asked to perform self-assessments on a regular basis for a semester during their English classes. The students improved their ability to self-assess their performance over time. A series of quantitative analyses found some positive effects of self-assessment on the students’ English performance as well as their confidence in learning English, though the effect sizes were rather small. The study also found that teachers and students perceived the effectiveness of self-assessment differently depending on their teaching/learning contexts. Individual teachers’ views towards assessment also influenced their perceived effectiveness in implementing the new self-assessment practice. A number of interesting insights were discovered through interviews with teachers regarding how best to implement self-assessment as part of foreign language instruction in contexts where teacher-centered teaching and measurement-driven assessment have been traditionally valued.

Washback of an oral assessment system in the EFL classroom

Munoz, A. P., Alvarez, M. E. — Wed, 27 Jan 2010 07:12:38 PST

This article reports the results of a research study to determine the washback effect of an oral assessment system on some areas of the teaching and learning of English as a Foreign Language (EFL). The research combined quantitative and qualitative research methods within a comparative study between an experimental group and a comparison group. Fourteen EFL teachers and 110 college students participated in the study. Data were collected by means of teacher and student surveys, class observations, and external evaluations of students’ oral performance. The data were analyzed using descriptive statistics for qualitative information and inferential statistics to compare the mean scores of the two groups by One Way Anova. Results showed positive washback in some of the areas examined. The implications for the classroom are that constant guidance and support over time are essential in order to help teachers use the system appropriately and therefore create positive washback.

A survey of aviation English tests

Alderson, J. C. — Wed, 27 Jan 2010 07:12:38 PST

The Lancaster Language Testing Research Group was commissioned in 2006 by the European Organisation for the Safety of Air Navigation (Eurocontrol) to conduct a validation study of the development of a test called ELPAC (English Language Proficiency for Aeronautical Communication), intended to assess the language proficiency of air traffic controllers. As part of that study, Internet searches for other tests of air traffic control identified a number of tests but found very little evidence available to attest to the quality of these tests. Therefore, it was decided to conduct an independent survey of tests of aviation English, since the consequences of inadequate language tests being used in licensing pilots, air traffic controllers and other aviation personnel are potentially very serious. A questionnaire was developed, based on the Guidelines for Good Practice of the European Association for Language Testing and Assessment (EALTA, 2006), and sent to numerous organizations whose tests were thought to be used for licensure of pilots and air traffic controllers. Twenty-two responses were received, which varied considerably in quantity and quality. This probably reflects a variation in the quality of the tests, in the availability of evidence to support claims of quality, and in low awareness of appropriate procedures for test development, maintenance and validation.

We conclude that we can have little confidence in the meaningfulness, reliability, and validity of several of the aviation language tests currently available for licensure. We therefore recommend that the quality of language tests used in aviation be monitored to ensure they follow accepted professional standards for language tests and assessment procedures.

Aspects of performance on line graph description tasks: influenced by graph familiarity and different task features

Xiaoming Xi, — Wed, 27 Jan 2010 07:12:38 PST

Motivated by cognitive theories of graph comprehension, this study systematically manipulated characteristics of a line graph description task in a speaking test in ways to mitigate the influence of graph familiarity, a potential source of construct-irrelevant variance. It extends Xi (2005), which found that the differences in holistic scores on graph tasks with varying characteristics, although significant, were small. Using an analytic scoring approach, it re-examined how visual chunks in line graphs and planning time influenced some specific components of examinees’ performance on line graph description tasks. The analytic dimensions examined were determined based on results of previous studies and hypotheses about the relationships among visual chunks, planning, graph familiarity, and features of elicited discourse. It was found that participants less familiar with graphs described the line graphs in a less organized fashion and that their descriptions were weaker in content. Graph familiarity thus introduced construct-irrelevant variance. However, providing planning time and using less complex graphical displays improved the fluency, organization and content of the elicited oral discourse and helped lessen the influence of graph familiarity, thus enhancing the validity of this task. The theoretical and practical implications of the findings are discussed.

A Rasch-based validation of the Vocabulary Size Test

Beglar, D. — Wed, 27 Jan 2010 07:12:38 PST

The primary purpose of this study was to provide preliminary validity evidence for a 140-item form of the Vocabulary Size Test, which is designed to measure written receptive knowledge of the first 14,000 words of English. Nineteen native speakers of English and 178 native speakers of Japanese participated in the study. Analyses based on the Rasch model were focused on several aspects of Messick’s validation framework. The findings indicated that (1) the items and examinees generally performed as predicted by a priori hypotheses, (2) the overwhelming majority of the items displayed good fit to the Rasch model, (3) the items displayed a high degree of unidimensionality with the Rasch model accounting for 85.6% of the variance, (4) the items showed a strong degree of measurement invariance with disattenuated Pearson correlations for person measures estimated with different sets of items of 0.91 and 0.96, and (5) various combinations of items provided precise measurement for this sample of examinees as indicated by Rasch reliability indices >0.96. The Vocabulary Size Test provides teachers and researchers with a new instrument that greatly extends the range of measurement provided by other measures of written receptive vocabulary size.

Assessing the accuracy and consistency of language proficiency classification under competing measurement models

Bo Zhang, — Wed, 27 Jan 2010 07:12:38 PST

This article investigates how measurement models and statistical procedures can be applied to estimate the accuracy of proficiency classification in language testing. The paper starts with a concise introduction of four measurement models: the classical test theory (CTT) model, the dichotomous item response theory (IRT) model, the testlet response theory (TRT) model, and the polytomous item response theory (Poly-IRT) model. Following this, two classification procedures are presented: the Livingston and Lewis method for CTT and the Rudner method for the three IRT-based models. The utility of these models and procedures are then evaluated by examining the accuracy of classifying 5000 language test takers from a large-scale language certification examination into two proficiency categories.

The most important finding is that the testlet format (multiple questions based on one prompt), which language tests usually rely on, has a great impact on the proficiency classification. All testlets in this study show a strong testlet effect. Hence, the TRT model is recommended for proficiency classification. Using the standard IRT model would inflate the classification accuracy due to the underestimated measurement error. Meanwhile, using the Poly-IRT model would give slightly less accurate classification results. Concerning the CTT model, while its classification accuracy is comparable to that of the TRT, there exists considerable inconsistency between their classification results.

Book review: Shaw SD and Weir CJ (2007): Examining writing: research and practice in assessing second language writing. Cambridge: Cambridge University Press. (Volume 26, Studies in Language Testing, edited by M Milanovic and C Weir). 358 pp. ISBN: 978--0521692939. US $51.00

Cushing Weigle, S. — Wed, 27 Jan 2010 07:12:38 PST