Language Testing recent issues

Do divisible subskills exist in second language (L2) comprehension? A structural equation modeling approach

Song, M.-Y. — 2008-09-23

This paper concerns the divisibility of comprehension subskills measured in L2 listening and reading tests. Motivated by the administration of the new Web-based English as a Second Language Placement Exam (WB-ESLPE) at UCLA, this study addresses the following research questions: first, to what extent do the WB-ESLPE listening and reading items measure different comprehension subskills? Second, to what extent can L2 listening and reading be considered similar or different, with regard to the divisibility of comprehension subskills? Employing a structural equation modeling (SEM) approach, the factor structures of 110 test takers' scores on the WB-ESLPE listening and reading items were examined with regard to three types of subskills, that is, `understanding main/topical ideas', `understanding supporting/specific details', and `making inferences'. The results of the analyses indicated that the WB-ESLPE listening and reading items measure two or three kinds of subskills, and that while L2 listening and reading might share a common comprehension process, they may be distinct in the decoding processes involved due to the difference in mode of presentation. It is suggested that this divisibility of subskills in L2 comprehension tests might depend on the test takers' L2 proficiency as well as on the task characteristics of the test administered to them.

Rater bias patterns in an EFL writing assessment

Schaefer, E. — 2008-09-23

The present study employed multi-faceted Rasch measurement (MFRM) to explore the rater bias patterns of native English-speaker (NES) raters when they rate EFL essays. Forty NES raters rated 40 essays written by female Japanese university students on a single topic adapted from the TOEFL Test of Written English (TWE). The essays were assessed using a six-category rating scale (Content, Organization, Style and Quality of Expression, Language Use, Mechanics, and Fluency). MFRM revealed several recurring bias patterns among rater subgroups. In rater—category bias interactions, if Content and/or Organization were rated severely, then Language Use and/or Mechanics were rated leniently, and vice versa. In rater—writer bias interactions, there tended to be more severe or lenient bias towards higher ability writers than lower ability writers. Some raters also rated higher ability writers more severely and lower ability writers more leniently than expected. This study has implications for issues of rater training in L2 writing assessment.

Development of a cognate awareness measure for Spanish-speaking English language learners

Malabonga, V., Kenyon, D. M., Carlo, M., August, D., Louguit, M. — 2008-09-23

This paper describes the development and validation of the Cognate Awareness Test (CAT), which measures cognate awareness in Spanish-speaking English Language Learners (ELLs) in fourth and fifth grade. An investigation of differential performance on the two subtests of the CAT (cognates and noncognates) provides evidence that the instrument is sensitive to English—Spanish cognate awareness among elementary school-age Spanish-speaking ELLs. Cognates were highly correlated with the children's Spanish WLPB-R Picture Vocabulary scores, whereas noncognates were highly correlated to children's English WLPB-R Picture Vocabulary scores.

Reading to summarize in English and Chinese: A tale of two languages?

Guoxing Yu, — 2008-09-23

The cognitive demands of summary writing are dependent upon the type of summary to be produced. This paper reports part of a larger study in which 157 Chinese undergraduates were asked to write summaries of extended English texts in both English and Chinese. It examines the differential effects of the use of the two languages on summarization as a measure of reading comprehension, drawing upon data from students' actual test performances as well as their perceptions of such differential effects, as evidenced in the post-summarization questionnaire and interviews. It was found that the use of the different languages had significant effects on both summarization processes and products. Students wrote significantly longer Chinese summaries but these were rated consistently of poorer quality than English summaries. However, Chinese summarization was found to be a better measure of students' English reading abilities. The implications of these findings are discussed with specific reference to summarization task design, in particular, how the use of the first and target languages may affect test takers' performance as well as rater behaviours and the potential of using first language in large-scale international language testing programmes incorporating integrated reading/writing tasks as a measure of reading comprehension.

EFL classroom peer assessment: Training effects on rating and commenting

Saito, H. — 2008-09-23

This study examined the effects of training on peer assessment and comments provided regarding oral presentations in EFL (English as a Foreign Language) classrooms. In Study 1, both the treatment and control groups received instruction on skill aspects, but only the treatment group was given an additional 40-minute training on how to rate performances. The results of the correlation difference analyses show no significant differences between the treatment and control groups, but the three instructors are misfitting, presumably because the frame of reference is set by the majority of student data. In the second study, the treatment groups received long training. Again, there are no significant correlation differences between the treatment and control groups; however, all the instructors are not misfitting, which indicates that the frame of reference in the data improved in Study 2. Analyses of comments reveal that the treatment groups are superior in both quality and quantity of comments. Along with a meta-analytic summary, it is argued that peer assessment is a robust system in which instruction on skill aspects may suffice to achieve a certain level of correlation with the criterion variable (instructor), but training may enhance student comments and reduce misfitting raters.

Editorial

2008-07-24

SAGE/ILTA Award for Best Book on Language Testing

2008-07-28

Introduction: Language Testing at 25: Maturity and responsibility?

Spolsky, B. — 2008-07-24

Tribute UCLES/ILTA 2007 Lifetime Achievement Award to Charles W. Stansfield

Davies, A. — 2008-07-24

Lecture: `Where we have been and where we should go'

Stansfield, C. W. — 2008-07-24

Textbook trends in teaching language testing

Davies, A. — 2008-07-24

The article examines changes in language testing textbooks in English since Lado (1961) and proposes that two trends may be discerned. The first shows how the growing professionalism of the field has required an expansion in teaching materials to meet the need for new training programmes. What the expansion also shows is the desire, again a mark of increasing professionalism, to provide all teaching resources from within the profession so that for needed skills (e.g. statistics and measurement) it is now less necessary to appeal to outsiders such as statisticians and psychometricians. The second trend explains the need for the profession to expand its view of the skills needed by its members. From Lado onwards, skills were always conjoined with knowledge about language and about testing. More recently, the profession has explicitly declared a concern for principles with regard, for example, to validity and to ethics. The increasing professionalism comes at a cost: that cost is twofold: in-housing all resources means that language testers are increasingly insulated from other potentially rewarding disciplines. And the complete resource offerings in the later teaching materials means that students may be denied empirical encounters with real language learners, spending all (or much of) their training within the resource material. The article also questions how far research has informed the changes in training materials.

Language testing courses: What are they in 2007?

Brown, J. D., Bailey, K. M. — 2008-07-24

In 1996, we published an article that investigated the characteristics of basic language testing courses in terms of the instructors, course characteristics, and students. The present study is designed to describe the same characteristics of basic language testing courses in 2007 and to examine how such courses have changed since 1996. To those ends, we used the same questionnaire that was used in 1996 with 20 new items added. The main topics of the questionnaire included the instructors' background, the topics covered in class, and the instructors' perceptions of students' general attitudes toward language testing at the beginning and end of the course. While most items were of the Likert-scale type, some were open-ended. In the first three months of 2007, email messages were sent out to the ltest-l and ilta-mem lists asking recipients to access a Survey Monkey website that contained our questionnaire and respond to it. A second emailing was sent out several weeks later to involve those who had not responded the first time. A total of 97 participants cooperated with usable data. The results describe the instructors, course characteristics, and students, as well as differences and similarities between the 1996 and 2007 results in quantitative and qualitative terms.

Constructing a language assessment knowledge base: A focus on language assessment courses

Inbar-Lourie, O. — 2008-07-24

The competencies required for conducting assessment in the educational context have recently been reformulated in view of social constructivist perspectives and the acknowledgement of the role of classroom assessment in promoting learning. These changes have impacted the knowledge base language assessors need to obtain, and hence the contents of language assessment courses. This paper considers the possible components of a language assessment literacy knowledge base, and proposes the establishment of a core knowledge framework for courses in language assessment.

Test review: GEPT: General English Proficiency Test

Roever, C., Pan, Y.-C. — 2008-07-24

Test review: College English Test (CET) in China

Ying Zheng, , Liying Cheng, — 2008-07-24

Book review: Steven M. Downing and Thomas M. Haladyna (Eds.) (2006). Handbook of test development. Mahwah, NJ: Erlbaum. 778 pp. $99.95 (paperback). ISBN 0--8058--5264--6

ElAtia, S. — 2008-07-24

Rater types in writing performance assessments: A classification approach to rater variability

Eckes, T. — 2008-03-13

Research on rater effects in language performance assessments has provided ample evidence for a considerable degree of variability among raters. Building on this research, I advance the hypothesis that experienced raters fall into types or classes that are clearly distinguishable from one another with respect to the importance they attach to scoring criteria. To examine the rater type hypothesis, I asked 64 raters actively involved in scoring examinee writing performance on a large-scale assessment instrument to indicate on a four-point scale how much importance they would attach to each of nine routinely used criteria. The criteria covered various performance aspects, such as fluency, completeness, and grammatical correctness. In a preliminary step, many-facet Rasch analysis revealed that raters differed significantly in their views on the importance of the various criteria. A two-mode clustering technique yielded a joint classification of raters and criteria, with six rater types emerging from the analysis. Each of these types was characterized by a distinct scoring profile, indicating that raters were far from dividing their attention evenly among the set of criteria. Moreover, rater background variables were shown to partially account for the scoring profile differences. The findings have implications for assessing the quality of large-scale rater-mediated language testing, rater monitoring, and rater training.

Post-hoc IRT equating of previously administered English tests for comparison of test scores

Saida, C., Hattori, T. — 2008-03-13

Despite growing concerns about declining scholastic abilities of Japanese students throughout Japan prior to the implementation of the revised Courses of Study in 2002, little empirical evidence was available at that time to support this perceived decline in academic performance. This research describes post-hoc IRT equating of previously administered English tests for retrospective comparison of test scores. Eight forms of a prefecture-(state-)wide English test for high school students administered from 1995 to 2002 were analyzed through the two-parameter IRT model. Because there were no common items and no common examinees among the eight forms, seven composite tests were created for equating with a common-examinee design. Using the mean and sigma method to estimate equating constants, the ability parameter estimates of approximately 140,000 examinees became comparable on a common IRT scale. The results showed that the mean scores among first-year high school students were in a declining trend during the eight-year period. This is a first step toward demonstrating trends in English ability among Japanese students year by year based on IRT using previously administered large-scale English tests. The post-hoc equating method presented here is applicable to any similar circumstance in language testing research.

The assessment of deep word knowledge in young first and second language learners

Schoonen, R., Verhallen, M. — 2008-03-13

The assessment of so-called depth of word knowledge has been the focus of research for some years now. In this article the construct of deep word knowledge is further specified as the decontextualized knowledge of word meanings and word associations. Most studies so far have involved adolescent and adult second language learners. In this article, we extend the focus to younger first and second language learning children in primary school in the Netherlands. A well-established test format involving word associations has been adapted for this younger target group and the feasibility of assessing deep word knowledge in primary school children is evaluated empirically. The results showed that the test provides a reliable and efficient method of assessing the children's vocabulary knowledge. Evidence of validity was obtained from a concurrent definition task, and from an analysis of the scores in terms of language background, which is known to be associated with different levels of vocabulary knowledge.

Construct validation of Bachman and Palmer's (1996) strategic competence model over time in EFL reading tests

Phakiti, A. — 2008-03-13

This article reports on a large-scale study that aims to validate the theory of strategic competence proposed by Bachman and Palmer (1996) through the use of structural equation modeling (SEM). The present study examines the relationship of test-takers' long-term strategic knowledge (i.e., trait strategies) and actual strategy use (i.e., state strategies) to second language (L2) reading test performance over time. The data were gathered on two occasions (during the mid-term and final examination periods). Five hundred and sixty-one Thai university test-takers answered a trait strategy use questionnaire prior to the mid-term and final reading achievement tests and, immediately after completing each test, they answered a state strategy use questionnaire. It was found that (1) trait metacognitive strategy use (MSU) directly and strongly affects trait cognitive strategy use (CSU); (2) trait CSU does not greatly affect state CSU; (3) trait MSU directly affects state MSU in a specific context, which in turn directly affects state CSU; and (4) state CSU directly affects a specific language test performance to a varying degree.

Book review: Brown, H. Douglas (2004). Language assessment: Principles and classroom practices. White Plains, NY: Pearson Education. 324 pp. $48.00 paper. ISBN 0--13--098834--0; Brown, James Dean (2005). Testing in language programs: A comprehensive guide to English language assessment. New York: McGraw-Hill. 307 pp. $32.85 paper. ISBN 0--07--294836--1

Lund, J., Winke, P. M. — 2008-03-13

Book review: Luoma, S. (2004). Assessing speaking. Cambridge: Cambridge University Press. 212 pp. ISBN: 0--521--80487--6 (paperback)

Lazaraton, A. — 2008-03-13

Language testing in Asia: Evolution, innovation, and policy challenges

Ross, S. J. — 2008-02-15

The key to success: English language testing in China

Liying Cheng, — 2008-02-15

The testing and examination history in China can be traced back to the imperial period nearly two thousand years ago. The existence of English language testing (tests), on the other hand, has a much shorter history. These English tests, developed and administered over the past 20 years, however, are taken by billions of learners of the English language in China. To many of these learners, doing well on these tests are the key to their academic success as well as the success of their life in general. The paper will first introduce major tests and examinations of English designed and administered in China, then provide an overview of the current research in language testing that has been conducted by Chinese researchers and published in Chinese academic journals over the past 10 years. This paper will focus on the discussion of the issues and concerns of language testing within the Chinese context.

The impact of EFL testing on EFL education in Korea

Choi, I.-C. — 2008-02-15

The present study provides an overview of the impact of standardized EFL tests on EFL education in Korea. To achieve this goal, the paper (1) presents the status quo of EFL testing in the Korean context, (2) explores the nature of the EFL tests prevalent in the EFL testing market, and (3) investigates the overwhelming washback effects of EFL tests on EFL teaching based on a survey of stakeholder viewpoints. The overall findings of the survey reveal that the majority of stakeholders (i.e. test-takers and teachers) do not think favorably of the EFL tests due to negative washback effects on their EFL learning and teaching. The survey also shows that considerable numbers of young students are under unwarranted pressure to take the EFL tests and that secondary education puts too much emphasis on preparation for the college entrance exam. Most respondents have negative views of the tests in terms of the mismatch between test scores and English proficiency and the failure of multiple-choice EFL test preparation to induce productive English skills. Some respondents voice complaints about the financial burden caused by mandatory submission of test scores for graduation and employment.

The 150-year history of English language assessment in Japanese education

Sasaki, M. — 2008-02-15

In the present study I describe the 150-year history of school-based English language assessment in Japan. The history is divided into four major periods according to the purposes of English language education set by the government in the different periods: (1) 1860 to 1945, when English was first introduced and taught in schools mainly for elite classes; (2) 1945 to 1970, when English became part of the compulsory education for the first time; (3) 1970 to 1990 when English began to be regarded as the most effective means to communicate with foreign people in the rapidly shrinking world; and (4) 1990 to the present, when several innovative policies have been introduced into classroom measurement systems. I describe how assessment practices for English education at schools in each of these periods were and still are affected by various factors, including political, economic, and demographic changes in society, as well as academic paradigm shifts in the fields of education and applied linguistics.

English language assessment in Hong Kong: A survey of practices, developments and issues

Qian, D. D. — 2008-02-15

English has been an official language of Hong Kong since it was first introduced to the region in the 1840s. Since then, its influence has invariably been strong and remains so even after 1997, when the sovereignty of the region was returned to China. This special background of English in Hong Kong heavily influences the policy and practices of English language assessment in the region. The present article starts with a survey of the status of the English language in Hong Kong and then reviews major English-language tests that are considered to be having a significant impact on Hong Kong society and its education system. Reviewed in the article are tests developed or imported for assessing students in primary and secondary schools and tertiary institutions, as well as tests that assess the English proficiency of English language school-teachers and the general workforce in Hong Kong. In particular, the article provides insights into the recent evolution of Hong Kong's English language assessment at various levels and for different purposes amidst the region's changing socio-political and socio-educational contexts. In anticipating the future of English language assessment, the article considers some pressing issues arising from the implementation of new testing and assessment procedures in the region.

Testing of English in India: A developing concept

Ramanathan, H. — 2008-02-15

English is the associate official language in India and serves as a unifying force in this multilingual country. The teaching of English in K-12 settings focuses on the skills of reading and writing. Listening and speaking skills are not awarded much time, if any, in most classrooms or test settings; only two Boards of Examinations mandate their inclusion. This status is sustained by factors such as large classes, teachers' inadequate command of English, and the lack of easy access to support materials and facilities. Most exams are summative; very few private schools offer and use formative tests for diagnostic and instructional purposes. The Boards of Education to which the schools are affiliated involve teachers in the setting and grading of exams at all levels. Issues that need attention prior to instituting large-scale testing reform are teacher competency in English, grade inflation, and securing conditions for internal assessment in a high-stakes testing environment. A major fall-out of these changes may be the legitimization of Indian English.

Issues and trends in language testing and assessment in Thailand

Prapphal, K. — 2008-02-15

This paper aims to present an overview of language testing in Thailand. Language testing practices in the past are also reviewed. Attention is paid to the washback effects of language tests, the use of language tests in school admissions, questions of test validity, the emergence of standardized tests, the influence of societal values on testing practices, the use of computers and technology in language testing, and the success or failure of policy implementation. Also reviewed are the following new trends of language testing and assessment in Thailand: human resources development and professional certification, the integration of IT and English teaching and testing, quality assurance and English assessment, learner autonomy and self-assessment, and advanced technology and innovations in language assessment.

Book review: Liying Cheng, 2005: Changing language teaching through language testing: A washback study. Cambridge: Cambridge University Press. xiv, 342 pp. ISBN: 9780521836142 (hbk), 9780521544733 (pbk), {pound}55.00 (hbk), {pound}21.50 (pbk)

Guoxing Yu, — 2008-02-15