This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Bachman, L. F. Articles by Milanovic, M. Search for Related Content Social Bookmarking                         What's this?

The use of test method characteristics in the content analysis and design of EFL proficiency tests

University of California, Los Angeles

Fred Davidson

University of Illinois at Urbana-Champaign

Michael Milanovic

University of Cambridge Local Examinations Syndicate

Content considerations are widely viewed to be essential in the design of language tests, and evidence of content relevance and coverage provides an important component in the validation of score interpretations. Content analysis can be viewed as the application of a model of test design to a particular measurement instrument, using judgements of trained analysts. Following Bachman (1990), a content analysis of test method characteristics and components of communicative language ability was performed by five raters on six forms of an EFL test from the University of Cambridge Local Examinations Syndicate. To investigate rater agreement, generalizability analysis and a new agreement statistic (the rater agree ment proportion or 'RAP') were used. Results indicate that the overall level of rater agreement was very high, and that raters were more consistent in rating method than ability. To examine interform comparability, method/ability content analysis characteristics (called 'facets') which differed by more than one standard deviation of either form were deemed to be salient. Results indicated that not all facets yielded substantive information about interform content comparability, although certain test characteristics could be targeted for further revision and development. The relationships between content analysis ratings and two-para meter IRT item parameter estimates (difficulty and discrimination) were also investigated. Neither test method nor ability ratings by themselves yielded consist ent predictions of either item discrimination or difficulty across the six forms examined. Fairly high predictions were consistently obtained, however, when method and ability ratings were combined. The implications of these findings, as well as the utility of content analysis in operational test development, are dis cussed.

Language Testing, Vol. 13, No. 2, 125-150 (1996)
DOI: 10.1177/026553229601300201


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This article has been cited by other articles:

				E. E. Jang Cognitive diagnostic assessment of L2 reading comprehension ability: Validity arguments for Fusion Model application to LanguEdge assessment Language Testing, January 1, 2009; 26(1): 031 - 73. [Abstract] [PDF]