This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Jennings, M. Articles by Shohamy, E. Search for Related Content Social Bookmarking                         What's this?

The test-takers' choice: an investigation of the effect of topic on language-test performance

Carleton University, Ontario, marthaj{at}cyberus.ca

Janna Fox

Carleton University, Ontario

Barbara Graves

Carleton University, Ontario

Elana Shohamy

Tel Aviv University, Israel

A fundamental issue in validating topic-based tests of language proficiency is the effect of the topic on the test takers’ performance. Topic-based test developers must ensure that test takers are neither advantaged nor disadvantaged in terms of their test results when presented with a given test topic. We have termed this threat a ‘topic effect’ and argue that this topic effect may constitute a source of construct-irrelevant variance (Messick, 1989). We contend that investigating the possibility of a topic effect is a critical step in establishing the validity of all topic-based tests. This research investigates the potential presence of a topic effect for the Canadian Academic English Language (CAEL) Assessment using the mechanism of choice. The principal research question is to determine if test-takers given a choice of topic perform significantly differently than test-takers not given a choice. ESL university applicants (n = 254) were randomly assigned to one of two conditions: no choice of topic or choice among five topics. Overall Proficiency Level, Reading, Lecture and Essay scores were compared for the two conditions. Ordinal level data were analysed using the Mann Whitney U, Chi-Square and Kolmogorov-Smirnov tests. While the scores for the choice groups were usually higher than the scores for the no-choice groups, the differences were not statistically significant. Despite the lack of significance, we felt that the scores warranted closer examination. For the topic where the difference between the choice and no-choice groups was largest, a textual analysis of the essays was undertaken to look for instances of the use of information not provided in the test. Again, no difference was found between the choice and no-choice groups. The results provide support for the validity of inferences drawn from this test. Because choice is an essential element of the research design, a second focus of the study is to explore the advantages and disadvantages of the use of choice in language testing settings from the perspective of both the tester and the test-taker. The potential value of choice as a testing feature is discussed and a call for further research is made.

Language Testing, Vol. 16, No. 4, 426-456 (1999)
DOI: 10.1177/026553229901600402


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This article has been cited by other articles:

				L. Plakans Discourse synthesis in integrated second language writing assessment Language Testing, October 1, 2009; 26(4): 561 - 587. [Abstract] [PDF]

				G. Yu Lexical Diversity in Writing and Speaking Task Performances Applied Linguistics, June 4, 2009; (2009) amp024v1. [Abstract] [Full Text] [PDF]

				Liying Cheng, D. A. Klinger, and Ying Zheng The challenges of the Ontario Secondary School Literacy Test for second language students Language Testing, April 1, 2007; 24(2): 185 - 208. [Abstract] [PDF]