Advanced Search

Journal Navigation

Journal Home

Subscriptions

Archive

Contact Us

Table of Contents

Click here to sign up for SAGE Journal Email Alerts today!

Sign In to gain access to subscriptions and/or personal tools.
Language Testing
This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Citing Articles
Right arrow Citing Articles via Google Scholar
Right arrow Citing Articles via Scopus
Google Scholar
Right arrow Articles by Abbott, M. L.
Right arrow Search for Related Content
Social Bookmarking
Add to CiteULike   Add to Complore   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati   Add to Twitter  
What's this?

A confirmatory approach to differential item functioning on an ESL reading assessment

Marilyn L. Abbott

Alberta Education, Edmonton, Alberta, Canada, marilyn.abbott{at}gov.ab.ca

In this article, I describe a practical application of the Roussos and Stout (1996) multidimensional analysis framework for interpreting group performance differences on an ESL reading proficiency test. Although a variety of statistical methods have been developed for flagging test items that function differentially for equal ability examinees from different ethnic, linguistic, or gender groups, the standard differential item functioning (DIF) detection and review procedures have not been very useful in explaining why DIF occurs in the flagged items (Standards for Educational and Psychological Testing 1999). To address this problem, Douglas, Roussos and Stout (1996) developed a confirmatory approach to DIF, which is used to test DIF hypotheses that are generated from theory and substantive item analyses. In the study described in this paper, DIF and differential bundle functioning (DBF) analyses were conducted to determine whether groups of reading test items, classified according to a bottom-up, top-down reading strategy framework, functioned differentially for equal ability Arabic and Mandarin ESL learners. SIBTEST (Stout and Roussos, 1999) analyses revealed significant systematic group differences in two of the bottom-up and two of the top-down reading strategy categories. These results demonstrate the utility of employing a theoretical framework for interpreting group differences on a reading test.

References

  • Ackerman, T., Gierl, M. and Walker, C. 2003: Using multidimensional item response theory to evaluate educational and psychological tests . Educational Measurement: Issues and Practice 22, 37-53 .[CrossRef]
  • Ackerman, T., Simpson, M. and de la Torre, J. 2000: A comparison of the dimensionality of TOEFL response data from different first language groups . Paper presented at the Annual Meeting of the National Council on Measurement in Education, New Orleans, LA.
  • Alderson, C. 1984: Reading in a foreign language: A reading problem or a language problem? In Alderson J. and Urquhart A., editors, Reading in a foreign language. London: Longman , 1-24.
  • Anderson, N. 1991: Individual differences in strategy use in second language reading and testing . Modern Language Journal 75, 460-472 .[CrossRef]
  • Angoff, W. and Ford, S. 1973: Item-race interaction on a test of scholastic aptitude . Journal of Educational Measurement 10, 95-106 .
  • Block, E. 1986: The comprehension strategies of second language readers . TESOL Quarterly 20, 463-494 .[CrossRef]
  • Bolt, D. 2002: Studying the potential of nuisance dimensions using bundle DIF and multidimensional IRT analyses . Paper presented the Annual Meeting of the National Council on Measurement in Education, New Orleans, LA.
  • Bolt, D. and Stout, W. 1996: Differential Item Functioning: Its multidimensional model and resulting SIBTEST detection procedure . Behaviormetrika 23, 67-95 .
  • Brown, J. 1999: The relative importance of persons, items, subtests and languages to TOEFL test variance . Language Testing 16, 217-238 .[Abstract/Free Full Text]
  • Cameron, J. and Derwing, T. 2004: Being Canadian. Second Edition. Saint-laurent, PQ: Longman .
  • Camilli, G. and Shepard, L. 1994: Methods for identifying biased test items. Newbury Park, CA: Sage .
  • Carrell, P. 1989: Metacognitive awareness and second language reading . Modern Language Journal 73, 121-133 .[CrossRef]
  • Chen, Z. and Henning, G. 1985: Linguistic and cultural bias in language proficiency tests . Language Testing 2, 155-163 .[Abstract/Free Full Text]
  • Douglas, J., Roussos, L. and Stout, W. 1996: Item-bundle DIF hypothesis testing: Identifying suspect bundles and assessing their differential functioning . Journal of Educational Measurement 33, 465-484 .[CrossRef]
  • Ercikan, K., Gierl, M., McCreith, T., Puhan, G. and Koh, K. 2002: Comparability of English and French Versions of SAIP for reading, mathematics and science items . Paper presented at the Annual Meeting of the Canadian Society for the Study of Education, Toronto.
  • Fender, M. 2003: English word recognition and word integration skills of native Arabic- and Japanese-speaking learners of English as a second language . Applied Psycholinguistics 24, 289-315 .[CrossRef]
  • Gierl, M., Bisanz, G. and Bisanz, J. 2001: Developing an interpretative framework for understanding group differences on national and international achievement tests: The case of excellence in Alberta. A research proposal submitted to Alberta Learning, Edmonton, Alberta .
  • Gierl, M., Bisanz, J., Bisanz, G. and Boughton, K. 2003: Identifying content and cognitive skills that produce gender differences in mathematics: a demonstration of the multidimensionality based DIF analysis framework . Journal of Educational Measurement 40, 281-306 .[CrossRef]
  • Gierl, M., Bisanz, J., Bisanz, G., Boughton, K. and Khaliq, S. 2001: Illustrating the procedures and content reviews for identifying translation DIF . Paper presented at the Annual Meeting of the National Council on Measurement in Education, Montreal, Canada.
  • Gierl, M., Rogers, T. and Klinger, D. 1999: Consistency between statistical procedures and content reviews for identifying translation DIF . Paper presented at the Annual Meeting of the National Council on Measurement in Education, Montreal, Canada.
  • Ginther, A. and Stevens, J. 1998: Language background and ethnicity, and the internal construct validity of the Advanced Placement Spanish Language Examination. In Kunnan, A., editor, Validation in language assessment. Mahwah, NJ: Erlbaum 169-194.
  • Hill, C. and Parry, K. 1992: The test at the gate: Models of literacy in reading assessment . TESOL Quarterly 26, 433-461 .
  • Holland P. and Thayer, D. 1986: Differential item performance and the Mantel-Haenszel procedure. ETS Research Report No. 86-31. Princeton, NJ: Educational Testing Service .
  • Jiang, H. and Stout, W. 1998: Improved type I error control and reduced estimation bias for DIF detection using SIBTEST . Journal of Educational and Behavioural Statistics 23, 291-322 .
  • Kunnan, A. 1994: Modelling relationships among some test-taker characteristics and performance on EFL tests: An approach to construct validation . Language Testing 11, 225-252 .[Abstract/Free Full Text]
  • Mantel, N. and Haenszel, W. 1959: Statistical aspects of the analysis pf data from retrospective studies of disease . Journal of the National Cancer Institute 22, 719-748 . Holland .[Medline] [Order article via Infotrieve]
  • Nandakumar, R. 1993: Simultaneous DIF amplification and cancellation: Shealy-Stout’s test for DIF . Journal of Educational Measurement 30, 293-311 .[CrossRef]
  • Parry, K. 1996: Culture, literacy and L2 reading . TESOL Quarterly 30, 665-692 .
  • Phakiti, A. 2003: A closer look at the relationship of cognitive and metacognitive strategy use to EFL reading achievement test performance . Language Testing 20, 26-56 .[Abstract/Free Full Text]
  • Pritchard, R. 1990: The effects of cultural schemata on reading processing strategies . Reading Research Quarterly 25, 273-295 .
  • Purpura, J. 1997: An analysis of the relationships between test takers’ cognitive and metacognitive strategy use and second language test performance . Language Learning 47, 289-325 .[CrossRef]
  • Roussos, L. and Stout, W. 1996: A multidimensionality-based DIF analysis paradigm . Applied Psychological Measurement 20, 355-371 .[Abstract]
  • Rumelhart, D. 1977: Toward an interactive model of reading. In S. Dornic (Ed.), Attention and performance. New York: Academic Press .
  • Ryan, K. and Bachman, L. 1992: Differential item functioning on two tests of EFL proficiency . Language Testing 9, 12-29 .[Medline] [Order article via Infotrieve]
  • Sasaki, M. 1991: A comparison of two methods for detecting differential item functioning in an ESL placement test . Language Testing 8, 95-111 .[Abstract/Free Full Text]
  • Scheuneman, J. 1979: A new method for assessing bias in test items . Journal of Educational Measurement 16, 143-152 .[CrossRef]
  • Schueller, J. 2000: The effects of two types of strategic training on foreign language reading comprehension. An analysis by gender and proficiency. Dissertation Abstracts International 60 (07), 2472. (UMI No. 9923247)
  • Schueller, J. 2004: Gender and foreign language reading comprehension: The effects of strategy training . Southern Journal of Linguistics 27, 45-45 and 65-65 .
  • Shealy, R. and Stout, W. 1993: A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DIF as well as item bias/DIF . Psychometrika 58, 159-194 .[CrossRef]
  • Shepard, L., Camilli, G. and Averill, M. 1981: Comparison of six procedures for detecting test item bias using both internal and external ability criteria . Journal of Educational Statistics 6, 317-375 .[CrossRef]
  • Standards for Educational and Psychological Testing. 1999: Washington, DC: American Educational Research Association, American Psychological Association, & National Council on Measurement in Education.
  • Stanovich, K. 1980: Toward an interactive-compensatory model of individual differences in the development of reading fluency . Reading Research Quarterly 16, 32-71 .[CrossRef]
  • Stanovich, K. 2000: Progress in understanding reading: Scientific foundations and new frontiers. New York: Guilford Press .
  • Stout, W. and Roussos, L. 1995: SIBTEST manual. Champaign-Urbana, IL: Department of Statistics, Statistical Laboratory for Educational and Psychological Measurement, University of Illinois .
  • Stout, W. and Roussos, L. 1999: Dimensionality-based DIF/DBF package [Computer program]. Champaign-Urbana, IL: William Stout Institute for Measurement, University of Illinois .
  • Young, D. and Oxford, R. 1997: A gender-related analysis of strategies used to process input in the native language and a foreign language . Applied Language Learning 8, 43-73 .

Language Testing, Vol. 24, No. 1, 7-36 (2007)
DOI: 10.1177/0265532207071510
Add to CiteULike CiteULike   Add to Complore Complore   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to Technorati Technorati   Add to Twitter Twitter    What's this?



This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Citing Articles
Right arrow Citing Articles via Google Scholar
Right arrow Citing Articles via Scopus
Google Scholar
Right arrow Articles by Abbott, M. L.
Right arrow Search for Related Content
Social Bookmarking
Add to CiteULike   Add to Complore   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati   Add to Twitter  
What's this?