Advanced Search

Journal Navigation

Journal Home

Subscriptions

Archive

Contact Us

Table of Contents

Click here to sign up for SAGE Journal Email Alerts today!

Sign In to gain access to subscriptions and/or personal tools.
Language Testing
This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Citing Articles
Right arrow Citing Articles via Google Scholar
Right arrow Citing Articles via Scopus
Google Scholar
Right arrow Articles by Llosa, L.
Right arrow Search for Related Content
Social Bookmarking
Add to CiteULike   Add to Complore   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati   Add to Twitter  
What's this?

Validating a standards-based classroom assessment of English proficiency: A multitrait-multimethod approach

Lorena Llosa

New York University, USA, lorena.llosa{at}nyu.edu

The use of standards-based classroom assessments to test English learners' language proficiency is increasingly prevalent in the United States and many other countries. In a large urban school district in California, for example, a classroom assessment is used to make high-stakes decisions about English learners' progress from one level to the next, and as one of the criteria for reclassifying students as Fluent English Proficient. Yet many researchers have questioned the validity of using classroom assessments for making high-stakes decisions about students (Brindley, 1998; 2001; Rea-Dickins and Gardner, 2000). One way to investigate the validity of the inferences drawn from these assessments is to examine them in relation to other measures of the same ability. In this study, a multivariate analytic approach was used to examine the extent to which the English Language Development (ELD) Classroom Assessment measures the same constructs as the CELDT (California English Language Development Test), the statewide standardized test of English proficiency. Using confirmatory factor analysis of multitrait-multimethod data, this study investigates the construct validity of these measures by focusing on evidence of convergence, discrimination, and method effects longitudinally over three years. The study concludes that the evidence gathered via the ELD Classroom Assessment is consistent with that provided by the CELDT, the standardized measure.

References

  • Arkoudis, S. and O'Loughlin, K. 2004: Tensions between validity and outcomes: Teacher assessment of written work of recently arrived immigrant ESL students. Language Testing 21: 284—304.[Abstract/Free Full Text]
  • Bachman, L.F., Davidson, F., Ryan, K. and Choi, I-C. 1995: An investigation into the comparability of two tests of English as a foreign language: The Cambridge—TOEFL comparability study. Studies in Language Testing, 1. Cambridge: Cambridge University Press.
  • Bachman, L.F. and Palmer, A. 1981: The construct validation of the FSI oral interview. Language Learning 31: 67—86.[CrossRef]
  • ——— 1982: The construct validation of some components of communicative proficiency. TESOL Quarterly 16: 449—65.[CrossRef]
  • Bae, J. and Bachman, L.F. 1998: A latent variable approach to listening and reading: Testing factorial invariance across two groups of children in the Korean/English two-way immersion program. Language Testing 15: 380—414.[Abstract/Free Full Text]
  • Bentler, P.M. 1985—2005: EQS for Windows 6.1. Encino, CA: Multivariate Software.
  • ———— 1990: Comparative fit indexes in structural models. Psychological Bulletin 107: 238—46.[CrossRef][Medline] [Order article via Infotrieve]
  • Bentler, P.M. and Bonnet, D.G. 1980: Significance tests and goodness-of-fit in the analysis of covariance structures. Psychological Bulletin 88: 588—606.[CrossRef]
  • Bentler, P.M. and Dijkstra, T. 1985: Efficient estimation via linearization in structural models. In Krishnaiah, P.R., editor, Multivariate analysis VI. Amsterdam: North-Holland, 9—42.
  • Breen, M.P., Barrat-Pugh, C., Derewianka, B., House, H., Hudson, C., Lumley, T. and Rohl, M. 1997: Profiling ESL Children: How teachers interpret and use national and state assessment frameworks. Canberra: Department of Employment, Education, Training, and Youth Affairs, Commonwealth of Australia.
  • Brindley, G. 1998: Outcomes-based assessment and reporting in language learning programmes: A review of the issues. Language Testing 15: 45—85.[Abstract/Free Full Text]
  • ———— 2001: Outcomes-based assessment in practice: Some examples and emerging insights. Language Testing 18: 393—407.[Abstract/Free Full Text]
  • Brown, A. 1995: The effect of rater variables on the development of an occupation-specific language performance test. Language Testing 12: 1—15.[Medline] [Order article via Infotrieve]
  • Byrne, B.M. 2006: Structural equation modeling with EQS: Basic concepts, applications, and programming. Mahwah, NJ: Lawrence Erlbaum.
  • Campbell, D.T. and Fiske, D.W. 1959: Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin 56: 81—105.[CrossRef][Medline] [Order article via Infotrieve]
  • Chalhoub-Deville, M. 1995: Deriving oral assessment scales across different tests and rater groups. Language Testing 12: 16—33.[Abstract/Free Full Text]
  • Conway, J.M., Scullen, S.E., Lievens, F. and Lance, C.E. 2004: Bias in the correlated uniqueness model for MTMM data. Structural Equation Modeling: A Multidisciplinary Journal 11: 535—59.[CrossRef]
  • Cumming, A., Grant, L., Mulcahy-Ernt, P. and Powers, D.E. 2004: A teacher-verification study of speaking and writing prototype tasks for a new TOEFL. Language Testing 21: 107—45.[Abstract/Free Full Text]
  • Davison, C. 2004: The contradictory culture of teacher-based assessment: ESL teacher assessment practices in Australian and Hong Kong secondary schools. Language Testing 21: 305—34.[Abstract/Free Full Text]
  • Elder, C. 1993: How do subject specialists construe classroom language proficiency? Language Testing 10: 233—54.
  • Epp, L. and Stawychny, M. 2001: Using the Canadian Language Benchmarks (CLB) to benchmark college programs/courses and language proficiency tests. TESL Canada Journal 18: 32—47.
  • Gattullo, F. 2000: Formative assessment in ELT primary (elementary) classrooms: An Italian case study. Language Testing 17: 278—88.[Free Full Text]
  • Gipps, C. 1994: Beyond testing: Towards a theory of educational assessment. London: Falmer Press.
  • Grant, L. 1997: Testing the language proficiency of bilingual teachers: Arizona's Spanish proficiency test. Language Testing 14: 23—46.[Abstract/Free Full Text]
  • Hoge, R. and Coldarci, T. 1989: Teacher-based judgments of academic achievement: A review of literature. Review of Educational Research 59: 297—313.[Abstract/Free Full Text]
  • Hu, L.T. and Bentler, P.M. 1999. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal 6: 1—55.
  • Kenny, D.A. 1976: Am empirical application of confirmatory factor analysis to the multitrait-multimethod matrix. Journal of Experimental Social Psychology 12: 247—52.[CrossRef]
  • ———— 1979: Correlation and causality. New York: Wiley.
  • Kenny, D.A. and Kashy, D.A. 1992: Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin 112: 165—72.[CrossRef]
  • Kunnan, A.J. 1995: Test taker characteristics and test performance: A structural modeling approach. Cambridge: Cambridge University Press.
  • Lance, C.E., Noble, C.L. and Scullen, S.E. 2002: A critique of the correlated trait-correlated method and correlated uniqueness models for multitrait-multimethod data. Psychological Methods 7: 228—44.[CrossRef][Medline] [Order article via Infotrieve]
  • Leung, C. and Mohan, B. 2004: Teacher formative assessment and talk in classroom contexts: Assessments as discourse and assessment of discourse. Language Testing 21: 335—59.[Abstract/Free Full Text]
  • Linquanti, R. 2001: The redesignation dilemma: Challenges and choices in fostering meaningful accountability for English learners. Policy Report 2001—1. Santa Barbara, CA: University of California Linguistic Minority Research Institute, University of California, Santa Barbara.
  • Llosa, L. 2005a: Assessing English learners' language proficiency: A qualitative investigation of teachers' interpretations of the California ELD Standards. The CATESOL Journal 17: 7—18.
  • ———— 2005b: Building and supporting a validity argument for a standards-based classroom assessment of English proficiency. Unpublished PhD dissertation, University of California, Los Angeles.
  • Lumley, T.J.N. and McNamara, T.F. 1995: Rater characteristics and rater bias: Implications for training. Language Testing 12: 54—71.[Abstract/Free Full Text]
  • Marsh, H.W. 1989: Confirmatory factor analyses of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement 15: 47—70.[CrossRef]
  • Marsh, H.W., Byrne, B.M. and Craven, R. 1992: Overcoming problems in confirmatory factor analyses of MTMM data: The correlated uniqueness model and factorial invariance. Multivariate Behavioral Research 27: 489—507.[CrossRef]
  • Marsh, H.W. and Grayson, D. 1995: Latent variable models of multitrait-multimethod data. In Hoyle, R.H., editor, Structural equation modeling: Concepts, issues, and applications. Thousand Oaks, CA: Sage, 177—98.
  • McNamara, T. 2001: Language assessment as social practice: Challenges for research. Language Testing 18: 333—49.[Abstract/Free Full Text]
  • Meisels, S.J., Bickel, D.D., Nicholson, J., Xue, Y. and Atkins-Burnett, S. 2001: Trusting teacher judgments: A validity study of a curriculum-embedded performance assessment in kindergarten to grade 3. American Educational Research Journal 38: 73—95.[Abstract/Free Full Text]
  • Muthén, B.O. and Kaplan, D. 1985: A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical and Statistical Psychology 38: 171—89.
  • North, B. 1993: The development of descriptors on scales of language proficiency. College Park, MD: The National Foreign Language Center.
  • ———— 1995: The development of a common framework scale of descriptors of language proficiency based on a theory of measurement. System 23: 445—65.[CrossRef]
  • ———— 2000: The development of a common framework scale of language proficiency. New York: Peter Lang.
  • O'Sullivan, B., Weir, C. and Saville, N. 2002: Using observation checklists to validate speaking-test tasks. Language Testing 19: 33—56.[Abstract/Free Full Text]
  • Popham, W.J. 2003: The trouble with testing: Why standards-based assessment doesn't measure up. American School Board Journal 190: 14—17.
  • Rea-Dickins, P. 2001: Mirror, mirror on the wall: Identifying processes of classroom assessment. Language Testing 18: 429—62.[Abstract/Free Full Text]
  • Rea-Dickins, P. and Gardner, S. 2000: Snares and silver bullets: Disentangling the construct of formative assessment. Language Testing 17: 215—43.[Abstract/Free Full Text]
  • Rindskopf, D. and Rose, T. 1988: Some theory and applications of confirmatory second-order factor analysis. Multivariate Behavioral Research 23: 51—67.[CrossRef]
  • Satorra, A. and Bentler, P.M. 1988: Scaling corrections for chi-square statistics in covariance structure analysis. Proceedings of the American Statistical Association, 308—13.
  • Sawaki, Y. 2007: Construct validation of analytic rating scales in a speaking assessment: Reporting a score profile and a composite. Language Testing 24(3), 355—90.[Abstract/Free Full Text]
  • Shepard, L.A. 2000: The role of classroom assessment in teaching and learning. CSE Technical Report 517. National Center for Research on Evaluation, Standards, and Student Testing (CRESST). University of California, Los Angeles. Retrieved on 10 April 2005 from: www.cse.ucla.edu/CRESST/Reports/TECH517.pdf
  • Shin, S.K. 2005: Did they take the same test? Examinee language proficiency and the structure of language tests. Language Testing 22: 31—57.[Abstract/Free Full Text]
  • SPSS for Windows, Rel. 14.0.0. 2005. Chicago: SPSS.
  • Stansfield, C. and Kenyon, D. 1996: Comparing the scaling of speaking tasks by language teachers and by the ACTFL guidelines. In Cumming, A. and Berwick, R., editors, Validation in language testing. Clevedon, UK: Multilingual Matters, 124—53.
  • Stevens, J.J. and Clauser, P. 1995: Multitrait-multimethod comparisons of a writing portfolio and the ITBS. Paper presented at the annual meeting of the National Council for Measurement in Education, San Francisco, CA.
  • Taylor, C.S. 2002: Incorporating classroom-based assessments into large-scale assessment programs. In Tindal, G. and Haladyna, T.M., editors, Large-scale assessment: Programs for all students. Mahwah, NJ: Lawrence Erlbaum, 233—59.
  • Widaman, K.F. 1985: Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement 9: 1—26.[Medline] [Order article via Infotrieve]

Language Testing, Vol. 24, No. 4, 489-515 (2007)
DOI: 10.1177/0265532207080770
Add to CiteULike CiteULike   Add to Complore Complore   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to Technorati Technorati   Add to Twitter Twitter    What's this?



This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Citing Articles
Right arrow Citing Articles via Google Scholar
Right arrow Citing Articles via Scopus
Google Scholar
Right arrow Articles by Llosa, L.
Right arrow Search for Related Content
Social Bookmarking
Add to CiteULike   Add to Complore   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati   Add to Twitter  
What's this?