This Article Abstract Free Full Text (Free PDF) Free Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Guoxing Yu Search for Related Content Social Bookmarking                         What's this?

Students' voices in the evaluation of their written summaries: Empowerment and democracy for test takers?

University of Bristol, UK, Guoxing.Yu{at}bristol.ac.uk

Two kinds of scoring templates were empirically derived from summaries written by experts and students to evaluate the quality of summaries written by the students. This paper reports students' attitudes towards the use of the two templates and its differential statistical effects on the judgment of students' summarization performance. It was found that a summary consistently received higher scores when judged by the popular template, regardless of the language and the language order (English then Chinese or Chinese then English) in which a summary was produced. However, the scores from the expert template were slightly better predictors of students' reading abilities as measured by FCE and TOEFL. The majority of the students strongly preferred the expert template. Their arguments centered mainly on the differences between students and experts in their language abilities and experience, their stereotypical status in educational assessment, and the dialectical interpretations of `quantity' and `quality'. Implications of these findings are discussed with specific reference to the value of involving test-takers in assessment criteria development.

References

• Afflerbach, P.P. 1990: The influence of prior knowledge on expert readers' main idea construction strategies. Reading Research Quarterly 25: 31—46.[CrossRef]
• Alderson, J.C. 1996: The testing of reading. In Nuttall, C., editor, Teaching reading skills in a foreign language. London: Heinemann.
• ——— 2000: Assessing reading. Cambridge: Cambridge University Press.
• Alderson, J.C., Clapham, C. and Wall, D. 1995: Language test construction and evaluation. Cambridge: Cambridge University Press.
• Axelrod, J. 1975: Getting the main idea is still the main idea. Journal of Reading 18: 383—87.
• Bachman, L.F. 1990: Fundamental considerations in language testing. Oxford: Oxford University Press.
• Bachman, L.F. and Palmer, A.S. 1996: Language testing in practice: Designing and developing useful language tests. Oxford: Oxford University Press.
• Bachman, L.F., Davidson, F., Ryan, K. and Choi, I.-C. 1995: An investigation into the comparability of two tests of English as a foreign language: The Cambridge-TOEFL comparability study. Cambridge: Cambridge University Press.
• Balajthy, E. and Weisberg, R. 1990: Transfer effects of prior knowledge and use of graphic organizers on college developmental readers' summarization and comprehension of expository text. National Reading Conference Yearbook 39: 339—45.
• Barnwell, D. 1989: "Naïve" native speakers and judgments of oral proficiency in Spanish. Language Testing 6: 152—63.[Abstract/Free Full Text]
• Basham, C.S. 1987: Summary writing as cultural artifact. Fairbanks, AK: Cross Cultural Communications, University of Alaska.
• Bensoussan, M. and Kreindler, I. 1990: Improving advanced reading comprehension in a foreign language: Summaries vs. Short-answer questions. Journal of Research in Reading 13: 55—68.[CrossRef]
• Bernhardt, E.B. 1991: Reading development in a second language: Theoretical, empirical, and classroom perspectives. Norwood, NJ: Ablex.
• Birenbaum, M. 1996: Assessment 2000: Towards a pluralistic approach to assessment. In Birenbaum, M. and Dochy, F.R.J., editors, Alternatives in assessment of achievements, learning processes, and prior knowledge. Dordrecht: Kluwer, 3—29.
• Boyd, K. and Davies, A. 2002: Doctors' orders for language testers: The origin and purpose of ethical codes. Language Testing 19: 296—322.[Abstract/Free Full Text]
• Brown, A.L., Campione, J.C. and Day, J.D. 1981: Learning to learn: On training students to learn from texts. Educational Researcher 10: 14—24.[Free Full Text]
• Brown, A.L. and Day, J.D. 1983: Macrorules for summarizing texts: The development of expertise. Journal of Verbal Learning and Verbal Behavior 22: 1—14.[CrossRef]
• Brown, A.L., Day, J.D., and Jones, R.S. 1983: The development of plans for summarizing texts. Child Development 54: 968—79.[CrossRef]
• Brown, A.L. and Smiley, S.S. 1978: The development of strategies for studying text. Child Development 49: 1076—88.[CrossRef]
• Bueckendorf, J.M. 1992: Comparative assessment of reading comprehension: Using multiple-choice and written summary formats with narrative and expository texts. Unpublished EdD thesis, University of Missouri, St Louis.
• Byrd, M. 1985: Age differences in the ability to recall and summarize textual information. Experimental Aging Research 11: 87—91.[Medline] [Order article via Infotrieve]
• Cohen, A.D. 1993: The role of instructions in testing summarizing ability. In Douglas, D. and Chapelle, C., editors, A new decade of language testing research. Washington, DC: TESOL, 132—59.
• ——— 1994: English for academic purposes in Brazil: The use of summary tasks. In Hill, C. and Parry, K., editors, From testing to assessment: English as an international language. London: Longman, 174—204.
• Cohen, J. 1988: Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum.
• Connor, U. and McCagg, P. 1983: Cross-cultural differences and perceived quality in written paraphrases of English expository prose. Applied Linguistics 4: 259—68.[CrossRef]
• Corder, S.P. 1973: Interpretive procedures: Seeing, reading and understanding discourse. In Anderson, R., editor, New dimensions in second language acquisition research. Rowley, MS: Newbury House.
• Courchêne, R. and Bayliss, D. 1995: Summary cloze: What is it? What does it measure? In Courchene, R., Burger, S., Cornaire, C., LeBlanc, R., Paribakht, S. and Seguin, H., editors, Twenty-five years of second language teaching at the University of Ottawa. Ottawa: Second Language Institute, University of Ottawa, 305—27.
• Cronbach, L.J., Linn, R.L., Brennan, R.L. and Haertel, E. 1995: Generalizability analysis for educational assessment. Los Angeles: UCLA's Center for the Study of Evaluation & National Center for Research on Evaluation, Standards, and Student Testing.
• Crowley, E. 1987: The relationship between ability to summarize and field independence or dependence. Unpublished PhD thesis, Illinois State University.
• Cumming, A., Rebuffot, J. and Ledwell, M. 1989: Reading and summarizing challenging texts in first and second languages. Reading and Writing: An Interdisciplinary Journal 2: 201—19.
• Davies, A. 2003: The native speaker: Myth and reality. Clevedon, Bristol: Multilingual Matters.
• Deville, C. and Chalhoub-Deville, M. 1993: Modified scoring, traditional item analysis and Sato's caution index used to investigate the reading recall protocol. Language Testing 10: 117—32.[Abstract/Free Full Text]
• Douglas, D. 2001: Language for specific purposes assessment criteria: Where do they come from? Language Testing 18: 171—85.[Abstract/Free Full Text]
• Douglas, D. and Myers, R. 2000: Assessing the communication skills of veterinary students: Whose criteria? In Kunnan, A., editor, Fairness and validation in language assessment: Selected papers from the 19th language testing research colloquium. Cambridge: University of Cambridge Press with University of Cambridge Local Examinations Syndicate, 60—81.
• Ets 2002: LanguEdgeTM Courseware Handbook for Scoring Speaking and Writing. Princeton, NJ: Educational Testing Service.
• Garner, R. 1982: Efficient text summarization: Costs and benefits. Journal of Educational Research 75: 275—79.
• Grenz, S.J. 1996: A primer on postmodernism. Grand Rapids, MI: William B. Eerdmans.
• Hare, V.C. and Borchardt, K.M. 1984: Direct instruction of summarization skills. Reading Research Quarterly 20: 62—78.[CrossRef]
• Head, M.H., Readence, J.E. and Buss, R.R. 1989: An examination of summary writing as a measure of reading comprehension. Reading Research & Instruction 28: 1—11.
• Hill, K. 1997: Who should be the judge? The use of non-native speakers as raters on a test of English as an international language. In Huhta, A., Kohonen, V., Kurki-Suonio, L. and Luoma, S., editors, Current developments and alternatives in language assessment. Tampere: Universities of Tampere and Jyväskylä.
• Huhta, A. and Randell, E. 1996: Multiple-choice summary: A measure of text comprehension. In Cumming, A. and Berwick, R., editors, Validation in language testing. Clevedon, UK: Multilingual Matters, 94—110.
• Jacoby, S. and McNamara, T.F. 1999: Locating competence. English for Specific Purposes 18: 213—41.[CrossRef]
• Johns, A.M. 1985: Summary protocols of "underprepared" and "adept" university students: Replications and distortions of the original. Language Learning 35: 495—517.[CrossRef]
• Johns, A.M. and Mayes, P. 1990: An analysis of summary protocols of university ESL students. Applied Linguistics 11: 253—71.[Abstract]
• Kiewit, S.F. 1997: The relationship of prior background knowledge to the summarization skills of developmental college students. Unpublished PhD thesis, University of Akron.
• Kintsch, W. 1974: The representation of meaning in memory. Hillsdale, NJ: Lawrence Erlbaum.
• Kintsch, W. and Kozminsky, E. 1977: Summarizing stories after reading and listening. Journal of Educational Psychology 69: 491—99.[CrossRef]
• Kintsch, W. and van Dijk, T.A. 1978: Toward a model of text comprehension and production. Psychological Review 85: 363—94.[CrossRef]
• Kobayashi, M. 2002: Method effects on reading comprehension test performance: Text organization and response format. Language Testing 19: 193—220.[Abstract/Free Full Text]
• Kuckartz, U. 1998: winMAX: Scientific text analysis for the social sciences (Version 98 Pro). Berlin: VERBI Software-Consult-Research GmbH (Germany).
• Lado, R. 1961: Language testing: The construction and use of foreign language tests. A teacher's book. London: Longman.
• ——— 1986: Analysis of native speaker performance on a cloze test. Language Testing 3: 130—46.[Medline] [Order article via Infotrieve]
• McNamara, T. 2001: Language assessment as social practice: Challenges for research. Language Testing18: 333—49.[Abstract/Free Full Text]
• Mast, C.O. 1988: The effects of cognitive styles on summarization of expository text. Unpublished PhD thesis, University of North Texas.
• Meyer, B.J.F. 1975: The organization of prose and its effects on memory. Amsterdam: North-Holland.
• Mills, C.B., Diehl, V.A., Birkmire, D.P. and Mou, L.C. 1993: Procedural text: Predictions of importance ratings and recall by models of reading comprehension. Discourse Processes 16: 279—315.
• Moore, T. 1997: From text to note: Cultural variation in summarization practices. Prospect 12: 54—63.
• Nuttall, C. 1996: Teaching reading skills in a foreign language. Oxford: Macmillan Heinemann.
• Penny, J., Johnson, R.L. and Gordon, B. 2000: The effect of rating augmentation on inter-rater reliability: An empirical study of a holistic rubric. Assessing Writing 7: 143—64.[CrossRef]
• Pollitt, A. and Hutchinson, C. 1987: Calibrating graded assessments: Rasch partial credit analysis of performance in writing. Language Testing 4: 72—92.[Abstract/Free Full Text]
• Riley, G.L. and Lee, J.F. 1996: A comparison of recall and summary protocols as measures of second language reading comprehension. Language Testing 13: 173—89.[Abstract/Free Full Text]
• Rivard, L.P. 2001: Summary writing: A multi-grade study of French-immersion and Francophone secondary students. Language, Culture and Curriculum 14: 171—86.
• Rosenau, P.M. 1992: Post-modernism and the social sciences: Insights, inroads and intrusions. Princeton, NJ: Princeton University Press.
• Sarig, G. 1989: Testing meaning construction: Can we do it fairly? Language Testing 6: 77—94.[Abstract/Free Full Text]
• Savignon, S. 2003: Teaching English as communication: A global perspective. World Englishes 22: 55—66.[CrossRef]
• Sawaki, Y. 2003: A comparison of summarization and free recall as reading comprehension tasks in web-based assessment of Japanese as a foreign language. Unpublished PhD thesis, University of California, Los Angeles (UCLA), Los Angeles.
• Schnotz, W. 1983: On the influence of text organization on learning outcomes. In Rickheit, G. and Bock, M., editors, Psycholinguistic studies in language processing. Berlin and New York: de Gruyter, 152—81.
• Scott, M.L., Stansfield, C.W. and Kenyon, D.M. 1996: Examining validity in a performance test: The listening summary translation exam (LSTE) — Spanish version. Language Testing 13: 83—109.[Abstract/Free Full Text]
• Shi, L. 2001: Native- and nonnative-speaking EFL teachers' evaluation of Chinese students' English writing. Language Testing 18: 303—25.[Abstract/Free Full Text]
• Shohamy, E. 1984: Does the testing method make a difference? The case of reading comprehension. Language Testing 1: 147—70.[Abstract/Free Full Text]
• Stansfield, C.W., Wu, W.M. and van der Heide, M. 2000: A job-relevant listening summary translation exam in Minnan. In Kunnan, A.J., editor, Fairness and validation in language assessment: Selected papers from the 19th LTRC. Cambridge: Cambridge University Press with University of Cambridge Local Examinations Syndicate, 177—200.
• Stevens, J. 2002: Applied multivariate statistics for the social sciences. Hillsdale, NJ: Lawrence Erlbaum.
• Taylor, L.B. 1996: An investigation of text-removed summary completion as a means of assessing reading comprehension. Unpublished PhD thesis, University of Cambridge.
• The New Encyclopaedia Britannica: Macropaedia: Knowledge in Depth 1992: 15th edition,Vol. 18: pp. 87—88.
• Turner, C.E. and Upshur, J.A. 2002: Rating scales derived from student samples: Effects of the scale maker and the student sample on scale content and student scores. TESOL Quarterly 36: 49—70.
• Urquhart, A.H. and Weir, C.J. 1998: Reading in a second language: Process, product and practice. London: Longman.
• van Dijk, T.A. and Kintsch, W. 1977: Cognitive psychology and discourse: Recalling and summarizing stories. In Dressler, W.U., editor, Current trends in textlinguistics. New York: de Gruyter, 61—80.
• ——— 1983: Strategies of discourse comprehension. New York: Academic Press.
• Wade, S.E., Buxton, W.M. and Kelly, M. 1999: Using think-alouds to examine reader-text interest. Reading Research Quarterly 34: 194—216.[CrossRef]
• Weir, C.J. 1993: Understanding and developing language tests. Hemel Hempstead: Prentice Hall International.
• Wenger, E. 1998: Communities of practice: Learning, meaning and identity. Cambridge: Cambridge University Press.
• Winograd, P.N. 1984: Strategic difficulties in summarizing texts. Reading Research Quarterly 19: 404—25.[CrossRef]
• Wolf, D.F., Bixby, J., Glenn, J.I. and Gardener, H. 1991: To use their minds well: Investigating new forms of student assessment. Review of Research in Education 17: 31—74.[Free Full Text]
• Yu, G. 2005: Towards a model of using summarization tasks as a measure of reading comprehension. Unpublished PhD thesis, University of Bristol.
• Zuck, L.V. and Zuck, J.G. 1984: The main idea: Specialist and non-specialist judgements. In Ulijn, J. and Pugh, A., editors, Reading for professional purposes: Studies and practices in native and foreign languages. London: Heinemann, 130—35.

Language Testing, Vol. 24, No. 4, 539-572 (2007)
DOI: 10.1177/0265532207080780


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This article has been cited by other articles:

				Guoxing Yu Reading to summarize in English and Chinese: A tale of two languages? Language Testing, October 1, 2008; 25(4): 521 - 551. [Abstract] [PDF]

This Article Abstract Free Full Text (Free PDF) Free Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Guoxing Yu Search for Related Content Social Bookmarking                         What's this?