Formative Automated Essay Scoring

July 19th, 2009

I usually think of automated essay scoring as something used for the big national and international tests (SAT, NAEP, etc.) However, an article by Scharber, Dexter, & Riedel looked at student responses to automated scorers a little differently; the automated essay scorer was used in a formative fashion. This is interesting as I think about all the students who have asked me if I will look at a draft before submission, and all the time it takes to do that. The system was designed so students could submit their essay, receive a score, and then have the opportunity to revise it before either submitting it to the automated scorer again or to the instructor.

The essays in the study were judged on a three point rubric. Looking at the data, it appears that students resubmitted to the automated scorer multiple times without getting much change in their scores. In addition, the automated scorer was not an accurate predictor of instructor’s scores; the instructor consistently scored the essays higher than the automated scorer. Finally, the scorer did not provide any feedback beyond the numerical score.

Not surprisingly, the students did not find the scorer helpful and did not believe it was accurate. This paper reads somewhat like a “what not to do” in designing a formative automated essay scorer, but we can learn that:

  • the automated essay scorer needs to be calibrated to the instructor teaching the class
  • the automated scorer needs to provide more information than just a score
  • the scorer (and perhaps the rubric) needs to be sensitive to changes in the essay

So, while this article does not make the automated scorer look very promising, I think it is worth trying with some of the tweaks noted. I think it could help improve students’ writing if they could get substantive feedback on drafts.

Reference: Scharber, C., Dexter, S., & Reidel, E. (2008). Students’ experiences with an automated essay scorer. Journal of Technology, Learning, and Assessment, 7(1). Retrieved July 18 from: http://escholarship.bc.edu/cgi/viewcontent.cgi?article=1116&context=jtla

Tags: ,
Posted in Assessment | Comments (0)

Trackback

Leave a Reply

Subscribe

About

Connections Research is the blog for Connections Learning & Education Research. Look for summaries and commentary on new education-related research, as well my own observations of the field.

Blogroll

The Bookshelf

Image of How We Think
Image of Why Don't Students Like School: A Cognitive Scientist Answers Questions About How the Mind Works and What It Means for the Classroom
Image of How People Learn: Brain, Mind, Experience, and School: Expanded Edition