This book is a welcome addition to the growing body of literature on automated essay scoring, also known as automated essay evaluation technology (AEET). I am writing my dissertation on this subject (from a composition instructor's perspective), and the book has helped me tremendously.
Automated Essay Scoring is organized into five major sections: (I) Teaching of Writing, (II) Psychometric Issues in Performance Assessment, (III) Automated Essay Scorers, (IV) Psychometric Issues in Automated Essay Scoring, and (V) Current Innovation in Automated Essay Evaluation. Section III contains a detailed discussion of the leading essay scorers on the market: Project Essay Grade, Intellimetric, e-rater, and the Intelligent Essay Assessor. There is also a chapter on Bayesian text classification. Curiously, though, no mention is made of BETSY-the Bayesian Essay Test Scoring sYstem, which, to my knowledge, is the most fully developed essay scorer using Bayes Theorem.
The authors and editors have done a remarkably good job of presenting their respective grading engines in a fair and balanced manner without bashing competing technologies. That said, it is clear at times where the authors have placed their allegiance. For example, Jill Burstein, one of the editors of the book, is employed by ETS and penned several chapters in which she touts the benefits of e-rater. Nothing wrong with that, though, as she does it without tearing down her competitors.
I have only a few closely related criticisms of this book. The first is that the subtitle is somewhat of a misnomer. The vast majority of the book appears to be written by, for, and to psychometricians or people with substantial psychometric training. It therefore can hardly be called "cross disciplinary." As a writing instructor, I had hoped to find much more discussion written from a composition teacher's point of view. I was disappointed, however, to find that only two of the book's thirteen chapters (chapters 1 and 3) made any attempts to address the concerns that writing instructors have regarding these technologies. This seems ironic since writing teachers comprise the population most likely to be affected by the proliferation of AEETs.
As a whole, teachers of writing and researchers in the field of rhetoric and composition have not responded favorably to the idea of having computers teach and assess writing. Some see it as a return to philosophical positivism or an abandonment of the postmodern values on which the "New English" was built. Regardless of one's position on the issue, such concerns deserve attention, and I was disappointed not to see them addressed in a volume as distinctive and comprehensive as this.
Be that as it may, Lawrence Erlbaum has a deserved reputation for publishing high quality academic material, and this volume is certainly fitting to that reputation. Despite its shortcomings, Automated Essay Scoring: A Cross-Disciplinary Perspective is a work of scholarship no one pursuing research in this field should do without.
