Expanding a National Network for Automated Analysis of Constructed Response Assessments to Reveal Student Thinking in STEM

TitleExpanding a National Network for Automated Analysis of Constructed Response Assessments to Reveal Student Thinking in STEM
Publication TypeJournal Article
Year of Publication2015
AuthorsUrban-Lurain, M, Cooper, MM, Haudek, KC, Kaplan, JJ, Knight, JK, Lemons, PP, Lira, CT, Merrill, JE, Nehm, RH, Prevost, LB, Smith, MK, Sydlik, M
JournalComputers in Education Journal
Type of ArticleJournal Article
AbstractImproving STEM education requires valid and reliable instruments for providing insight into student thinking. Constructed response (CR) assessments reveal more about student thinking and the persistence of misconceptions than do multiple-choice questions, but require more analysis on the part of educators. In the Automated Analysis of Constructed Response (AACR) Research Group (www.msu.edu/ aacr) we have developed constructed response versions of well-established conceptual assessment inventories and created computer automated analysis resources that predict human ratings of student writing about these topics in introductory STEM courses. The research uses a two-stage, feature-based approach to automated analysis of constructed response assessments. First, we design items to identify important disciplinary constructs based on prior research. The items are administered via online course management systems where students enter responses. We use lexical analysis software to extract key terms and scientific concepts from the students’ writing. These terms and concepts are used as variables for statistical classification techniques to predict expert ratings of student responses. The inter-rater reliability (IRR) between automated predictions and expert human raters is as high as IRR between human experts. We recently received another round of funding to extend our work to provide an online community where instructors may obtain, score and contribute to the library of items and resources necessary for their analyses. We provide an overview of the goals of the project and introduce the opportunities to participate in the development of a national network of faculty using these techniques.

thumbnail of small NSF logo in color without shading

This material is based upon work supported by the National Science Foundation (DUE grants: 1438739, 1323162, 1347740, 0736952 and 1022653). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.