AACR III: Expanding a National Network for Automated Analysis of Constructed Response Assessments to Reveal Student Thinking in STEM

TitleAACR III: Expanding a National Network for Automated Analysis of Constructed Response Assessments to Reveal Student Thinking in STEM
Publication TypeConference Proceedings
Year of Publication2014
AuthorsUrban-Lurain, M, Merrill, JE, Cooper, M, Lira, C, Haudek, KC
Conference NameCREATE4STEM mini-conference
Date Published02/2014
Conference LocationMichigan State University, East Lansing, MI
KeywordsAACR, Assessment, constructed response, Lexical analysis, student thinking
AbstractImproving STEM education requires valid and reliable instruments for providing insight into student thinking. Constructed response (CR) assessments reveal more about student thinking and the persistence of misconceptions than do multiple-choice questions, but require more analysis on the part of educators. We have developed constructed response versions of well-established conceptual assessment inventories and created computer automated analysis resources that predict human ratings of student writing about these topics in introductory STEM courses. The research uses a two-stage, feature-based approach to automated analysis of constructed response assessments. First, we design items to identify important disciplinary constructs based on prior research. The items are administered via online course management systems where students enter responses. We use lexical analysis software to extract key terms and scientific concepts from the students’ writing. These terms and concepts are used as variables for statistical classification techniques to predict expert ratings of student responses. The inter-rater reliability (IRR) between automated predictions and expert human raters is as high as IRR between human experts. We recently received another round of funding to extend our work to provide an online community where instructors may obtain, score and contribute to the library of items and resources necessary for their analyses. The specific goals of this project are to: 1 Create a community web portal for the assessments to expand and deepen collaborations among STEM education researchers. This portal will provide the infrastructure for expanding the community of researchers and supporting the adoption and implementation of the innovative instructional materials by instructors at other institutions. 2 Transport the innovations by providing instructors professional development and support to use the assessments. This includes information about common student conceptions revealed by the questions, instructional materials for addressing conceptual barriers, and the opportunity to join a community of practitioners who are using the questions and exchanging materials, and long-term, ongoing support. 3 Expand the basic research to create and validate items in introductory chemistry, chemical engineering, and statistics. 4 Engage in ongoing project evaluation for continuous quality improvement and to document the challenges and successes the project encounters. 5 Lay the foundation for sustainability by providing interfaces for e-text publishers, Learning Management System (LMS) vendors and Massively Open Online Courses (MOOCs) as potential revenue streams to operate and maintain the online infrastructure. This poster will provide an overview of the goals of the project and introduce the development of a national network of faculty using these techniques.
Non-MSUCREATE4STEM mini-conference


thumbnail of small NSF logo in color without shading

This material is based upon work supported by the National Science Foundation (DUE grants: 1438739, 1323162, 1347740, 0736952 and 1022653). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.