Automated analysis of constructed responses: What are we modeling?

TitleAutomated analysis of constructed responses: What are we modeling?
Publication TypeConference Paper
Year of Publication2015
AuthorsUrban-Lurain, M, Merrill, J, Haudek, K, Nehm, R, Moscarella, R, Steele, M, Park, M
Conference NameSociety for the Advancement of Biology Education Research
AbstractIn the Automated Analysis of Constructed Response (AACR) research group, we are exploring a number of approaches to computerized analysis of student writing in introductory STEM courses. We use an iterative process to develop questions, collect data, create analytic models that predict expert evaluations, and validate the models. We then use these models to analyze student writing and generate reports for faculty that summarize their students’ thinking about core ideas in the disciplines. With sufficient data and refinement, our models predict expert ratings with inter-rater reliability (IRR) between the computer models and experts as good as IRR among experts, generally > .8. We find that faculty are enthusiastic about the opportunity to have their students write in large introductory courses, but many are skeptical about how computers can analyze writing. In this talk, we will outline the conceptual and theoretical basis for the AACR work, describe the process by which we create the models, and discuss the linguistic and cognitive structures that we are modeling. By opening the “black box” of this process, we anticipate that faculty will be more confident in the information that AACR can provide and will better understand the cognitive processes by which learners move from novice towards expertise in a discipline.

thumbnail of small NSF logo in color without shading

This material is based upon work supported by the National Science Foundation (DUE grants: 1438739, 1323162, 1347740, 0736952 and 1022653). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.