Construction of rubrics to evaluate content in students' scientific explanation using computerized text analysis

TitleConstruction of rubrics to evaluate content in students' scientific explanation using computerized text analysis
Publication TypeConference Paper
Year of Publication2015
AuthorsHaudek, KC, Moscarella, RA, Weston, MM, Merrill, J, Urban-Lurain, M
Conference NameNARST 2015 Annual International Conference
PublisherNARST
Keywordslexical analysis rubric development text analysis
AbstractA major challenge to using constructed response items in high enrollment undergraduate STEM courses is the ability to evaluate the scientific content of student explanations, which can be aided using evaluation rubrics. However, traditional rubric development is highly qualitative and is often an iterative process and requires subject expertise and the ability to identify emerging themes from student writing. Here, we report on leveraging the results of lexical and statistical analysis of students’ writing to develop a rubric to evaluate scientific content contained in student responses to two questions used in undergraduate biology courses. Lexical analysis was used to identify and categorize relevant content in student responses. These lexical categories were used as variables in K-means clustering, which helped identify emergent themes or patterns across student responses. Each cluster was defined by categories that included relevant disciplinary content. These clusters were used as the basis of an initial rubric. Several raters applied this rubric to a subset of student responses and after revisions to the rubric and scoring iterations, achieved varying levels of inter-rater reliability (from 0.4 to 0.8 Cohen’s kappa) on different rubric criteria. We believe this methodology may be broadly useful for reducing the effort necessary for rubric creation.

Attachments: 

thumbnail of small NSF logo in color without shading

This material is based upon work supported by the National Science Foundation (DUE grants: 1438739, 1323162, 1347740, 0736952 and 1022653). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.