Evaluation of construct-irrelevant variance yielded by machine and human scoring of a science teacher PCK constructed response assessment

Abstract

Machine learning has been frequently employed to automatically score constructed response assessments. However, there is a lack of evidence of how this predictive scoring approach might be compromised by construct-irrelevant variance (CIV), which is a threat to test validity. In this study, we evaluated machine scores and human scores with regard to potential CIV. We developed two assessment tasks targeting science teacher pedagogical content knowledge (PCK); each task contains three video-based constructed response questions. 187 in-service science teachers watched the videos with each had a given classroom teaching scenario and then responded to the constructed-response items. Three human experts rated the responses and the human-consent scores were used to develop machine learning algorithms to predict ratings of the responses. Including the machine as another independent rater, along with the three human raters, we employed the many-facet Rasch measurement model to examine CIV due to three sources: variability of scenarios, rater severity, and rater sensitivity of the scenarios. Results indicate that variability of scenarios impacts teachers’ performance, but the impact significantly depends on the construct of interest; for each assessment task, the machine is always the most severe rater, compared to the three human raters. However, the machine is less sensitive than the human raters to the task scenarios. This means the machine scoring is more consistent and stable across scenarios within each of the two tasks.

Author

Xiaoming Zhai, Kevin Haudek, Molly Stuhlsatz, Christopher Wilson

Keywords

Construct-irrelevant variance, Machine learning, Pedagogical content knowledge, Science teacher, Constructed response assessment

Year of Publication

2020

Journal

Studies in Educational Evaluation

Volume

67

Number of Pages

100916

ISSN Number

0191-491X

URL

https://www.sciencedirect.com/science/article/pii/S0191491X20301644

DOI

https://doi.org/10.1016/j.stueduc.2020.100916