Covariational reasoning and item context affect student language in undergraduate mass balance explanations
Abstract
Study Context: A major goal of undergraduate STEM education is to increase scientific thinking, and skills, such as constructing explanations. To do this, these abilities must be provoked and assessed in multiple contexts as student abilities and thinking can change with item context (Nehm & Ridgeway, 2011; Nehm & Ha, 2011). Previous work found that during group work, students progress through different stages of language to successfully communicate understanding (Capraro et al, 2018). Constructed response (CR) assessments allow students to use their own words and reflect student thinking revealed in interviews, but are easier to collect (Scott et al., in review, Nehm & Schonfeld, 2008). Student language in CRs can be examined by comparing word usage (text analysis) or quantifying lexical diversity. In combination, these techniques provide quantitative measurements of language based on categorical data such as the degree of scientific thinking (Shiroda et al. in review). Mass balance reasoning (accumulation or pool and flux) offers a rich topic for scientific thinking and skills, as it requires students to account for multiple inputs and outputs within a system and apply covariational reasoning (CVR). This skill is broadly applicable across science disciplines, but difficulty persists even for graduate students (Sweeney & Sterman, 2007; Scott et al., in review). Little work examines the language students use to demonstrate such scientific skills in disciplinary contexts.
Research Design: We used the same general CR prompt about mass balance in six contexts (Scott et al., in review). Three organisms undergo an identical change or stimulus, resulting in an influx of matter. Students are asked to explain how one organism has a larger mass pool and how another organism shows no change in total mass despite the influx. We used 1,920 CRs from undergraduates in physiology courses at eight institutions that evenly represent four levels of CVR within the six contexts. We asked how context and level affect student language in the CRs in the overall dataset and within each context. We compared word usage using text analysis and examined lexical diversity using ecological metrics including Bray-Curtis distance (BCd). BCd ranges from 0-100 with 0 indicating low diversity. We also visualized the corpus using ordination, a data reduction technique.
Analyses and Interpretations: The BCd of the dataset is 83.1, revealing it to be relatively diverse. In comparison, groupings of CRs by individual contexts have lower values (range: 72.1-76.3) making them more similar to each other than the overall data set. In comparison, all but the highest CVR level (CVR4; 73.5) have similar BCds to the overall set (range: 80.7-87.6), indicating CVR4 responses are uniquely more similar to each other regardless of context. We observed this pattern in the six individual contexts, as CVR4 (range: 57.7-61.4) is much lower than CVR1 (73.8-84.3). From text analysis, we found item contextual words at each CVR; however, the type of context word varied. Specifically, at lower CVR levels, students use context words that are tangential to mass balance reasoning, while higher CVR responses use words that specify inputs and outputs of the system. For example, in the context of glucose accumulation in a leaf, CVR4 responses use photosynthesis (input) and respiration (output), while CVR1 responses include chlorophyll and photosystem. By examining language within each context, we found that students used certain words across all six contexts at CV4, including rate, equal, and some form of slower/lower/smaller, while there are no shared words in each context at lower CVRs. Finally, ordination visualized relatedness of the CRs. Overall, context had more distinct groupings of responses (113 DCA units) than CVR (226). Within individual contexts, language became increasingly similar with increased CVR (CVR1 191; CVR4 111). Together, these data demonstrate that context affects undergraduate mass balance language at all CVR levels, but that the language becomes more specific as CVR increases. It also establishes that student language has a similar progression in each context and ends with similar language at the highest CVR levels.
Contribution: This work indicates each context functionally builds the same language, further encouraging instructors to seek out multiple contexts for instruction and assessment. The language found herein provides instructors with building blocks for improving student explanations of mass balance problems. Namely, the words used in higher CVR levels are not technical outside of identifying fluxes, but comparative and summative. Broadly, student language in textual CRs is understudied. While this work focuses on CVR in mass balance, the combination of text analysis and lexical diversity will be useful throughout STEM education to better understand the impacts of different variables on development of student reasoning and explanations.