The seductive appeal of measuring outcomes in edtech research – SAAHE SIG webinar

This is a presentation I gave as part of a SAAHE special interest group webinar on edtech research.

Download the slides.

I wanted to explore what I think is a problematic area of education research in general, and that’s especially common in education technology. The problem, as I see it, is the tendency of researchers to use comparative study designs in edtech research. Researchers typically use an experimental-type study design where one group of students is ‘exposed’ to the edtech intervention, while another goes through ‘business as usual’. I don’t go into all the other reasons that these designs don’t work well, but rather focus this presentation on the emphasis that authors usually place on the achievement of certain outcomes.

There are two common findings in these types of studies; no significant difference in outcomes (but students enjoyed the experience), or the experimental arm of the design has a small but statistically significant improvement in outcomes. In most cases, the authors assume that it was the edtech intervention that led to the change, without controlling for any of the potential confounding variables. So that’s the short version of why I think comparative-type studies in edtech research are problematic (although I think there are several other good reasons, which we cover in this article from a few years ago).

My intention with this presentation was to briefly describe my concerns and then explore alternatives. But instead, I argued myself into the conclusion that there’s no point in even trying to study the effect of teaching on learning, and that there’s certainly no point in attempting to use comparative study designs in edtech research.

After sitting with the problem for a while, I stumbled across an unanticipated ‘solution’: What if decouple the assumed cause-effect relationship between teaching and learning? What if we separate out the practice of teaching from the process of learning, where teaching is not evaluated for its ability to cause learning, and learning is not evaluated as an outcome of teaching. Instead of asking if A is better than B, ask students about their relationships with A and B. Are students’ directly observable behaviours more closely aligned with what we know from cognitive science and educational psychology, to be ‘good’ learning behaviours? Are teaching practices more, or less, closely aligned with what we know constitutes ‘good’ practice?

I’m not 100% sure if this is possible, or practical, but it’s an interesting line of inquiry (IMO), not only for edtech researchers but for anyone interested in pedagogical research in general.