education learning scholarship

When a metric becomes a target it fails to be a good metric.

Lately I’ve been thinking about metrics and all the ways that they can be misleading. Don’t get me wrong; I think that measuring is important. Measuring is the reason that our buildings and bridges don’t collapse. Measurements help tell us when a drug is working. GPS would be impossible without precise measurements of time. My Fitbit tells me when I’m exercising close to my maximum heart rate. So I’m definitely a fan of measuring things.

The problem is when we try to use measurements for things that aren’t easy to measure. For example, it’s hard to know when an article we publish has had an impact, so we look at the number of times that other researchers have used our articles as proxy indicators for their influence on the thinking of others. But this ignores the number of times that the articles are used to change a programme or trigger a new line of thinking in someone who isn’t publishing themselves. Or we use the number of articles being published in a department as a measure of “how much” science that department is doing. But this prioritises quantity over quality and ignores the fact that what we really want is a better understanding of the world, not “more publications”.

It sometimes feels like academia is just a weird version of Klout where we’re all trying to get better at increasing our “engagement” scores and we’ve forgotten the purpose of the exercise. We’ve confused achieving better scores on the metric rather than workign to move the larger project forward. We publish articles because articles are evidence that we’re doing research, and we use article citations and journal impact factors as evidence that our work is influential. But when a metric becomes a target it fails to be a good metric.

We see similar things happening all around us in higher education. We use percentages and scores to measure learning, even though we know that these numbers in themselves are subjective and sometimes arbitrary. We set targets in departments that ostensibly help us know when we’ve achieved an objective but we’re only mildly confident that the behaviours we’re measuring will help achieve the objective. For example, you have to be in the office for a certain number of hours each week so that we know that you’re working. But I don’t really care how often you’re in your office; I only really care about the quality of the work you do. But it’s hard to measure the quality of the work you do so I measure the thing that’s easy to measure.

This isn’t to say that we shouldn’t try to measure what we value, only that measurement is hard and that the metrics we choose will influence our behaviour. If I notice that people at work don’t seem to like each other very much I might start using some kind of  likeability index that aims to score everyone. But then we’d see people trying to increase their scores on the index rather than simply being kinder to each other. What I care about is that we treat each other well, not how well we each score on a metric.

We’ve set up the system so that students – and teachers – care more about the score achieved on the assessment rather than learning or critical thinking or collaborating. We give students page limits for writing tasks because we don’t want them to write everything in the hope that some of what they write is what we’re looking for. But then they play around with different variables (margin and font sizes, line spacing, title pages, etc.) in order to hit the page limit. What we really care about are other things, for example the ability to answer a question clearly and concisely, from a novel perspective, and to support claims about the world with good arguments.

I don’t have any solutions to the problem of measurement in higher education and academia. It’s a ahrd problem. I’m just thinking out loud about the fact that our behaviours are driven by what we’ve chosen to measure, and I’m wondering if maybe it’s time to start using different metrics as a way to be more intentional about achieving what we say we care about. Maybe it doesn’t even matter what the metrics are. Maybe what matters is how the choice of metrics can change certain kinds of behaviours.