Assessment and data assemblages: Changing the world simply by ‘being’ in it

We use student achievement in tests and exams to try and make accurate predictions about their future performance but we know that this practice is neither valid nor reliable. The test environment doesn’t look like the real world environment (so it’s not valid), and the metrics we use to measure test outcomes aren’t reliable because different assessors give different grades for the same student performance.

So we know what assessment as it currently exists is a terrible way to evaluate students. And yet this is what everyone does. However, I think that this is going to change as we move into an era where who we are and what we do can be inferred by the data we generate and how that data interacts with the data generated by devices. In other words, our being in the world creates a data assemblage from which we can make fairly good inferences without having to rely on unreliable and invalid proxy indicators (like tests).

As an example, here are just a few articles that showed up in my feed over the past few days, each demonstrating a different way of measuring different aspects of our lives, without even being designed to do so. Note that this is just what researchers are able to infer about us from how our mere presence in a space interacts with devices and generates new data.

By analyzing the exact ways a Wi-Fi signal is altered when a human moves through it, researchers can ‘see’ what someone writes with their finger in the air, identify a particular person by the way that they walk, and even read a person’s lips with startling accuracy…Once a WiFi router is already aware of a given person’s height and physique, experiments have proven their ability to identify humans with alarming veracity. Even when the sample size increases to six people (even with a wall separating you!), WiFi routers STILL will be able to accurately identify their target 89% of the time.

Scott, D. (2016). The truth about what your own WiFi router knows about your life.

By shining a laser through the fiber optics, the scientists could detect vibrations from above ground thanks to the way the cable ever so slightly deformed. As a car rolled across the subterranean cable or a person walked by, the ground would transmit their unique seismic signature. So without visually surveilling the surface, the scientists could paint a detailed portrait of how a once-bustling community ground to a halt, and slowly came back to life as the lockdown eased.

Simon, M. (2021). How Underground Fiber Optics Spy on Humans Moving Above.

Standard headphones with no microphone or sensors can detect your heart rate, identify you from the shape of your ear canal and even record your voice.

Sparkes, M. (2021). Regular headphones can detect a heart rate and hear your conversations. (article behind a paywall)

But the one that really caught my attention was this paper demonstrating what can be inferred about us from data generated by the accelerometers (ACC) in our phones.

Drawing from patents and literature of diverse disciplines, our paper shows that ACC data alone can be sufficient to obtain information about a device holder’s daily routines, physical activities, social interactions, health condition, gender, age, and emotional state… “physical activities” not only include high-level motion states (e.g., walking, cycling, sitting, climbing stairs) but – with ACC data from wrist wearables – also more fine-grained activities (e.g., writing, eating, smoking, sorting paperwork, searching the Internet). ACC data can also be used to uniquely identify users (based on biometric movement patterns) and to reconstruct sequences of text entered into a device, including passwords (based on micro-motions of the user’s hand). Further, ACC data can be analyzed to assess a user’s driving style, to estimate a user’s level of intoxication by the way they move, and to locate a user – even when GPS is disabled(!)

Kröger, J. L., Raschke, P., & Bhuiyan, T. R. (2019). Privacy implications of accelerometer data: A review of possible inferences. Proceedings of the 3rd International Conference on Cryptography, Security and Privacy – ICCSP ’19, 81–87.

What lesson do I take from this? We change the world simply by being in it (which isn’t a new idea) and that the ways in which we change the world can be measured very accurately (which is). It seems reasonably clear that personal behaviour will become increasingly quantifiable and that this is going to have an impact on how we think about assessment.

A prediction? At some point we’ll end summative assessment of students and will simply collect data from their “being in the world”. These data assemblages will be used to make inferences about student performance as they go about the process of learning in an ongoing formative assessment that will include feedback provided by AI-based systems as well as learning coaches. Our job as teachers will be to ensure that the learning spaces we design should look more and more like “the world”.

We’ll move from making (relatively poor) predictions about future performance based on current proxy indicators, to making (increasingly accurate) inferences of current performance based on quantifiable behaviours.