Emotions and assessment: considerations for rater‐based judgements of entrustment

We identify and discuss three different interpretations of the influence of raters’ emotions during assessments: (i) emotions lead to biased decision making; (ii) emotions contribute random noise to assessment, and (iii) emotions constitute legitimate sources of information that contribute to assessment decisions. We discuss these three interpretations in terms of areas for future research and implications for assessment.

Source: Gomez‐Garibello, C. and Young, M. (2018), Emotions and assessment: considerations for rater‐based judgements of entrustment. Med Educ, 52: 254-262. doi:10.1111/medu.13476

When are we going to stop thinking that assessment – of any kind – is objective? As soon as you’re making a decision (about what question to ask, the mode of response, the weighting of the item, etc.) you’re making a subjective choice about the signal you’re sending to students about what you value. If the student considers you to be a proxy of the profession/institution, then you’re subconsciously signalling the values of the profession/institution.

If you’re interested in the topic of subjectivity in assessment, you may be interested in two of our In Beta episodes:

Objectifying students in the same way we objectify patients

What is it about the way we teach that sets up behaviours in our students for the way that they practice? I noticed that many of our students don’t consciously think of their patients as fully human beings. I don’t think it’s intentional, or that they’re disrespecting the patients…they just don’t think of them as a person in the same way they think of themselves.

I believe that one of the major reasons for this is in the language we use when we talk to, and about patients. One of the ways we see this quite clearly is in the very formal and cold manner that students address patients when first meeting them. My favourite is: “Good morning sir. My name is John. I’ll be your student physio for the day. How may I address you?” Who talks like that? Somewhere in our curriculum, our students are being taught to speak to patients as if they’re a different species. Now, I know what objectivity and emotional distance are, I just don’t believe that they have much value in the clinical context. Healthcare is a deeply emotional space, and by teaching students to avoid that aspect of it we’re cutting them off from establishing any real connection with their patients, which will have a negative impact on patient care. This is also true of teaching. Don’t ignore the emotional context.

Could it be that as physiotherapists we treat patients as objects that are broken, and we bring that kind of thinking into the way we teach? Are we unconsciously treating students as objects that are broken in the sense that they’re not physios, and thereby graduating physios who treat patients as objects that are broken?

Many of my colleagues still talk about the patients they treat as the set of conditions they have (e.g. going to see the stroke in bed 12). We instruct and direct patients to do things, rather than working with them as part of a collaborative team. We do the same thing with students. We “educate” our patients by giving them a set of instructions and then call them non-compliant when they don’t do the home exercise programme because they didn’t understand why the exercises are important. But this the same thing we do with students when we similarly label them “non-compliant” because they didn’t study for the test or prepare for class by doing the readings we instructed them to. But, did we explain to them how preparing for class would help them learn? Did we actually structure the lesson so that the benefit of the preparation was explicit?

I think that sometimes as healthcare practitioners we believe that if we give the right instructions and control the right variables we’ll “fix” our patients. Isn’t this the same thing that we do with students – give them them the right instructions and the right content and then they must do their bit in order to be a good physio. With patients we often ignore the influence of factors like motivation, the capacity to endure discomfort, and emotional state on the qualitative outcome. We do the same thing with students, ignoring these factors as if they don’t have anything to do with learning.

I think we need to be aware that as clinicians we can sometimes make the mistake of thinking of our patients as the objects of our practice and the things upon which we act. We should be aware that this mindset can have implications for our teaching, especially when we think of students as empty receptacles into which our knowledge can be poured. We must also model for students that caring about someone else’s outcomes has real value in both professional and personal domains. We must work on the re-introduction of emotion into the clinical and educational contexts, and then help students learn how to manage their emotional responses to what can sometimes be deeply disturbing patient encounters.

Note: Of course these are generalisations and many colleagues do not treat patients as objects upon which to practice, just as many clinical educators do not treat students as the passive recipients of knowledge and skills.

Results of my Delphi first round

I’ve recently finished the analysis of the first round of the Delphi study that I’m conducting as part of my PhD. The aim of the study is to determine the personal and professional attributes that determine patient outcomes, as well as the challenges faced in clinical education. These results will serve to inform the development of the next round, in which clinical educators will suggest teaching strategies that could be used to develop these attributes, and overcome the challenges.

Participants from the first round had a wide range of clinical, supervision and teaching experience, as well as varied domain expertise. Several themes were identified, which are summarised below.

In terms of the knowledge and skills required of competent and capable therapists, respondents highlighted the following:

  • They must have a wide range of technical and interpersonal skills, as well as a good knowledge base, and be prepared to continually develop in this area.
  • Professionalism, clinical reasoning, critical analysis and understanding were all identified as being important, but responses contained little else to further explain what these concepts mean to them.

In terms of the personal and professional attributes and attitudes that impact on patient care and outcomes, respondents reported:

  • A diverse range of personal values that they believe have relevance in terms of patient care
  • These values were often expressed in terms of a relationship, either between teachers and students, or between students and patients
  • Emotional awareness (of self and others) was highlighted

In terms of the challenges that students face throughout their training:

  • Fear and anxiety, possibly as a result of poor confidence and a lack of knowledge and skills, leading to insecurity, confusion and uncertainty
  • Lack of self-awareness as it relates to their capacity to make effective clinical decisions and reason their way through problems
  • A disconnect between merely “providing a service” and “serving”
  • They lack positive and supportive clinical learning environments, have poor role models and often aren’t given the time necessary to reflect on their experiences
  • The clinical setting is complex and dynamic, a fact that students struggle with, especially when it comes to dealing with complexity and uncertainty inherent in clinical practice
  • Students often “silo” knowledge and skills, and struggle to transfer between different contexts
  • Students struggle with the “hidden culture” of the professional i.e. the language, values and norms that clinicians take for granted

These results are not significantly different from the literature in terms of the professional and personal attributes that healthcare professionals deem to be important for patient outcomes.

The second round of the Delphi is currently underway and will focus on the teaching  strategies that could potentially be used to develop the attitudes and attributes highlighted in the first round.

Twitter Weekly Updates for 2011-01-17

Posted to Diigo 01/13/2011

    • What makes you vulnerable also makes you beautiful
    • We are hardwired for connection, says Brown. Yet all too often, connection — in relationships, in classrooms, etc. — is missing
    • Fear is a big reason why we fail at making connections. We fear many things, but mostly we fear that if we put our true self “out there” for all to see we will expose our self-doubt and our private worries about whether or not we are really “good enough” or worthy of the connection
    • What we may fear most of all is allowing ourselves to be vulnerable, but without vulnerability there can be no true human-to-human connection
    • Where you hesitate and hold back, no connection can be created, and in a deeper sense, this hesitation to allow ourselves to be vulnerable is a source of much dissatisfaction and disharmony in our lives
    • Some of the things that “wholehearted” people have in common, Brown says, is (1) the courage to be imperfect, (2) the compassion to be kind to themselves, and (3) the ability to let go of the idea of who they are “suppose to be.”
    • The ability to allow ourselves to be imperfect and vulnerable in our personal and professional relationships is the very thing that can open our world up to the possibility of deeper connections and more meaningful engagement with others
    • will we jump on and embrace change and see where our passion will take us, or will we cling cautiously to the past and to that which is known and safe?
    • Passion dies in an environment of fear and a yearning for guarantees and certainty
    • “Follow your heart” sounds trite and cliche perhaps — but following your heart is exactly what you’ve got to do — this is where connection and meaning live
    • How do you want to change the world?
    • What’s your contribution?
    • The attachment to the fear and doubt keeps us from making our best contribution, or even from truly loving another or being loved

Posted to Diigo 12/10/2010

    • The zone of proximal development is the area between what an individual can achieve on their own and what they can achieve with assistance
    • A student should constantly be reaching slightly beyond their capabilities rather than working within them
    • students should lead their learning and teachers simply assist and rather than judging students on what they know in standardised tests, learning should be done through looking closely at their zone of proximal development
    • If informal learning is as important as formal learning, then varying the way students are assessed can only work in their favour
    • Relevant, meaningful activities that both engage students emotionally and connect with what they already know are what help build neural connections and long-term memory storage
    • it’s necessary for learners to attach a new piece of information to an old one
    • If a student acquires new information that’s unrelated to anything already stored in his brain, it’s tough for the new information to get into those networks because it has no scaffolding to cling to
    • a solid amount of research also links personal relevance and emotional engagement to memory storage
    • “the learner’s emotional reaction to the outcome of his efforts … shapes his future behavior,”
    • if [a student] doesn’t believe a particular activity is interesting, relevant, or within the scope of his capabilities, it’s probably not going to sink in
    • too much emotion can be as detrimental to learning as too little: distractions and stress can also block receptivity to new ideas
    • Make it student directed. Give students a choice of assignments on a particular topic, or ask them to design one of their own. “When students are involved in designing the lesson, they better understand the goal…and become more emotionally invested in and attached to the learning outcomes.”
    • Connect it to their lives and what they already know. Taking the time to brainstorm about what students already know and would like to learn about a topic helps them to create goals — and helps teachers see the best points of departure for new ideas. Making cross-curricular connections also helps solidify those neural loops
    • With no reference point and no intrigue, information is fairly likely to go in one ear and straight out the other
    • Happy learners are healthy learners, if students do not feel comfortable in a classroom setting, they will not learn. Physiologically speaking, stressed brains are not able to form the necessary neural connections
    • The amygdala, for instance, processes emotions, stores the memories of emotional reactions, and reacts so aggressively to stress that it will physically prevent information from reaching the centers of the brain necessary for absorbing new knowledge
    • Even feelings like embarrassment, boredom, or frustration — not only fear — can spur the brain to enter the proverbial “fight or flight” mode
    • The amygdala goes into overdrive and gets in the way of the parts of the brain that can store memories
    • it makes sense — on many levels — to cultivate the learning atmosphere as much as the learning itself. “Reducing stress and establishing a positive emotional climate in the classroom is arguably the most essential component of teaching,”
    • Make the classroom stress free. Lighten the mood by making jokes and spurring curiosity; create a welcoming and consistent environment; give students frequent opportunities to ask questions and engage in discussions without judgment; and determine achievable challenges for each learner
    • Encourage participation, not perfection. A classroom in which mistakes are encouraged is a positive learning environment, both neurologically and socially speaking
    • “Students will allow themselves to experience failure only if they can do so within an atmosphere of trust and respect.”
    • This kind of positive reinforcement from the get-go allows students to let their guard down (known in neuro-speak as calming their “affective filters”). Listening to students in general, and listening to their intentions in particular, can help relax anxious brains.
    • Practice active listening. “Focus on what students are trying to say
    • Intelligence is not fixed, it turns out, nor planted firmly in our brains from birth. Rather, it’s forming and developing throughout our lives
    • neuroplasticity is defined as the selective organizing of connections between neurons in our brains
    • neuroplasticity is defined as the selective organizing of connections between neurons in our brain
    • “cells that fire together, wire together”
    • “Practice makes permanent. The more times the network is stimulated, the stronger and more efficient it becomes.”
    • both morale and grade points increase when students understand the idea that intelligence is malleable
    • Practice, practice, practice. Repeating an activity, retrieving a memory, and reviewing material in a variety of ways helps build thicker, stronger, more hard-wired connections in the brain
    • Put information in context. Recognizing that learning is, essentially, the formation of new or stronger neural connections, it makes sense to prioritize activities that help students tap into already-existing pathways
    • “Whenever new material is presented in such a way that students see relationships between concepts, they generate greater brain cell activity and achieve more successful long-term memory storage and retrieval.”
    • Let students know that this is how the brain works. “Especially for students who believe they are ‘not smart,’ the realization that they can literally change their brains through study and review is empowering.”

SAFRI: conflict resolution

Someone told me that the SAFRI programme had changed their life, and I remember thinking that that might be taking it a bit far. But today brought me closer to thinking that it might not be that far from the truth. It wasn’t so much the content of the session, but the reflection and discussion that happened as a result of an exercise on conflict management. During the session, I was able to confront a part of me that isn’t the rational, logical person I usually think I am, and gave me a greater appreciation for the poor souls who have to try and understand why I think and do things so differently to them.

We spent a lot of time talking about the different approaches to managing conflict, with people who share similar psychological attributes identifying with certain approaches. I realised that I have ways of dealing with difficult situations that aren’t shared by most other people (I was the only person in my group, besides the facilitator, as opposed to 3 other groups of almost 10 in each group). My MBTI type is:

  • Introvert – draw energy by looking internally, prefer reflection over action, prefer written communication
  • iNtuitive – prefer theory and abstraction, imaginative, desire change
  • Thinker – use logic and objectivity to make decisions, remain detached, truthful rather than tactful
  • Perceiver – remain open and adapt to new information, be flexible, enjoy surprises, routines are limiting

The exercise I got the most out of today was to analyse a conflict and reflect on my own responses, as well as how I respond to the responses of others. Here’s the short reflection I put together after a few minutes of discussion with the facilitator:

I approach conflict logically, which is good for mediating the conflict of 3rd parties, but not so good when I’m personally involved. While other personality types might avoid conflict, I will sometimes create it by playing devil’s advocate. I’ll probe and push buttons to get a reaction and will sometimes take an opposing viewpoint just to have an interesting discussion (I’ll also not understand when the other party doesn’t appreciate this attempt to engage with them).

When I am involved in a conflict, I experience a rapid escalation of my own emotional response if I feel that those emotions aren’t being acknowledged, yet I have no natural tendency to acknowledge the emotions of other’s (“I’m right, so you must be wrong”). If my emotions are not acknowledged, I tend to withdraw and switch off emotionally. In those cases I find it difficult to let go and will definitely refuse to acknowledge the other person’s emotion…as a form of retribution (when I write it down like this, it seems insane, but in the moment, it’ perfectly clear to me).

On the other hand, if my emotions are acknowledged, there is a complete collapse of my resistance and I’m able to move towards resolution. However, I struggle to close the issue and will often find myself prolonging an argument to make the point that “I’m right”. When I do manage to avoid that and the conflict is resolved, I forget about it in minutes.

This experience,  and the wonderful conversations it generated afterwards, really gave me a greater insight into who I am, as well as how I relate to others. For the rest of the day I was acutely aware that almost everyone else in the room sees, and responds to the world differently to me, which I found both sobering (“I’m alone”) and inspiring (“I’m special”).

Predictably irrational: decision-making in teaching

I’m busy reading Dan Ariely’s Predictably Irrational: The Hidden Forces that Shape Our Decisions, and wanted to share some of the takeaway points that made me think about how I could change my own teaching practice. I haven’t finished the book yet, so I may update this post when I do.

The premise of the book is that we aren’t always the rational beings we believe we are, and that there are powerful emotional factors that cause us to make decisions that are often counterintuitive. If we understand how these factors predictably make us less rational, we might be able to affect greater self-control over our lives, and be better off for it.

One of the ideas that really got my attention was how we respond to social and market norms in our everyday interactions with other people. When you think about it, a lot of what we do as teachers has nothing to do with market norms i.e. we don’t work the hours we do because we’re paid appropriately. Social norms mean that we go above and beyond what is required of us possibly because we have a sense of shared purpose or a belief that we’re contributing to something more important than money. In other words, people are motivated to work harder when they believe they’re in a socially-orientated relationship, rather than one in which market values dominate. Ariely also conducted experiments showing that when market and social norms collide, relationships that were based on the social norms are disrupted and can take years to rebuild. This has implications when we start thinking about building communities of practice in our professional domains, and it seems that we would do well to base our interactions on a shared sense of purpose, rather than financial reward. I know from recent conversations with students with whom I have a good relationship, that they try harder to impress me with their work, and worry less about the mark they receive, than they do with other lecturers who don’t engage with them at all. For me, this is a powerful incentive to engage with students not only on a cognitive level, but on a social level as well.

Ariely also shows clearly how emotionally heightened states cause us to make bad decisions for ourselves and for those around us. How many times have we made a bad decision when we’re angry? When I think about it (and if I’m honest with myself), I know that I’ve been guilty of being a stricter assessor when I’m in a bad mood, than when I’m having a good day. I know that my marking isn’t as objective as I’d like it to be, but to be shown the evidence of how much it influences my behaviour has made me commit to avoid marking students’ work when I’m upset.

When discussing procrastination, Ariely makes the observation that when students are given absolute submission dates for assignments that are appropriately spaced, they do better than students who are given flexibility in determining their own submission dates. I know that recently I’ve started including draft deadlines into assignments to “encourage” students to begin work their on assignments early, and to continue improving their work up until the final submission date. Last year I saw students who participated in the drafting process score significantly higher marks than those who chose to submit only one, final version of the assignment. Students will procrastinate if left to themselves, and I guess we need to decide if we’re OK with that, or to rather try and figure out how to more effectively guide them through the process of making regular improvement through regular feedback.

The final point I wanted to highlight is from a TED presentation that Ariely gave (although it might be in the book too), where he finds that students are less likely to cheat after thinking about the 10 commandments. It turns out that signing an honour code might not be as pointless as I’d previously thought.

You can also see Ariely discuss some of his ideas at these 2 TED talks: