Emotions and assessment: considerations for rater‐based judgements of entrustment

We identify and discuss three different interpretations of the influence of raters’ emotions during assessments: (i) emotions lead to biased decision making; (ii) emotions contribute random noise to assessment, and (iii) emotions constitute legitimate sources of information that contribute to assessment decisions. We discuss these three interpretations in terms of areas for future research and implications for assessment.

Source: Gomez‐Garibello, C. and Young, M. (2018), Emotions and assessment: considerations for rater‐based judgements of entrustment. Med Educ, 52: 254-262. doi:10.1111/medu.13476

When are we going to stop thinking that assessment – of any kind – is objective? As soon as you’re making a decision (about what question to ask, the mode of response, the weighting of the item, etc.) you’re making a subjective choice about the signal you’re sending to students about what you value. If the student considers you to be a proxy of the profession/institution, then you’re subconsciously signalling the values of the profession/institution.

If you’re interested in the topic of subjectivity in assessment, you may be interested in two of our In Beta episodes:

Using online multimedia to teach practical skills

During 2016 I supervised an undergraduate research group in my department and we looked at the possibility of using multimedia – including video, images and text – to teach students practical skills. Traditionally, we teach these skills by having the lecturer demonstrate the technique on a model while the class watches. Students then break into small groups to practice while the lecturer moves around class, giving feedback, correcting positions and answering questions.

This process was pretty much the only option for as long as we’ve been teaching practical techniques, but it has it’s disadvantages:

  • As class sizes have grown, it’s increasingly difficult for every student to get a good view of the technique. Imagine 60 students crowded around a plinth trying to see what the lecturer is demonstrating.
  • Each student only gets one perspective of the technique. If you’re standing at the head of the module (maybe 1 or two rows back) and the demonstration is happening at the feet, you’re not going to get any other angle.
  • There are only so many times that the technique will be demonstrated before students need to begin practising. If you’re lucky the lecturer will come around to your station and offer a few more demonstrations, but owing to the class size, this isn’t always the case.

We decided that we’d try and teach a practical technique to half the class using only a webpage. The page included two videos of the technique, step by step instructions and images. We randomly selected half the class to go through the normal process of observing the lecturer demonstrate the technique and half the class were taken to another venue,  given the URL of the webpage and asked to practice among themselves. Two weeks later we tested the students using an OSCE. Students were evaluated by two researchers using a scoring rubric developed by the lecturer, where both assessors were blinded to which students had learned the technique using the webpage.

We found that the students who only had access to the multimedia and no input from the lecturer performed better in the OSCE than the students who had observed the lecturer. This wasn’t very surprising when you consider the many advantages that video has over face-to-face demonstration (rewind, pause, watch later, etc.) but nonetheless caused a stir in the department when the students presented their findings. We had to be careful how we framed the findings so as not to suggest that this could be considered as a replacement but rather as a complement to the traditional approach.

There were several limitations to the study:

  • The sample size was very small (only 9 students from the “multimedia” class took the OSCE, as it was voluntary)
  • We have no idea whether students in the multimedia class asked students from the “traditional” class to demonstrate the technique for them
  • We only taught and tested one technique, and it wasn’t a complex technique
  • Students knew that we were doing some research and that this was a low stakes situation (i.e. they may not have paid much attention in either class since they knew it would not affect their final grades)

Even taking the above into consideration though, in principle I’m comfortable saying that the use of video, text and images to teach undergraduate students uncomplicated practical techniques is a reasonable approach. Instead of being defensive and worrying about being replaced by a video, lecturers could see this as an opportunity to move tedious, repetitive tasks outside the classroom, freeing up time in the classroom for more meaningful discussion; Why this technique and not this one? Why now? At what level? For which patients? It seems to me that the more simple, content-based work we can move out of the classroom, the more time we have with students to engage in deeper work. Wouldn’t that be a good thing?

Public posting of marks

My university has a policy where the marks for each assessment task are posted – anonymously – on the departmental notice board. I think it goes back to a time when students were not automatically notified by email and individual notifications of grades would have been too time consuming. Now that our students get their marks as soon as they are captured in the system, I asked myself why we still bother to post the marks publicly.

I can’t think of a single reason why we should. What is the benefit of posting a list of marks where students are ranked against how others performed in the assessment? It has no value – as far as I can tell – for learning. No value for self-esteem (unless you’re performing in the higher percentile). No value for the institution or teacher. So why do we still do it?

I conducted a short poll among my final year ethics students asking them if they wanted me to continue posting their marks in public. See below for their responses.

selection_001

Moving forward, I will no longer post my students marks in public nor will I publish class averages, unless specifically requested to do so. If I’m going to say that I’m assessing students against a set of criteria rather than against each other, I need to have my practice mirror this. How are students supposed to develop empathy when we constantly remind them that they’re in competition with each other?

Interrogating the mistakes

We tend to focus our attention on the things that students got right. This seems perfectly appropriate at first glance because we want to celebrate what they know. Their grades are reported in such a way as to highlight the number of questions answered correctly. The cut score (pass mark) is set based on what we (often arbitrarily) decide a reasonably competent student should know (there is no basis for setting 50% as the cut score, but that’s for another post). The emphasis is always on what is known rather than what is not known.

But if you think about it getting the right answer is a bit of a dead end as far as learning is concerned. There’s nowhere to go from there. But the wrong answer opens up a whole world of possibility. If the capacity to learn and move forward sits in the spaces taken up by faulty reasoning shouldn’t we pay more attention to the errors that students make? The mistakes give us a starting point from which to proceed with learning.

What if we changed our emphasis in the curriculum to focus attention on the things that students don’t understand? Instead of celebrating the points they scored for getting the right answer could we pay closer attention to the areas where they lost marks? And not in a negative way that makes students feel inferior or stupid. I’m talking about actually celebrating the wrong answers because it gives us a starting point and a direction to move. “You got that wrong. Great! Let’s talk about it. What was the first thing you thought when you read the question? Why did you say that? Did you consider this other option? What is the logical end point of the reasoning you used? Do you see now how your answer can’t be correct?” Imagine a conversation going like that. Imagine what it would mean for students’ ability to reflect on their thinking and practice.

We might end up with some powerful shared learning experiences as we get into students’ heads as we try to understand what and how they think. The faulty reasoning that got them to the wrong answer is way more interesting than the correct reasoning that got them to the right answer. A focus on the mistakes that they make would actually help improve students ability to learn in the future because you’d be helping to correct their faulty reasoning.

But we don’t do this. We focus on counting up the the right answers and celebrating them, which means that we deflect attention from the wrong answers. We make implicit the idea that getting the right answer is important and the getting the wrong answers are bad. But learning only happens when we interrogate the faulty reasoning that got us to the wrong answer.

How my students do case studies in clinical practice

Our students do small case studies as part of their clinical practice rotations. The basic idea is that they need to identify a problem with their own practice; something that they want to improve. They describe the problem in the context of a case study which gives them a framework to approach the problem like a research project. In this post I’ll talk about the process we use for designing, implementing, drafting and grading these case studies.

There are a few things that I consider to be novel in the following approach:

  1. The case studies are about improving future clinical practice, and as such are linked to students’ practices i.e. what they do and how they think
  2. Students are the case study participants i.e. they are conducting research on themselves
  3. We shift the emphasis away from a narrow definition of “The Evidence” (i.e. journal articles) and encourage students to get creative ideas from other areas of practice
  4. The grading process has features that develop students’ knowledge and skills beyond “Conducting case study research in a clinical practice module”

Design

Early on in their clinical practice rotations, the students identify an aspect of that block that they want to learn more about. We discuss the kinds of questions they want to answer, both in class and by email. Once the topic and question are agreed, they do mini “literature” reviews (3-5 sources that may include academic journals, blogs, YouTube videos, Pinterest boards…whatever) to explore the problem as described by others. They also use the literature to identify possible solutions to their problems, which then get incorporated into the Method. They must also identify what “data” they will use to determine an improvement in their performance. They can use anything from personal reflections to grades to perceived level of comfort…anything that allows them to somehow say that their practice is getting better.

Implementation and drafting of early case studies

Then they try an intervention – on themselves, because this is about improving their own practice – and gather data to analyse as part of describing a change in practice or thinking.  They must also try to develop a general principle from the case study that they can apply to other clinical contexts. I give feedback on the initial questions and comment on early drafts to guide the projects and also give them the rubric that will be used to grade their work.

Examples of case studies from last semester include:

  • Exploring the impact of meditation and breathing techniques to lower stress before and during clinical exams, using heart rate as a proxy for stress – and learning that taking a moment to breathe can help with feeling more relaxed during an exam.
  • The challenges of communicating with a patient who has expressive aphasia – and learning that the commonly suggested alternatives are often 1) very slow, 2) frustrating, and 3) not very effective.
  • Testing their own visual estimation of ROM against a smartphone app – and learning that visual estimation is (surprise) pretty poor.
  • Exploring the impact of speaking to a patient in their own language on developing rapport – and learning that spending 30 minutes every day learning a few new Xhosa words made a huge difference to how likely the patient was to agree to physio.

Submission and peer grading

Students submit hard copies to me so that I can make sure all submissions are in. Then I take the hard copies to class and randomly assign 1 case study to each student. They pair up (Reviewer 1 and 2) and we go through the case studies together, using the rubric as a guide. I think out loud about each section of the rubric, explaining what I’m looking for in each section and why it’s important for clinical practice. For example, if we’re looking at the “Language” section I explain why clarity of expression is important for describing clinical presentations, and why conciseness allows them to practice conveying complex ideas quickly (useful for ward rounds and meetings). Spelling and grammar are important, as is legibility, to ensure that your work is clearly understandable to others in the team. I go through these rationales while the students are marking and giving feedback on the case studies in front of them.

Then they swap case studies and fill out another rubric for the case study that their team member has just completed. We go through the process again, and I encourage them to look for additional places to comment on the case study. Once that’s done they compare their rubrics for the two case studies in their team, explaining why certain marks and comments were given for certain sections. They don’t have to agree on the exact mark but they do have to come to consensus over whether each section of the work is “Poor”, “Satisfactory” or “Good”. Then they average their marks and submit it to me again.

I take all the case studies with their 2 sets of comments (on the rubric) and feedback (on the case study itself) and I go through them all myself. This means I can focus on more abstract feedback (e.g. appropriateness of the question, analysis, ethics, etc.) because the students have already commented on much of the structural, grammatical and content-related issues.

Outcomes of the process

For me, the following outcomes of the process are important to note:

  1. Students learn how to identify an area of their own clinical practice that they want to improve. It’s not us telling them what they’re doing wrong. If we want lifelong learning to happen, our students must know how to identify areas for improvement.
  2. They take definite steps towards achieving those improvements because the case study requires them to implement an intervention. “Learning” becomes synonymous with “doing” i.e. they must take concrete steps towards addressing the problem they identified.
  3. Students develop the skills they need to find answers to questions they have about their own practice. Students learn how to regulate their own learning.
  4. Each student gets 3 sets of feedback on their case study. It’s not just me – the external “expert” – telling them how to improve, it’s their peers as well.
  5. Students get exposed to a variety of other case studies across a spectrum of quality. The peer reviewers need to know what a “good” case study looks like in order to grade one. They learn what their next case study should look like.
  6. The marking time for 54 case studies goes down from about 10 hours (I give a lot of feedback) to about 3 hours. I don’t have to give feedback on everything because almost all of the common errors are already identified and highlighted.
  7. Students learn how I think when I’m marking their work, which helps them to make different choices for the next case study. This process allows them access to how I think about case study research in clinical practice, which means they are more likely to improve their next submission, knowing what I’m looking for.

In terms of the reliability of the peer marking and feedback, I noted the following when I reviewed the peer feedback and grades from earlier in the year:

  • 15 (28%) students’ marks went up when I compared my mark with the peer average, 7 (13%) students’ marks went up by 5% or more, and 4 (7%) students went from “Fail” to “Pass”.
  • 7 (13%) students’ marks went down, 3 (6%) by 5% or more, and 0 students went from “Pass” to “Fail”.
  • 28 (52%) students’ marks stayed the same.

The points I take from the above is that it’s really important for me to review the marks and that I have a tendency to be more lenient with marking; more students had mark increases and only 3 students’ marks went down by what I would consider a significant amount. And finally, more than half the students didn’t get a mark change at all, which is pretty good when you think about it.

 

 

Towards a competency-based curriculum in physiotherapy

I’ve been thinking about the concept of competency based education (CBE) in relation to the altPhysio series that I’m busy with. I’m drawn to the idea of CBE but am aware that there are some criticisms against it, especially from a theoretical and pedagogical perspective. This post is a short note to clarify some of my thinking around CBE.

I started with Frank et al. (2010) Toward a definition of competency-based education in medicine: a systematic review of published definitions to get a bit of an idea about how others think about CBE and to have a working definition of the concept. From the article:

We identified 4 major themes (organizing framework, rationale, contrast with time, and implementing CBE) and 6 sub-themes (outcomes defined, curriculum of competencies, demonstrable, assessment, learner-centred and societal needs)….From this research we have developed a proposed 21st century definition of CBE for medical education, namely:

Competency-based education (CBE) is an approach to preparing physicians for practice that is fundamentally oriented to graduate outcome abilities and organized around competencies derived from an analysis of societal and patient needs. It de-emphasizes time-based training and promises greater accountability, flexibility, and learner-centredness.

I quite like this definition of CBE and think that it addresses some of the concerns that are commonly levelled against a CBE approach. This is far from a foregone conclusion and there are still many contrasting points of view. But for my purposes this was a good place to start, especially since I’m looking at a physiotherapy curriculum, which has a significant emphasis on clinical performance, as opposed to another programme that emphasises different outcomes.

I’m obviously also interested in the use of technology, and Tony Bates’ The strengths and weaknesses of competency-based learning in a digital age was useful in this regard. From the post:

Competency-based learning is a relatively new approach to learning design which is proving increasingly popular with employers and suits certain kinds of learners such as adult learners seeking to re-skill or searching for mid-level jobs requiring relatively easily identifiable skills. It does not suit though all kinds of learners and may be limited in developing the higher level, more abstract knowledge and skills requiring creativity, high-level problem-solving and decision-making and critical thinking.

I’m not sure that I agree with the last bit; it may be limited in developing higher level, more abstract knowledge and skills like problem solving, decision making and critical thinking. I think that CBE does not inherently preclude the possibility of developing these skills. The fact that it may not doesn’t mean that it can’t (IMO).

Then there’s the CanMEDS framework, which is another piece of work that I’m a big fan of. Without going into the detail CanMEDS is a way of thinking about the different roles that a physician must demonstrate proficiency in. Again, this framework seems to be a great place to start when looking at a CBE curriculum.

canmeds-2015-diagram-e

So that’s how far I’ve gotten with looking at CBE as a possible basis for a physiotherapy curriculum. If you know of any physiotherapy curricula that are currently competency-based, or are aware of any other resources that you think would be good to read, I’d really like to hear from you.

Note: The featured image of this post is a map of the London underground that I wasn’t familiar with. The point I was trying to make is that there are many different ways of getting to the same end point, and it might be useful to allow people to take the route that most suits them.

How do we choose what to assess?

Assessing content (facts) for the sake of it – for the most part – is a useless activity because it tells us almost nothing about how students can use the facts to achieve meaningful objectives. On the other hand, how do you assess students’ ability to apply what they’ve learned? The first is easy (i.e. assessing content and recall), while the second is very difficult (i.e. assessing how students work with ideas). If we’re honest with ourselves, we have a tendency to assess what is easy to assess, rather than what we should assess.

You can argue that your assessment is valid i.e. that you are, in fact, assessing what you say you’re assessing. However, even if the assessment is valid, it may not be appropriate. In other words, your assessment tasks might match your learning outcomes (i.e. they are valid) but are you questioning your outcomes to make sure that they’re the right outcomes?

Are we assessing the things that matter?

Where does the path of least resistance lead?

Human beings are psychologically predisposed to do the easiest thing because thinking is hard and energy intensive. We are geared through evolution to take short cuts in our decision making and there is little that we can do to overcome this natural predisposition to take the path of least resistance (see System 1 and System 2 thinking patterns in Kahneman, 2011). The problem with learning is that the easy choice is often the least effective. In order to get students to do the hard work – overcome the resistance – we should encourage them to strive towards a higher purpose in their learning, as opposed to simply aiming for a pass. Students – and lecturers for that matter – almost always default to the path of least resistance unless they have a higher purpose that they are working towards. If we want students to achieve at high levels, then the path of least resistance must lead to failure to complete the task. Making the easy choice must lead to poorer outcomes than doing the hard work, but so often students can pass without doing the hard work. We must therefore create tasks that are very difficult to pass without doing hard cognitive work.

Kahneman, D. (2011). Thinking Fast and Slow.

Providing students with audio feedback

I’ve started providing my students with audio feedback on a set of about 60 clinical case studies that they recently submitted. I was depressed at the thought of having to write out my feedback; I tend to provide a lot of detail because I almost always try to provide a rationale for the comments I’ve made. I want the students to understand why I’m suggesting the changes, which can be really time consuming when I have a lot of documents.

This semester I decided to try audio feedback (Cavanaugh & Song, 2010) as a method of providing input on the students’ drafts and I have to say, it’s been fantastic. I take about the same amount of time per document (10 – 15 minutes) because I find that I give a more detail in my spoken feedback, compared to the written feedback, so this is not about saving time. I realised that when I write / type comments there are some points I don’t make because in order to explain the reason for the comment would take more space than the margin allows.

In addition, I’ve found that I use a more conversational tone – which the students really appreciate – and because I’m actually speaking to the student, I pay less attention to line items e.g. spelling corrections and punctuation issues. In other words, I give more global comments instead of local comments, and obviously don’t use Track Changes. As I mentioned earlier, I provide more detail, explaining the reasons behind certain points I make, going into the reasons for why it’s important that they address the comment.

Students’ have given me feedback on this process and 100% of those who responded to my request for comment have suggested that this method of receiving feedback is preferable for them. One of them reported that hearing my comments on his draft allowed him to “hear how I think”. This comment reminded me of the thinking aloud protocol, which is a way for experts to model thinking practices to novices (Durning et al. 2014). This insight led to a slight change in how I structured the feedback, where I now “think” my way through the piece, even pausing to tell the student that I’m struggling to put into words an impression or feeling that I experienced while reading. I try to make it as “real time” as possible, imagining that I’m speaking to the student directly.

I record to .mp3 at a sample rate of 44 K/Hz and a bit rate of 128 kbit/s, which offers decent audio quality at a low enough file size to make emailing feasible. This is my basic process for recording audio feedback:

  1. Read through the entire document, making mental notes of the most important points I want to make
  2. Go back to the beginning of the document and start the recorder
  3. Greet the student and identify the piece I’m commenting on
  4. Provide an overview of my thoughts on the document in it’s entirety (structure, headings, logical flow, etc.)
  5. Work through the different sections of the document (e.g. Introduction, Method, Results, etc.), providing more detailed thoughts on the section, pausing the recorder between sections to give myself time to identify specific points I want to make
  6. End with a summing up of what was done well and the 3-5 major points that need to be addressed
  7. Stop the recorder, rename the audio file (student name – course code – title) and email it to the student

Reference

Abstract for RCTs in educational research

There seems to have been a resurgence in calls for the use of systematic reviews and randomised controlled trials in educational research lately. There’s a lot to like (in my opinion) about RCTs in certain contexts because of how they are designed.  For example, when you want to figure out the effect of variable A on variable B, it’s a very useful approach because of the randomisation of the sample and the blinding of assessors and participants.

However, the method doesn’t translate well into most educational contexts for a variety of reasons, usually in the form of arguments for how RCTs in educational research are unethical and logistically difficult. I recently wrote a position paper with a colleague from Rhodes University where we looked at the argument against RCTs where we basically ignore the arguments just mentioned. We focus instead on how using an RCT pre-supposes an understanding of teaching and learning that is at odds with what we know about how learning happens. The article will be published soon in the African Journal of Health Professions Education. Here’s the abstract:

Randomised controlled trails (RCTs) are a valued research method in evidence-based practice in the medical and clinical settings. However, the use of RCTs is associated with a particular ontological and epistemological perspective that is situated within a positivist world view. It assumes that environments and variables can be controlled in order to establish cause-effect relationships. But current theories of learning suggest that knowledge is socially constructed, and that learning occurs in open systems which cannot be controlled and manipulated as would be required in a RCT. They recognise the importance and influence of context on learning, something that positivist research paradigms specifically aim to counter. We argue that RCTs are inappropriate in education research because they force us to take up ontological and epistemological positions within a technical rationalist framework that is at odds with current learning theory.