@GoodTasteMag loved the rib eye steak Service was fantastic, really good experience #

@ShanLatimer sitting outside at #1800 in the middle of winter at the #caperoyale Cape Town is fantastic #

Share

@rochellesa Everyone needs some downtime, especially at 10 at night when you’re out with your wife Seems like a nice guy, very quiet #

Share

Validity

Important for assessment, not only for research

It’s the scores that are valid and reliable, not the instrument

Sometimes the whole is greater than the sum of the parts e.g. when a student gets all the check marks but doesn’t perform competently overall e.g. the examiner can tick each competency being assessed but the student doesn’t establish rapport with the patient. Difficult to address

What does the score mean?

Students are efficient in the use of their time i.e. they will study what is being assessed because the inference is that we’re assessing what is important

Validity can be framed as an “argument / defense” proposition

Our Ethics exam is a problem of validity. Written tests measure knowledge, not behaviour e.g. students can know and report exactly what informed consent is and how to go about getting it, but may not pay it any attention in practice. How do we make the Ethics assessment more valid?

“Face” validity doesn’t exist, it’s more accurately termed “content” validity. “Face” validity basically amounts to saying that something looks OK

What are the important things to score? Who determines what is important?

There are some things that standardised patients can’t do well e.g. trauma

Assessment should sample more broadly from a domain. This improves validity and also students don’t feel like they’ve wasted their time studying things that aren’t assessed. The more assessment items we include, the more valid the results

Scores drop if timing of assessment is inappropriate e.g. too much or too little time → lower scores as students either rush or try to “fill” the time something that isn’t appropriate for the assessment

First round scores in OSCEs are often lower then later rounds

Even though the assessment is meant to indicate competence, there’s actually no way to predict if practitioners are actually competent

Students really do want to learn!

Reliability

We want to ensure that a students observed score is a reasonable reflection of their “true ability”

In reliability assessments, how do you reduce the learning that occurs between assessments?

In OSCEs, use as many cases / stations as you can, and have different assessor for each station. This is the most effective rating design

We did a long session on standard setting, which was fascinating especially when it came to having to defend the cut-scores of exams i.e. what criteria do we use to say that 50% (or 60 or 70) is the pass mark? What data do we have to defend that standard?

Didn’t even realise that this was something to be considered, good to know that methods exist to use data to substantiate decisions made with regards to standards that are set (e.g. Angoff Method)

Should students be able to compensate for poor scores in one area, with good scores in another. Should they have to pass every section that we identify as being important? If it’s not important, why is it being assessed?

Norm-referenced critera are not particularly useful to determine competence. Standards should be set according to competence, not according to the performance of others

Standard setting panels shouldn’t give input on the quality of the assessment items

You can use standard setting to lower the pass mark in a difficult assessment, and to raise the pass mark in an easier exam

Alignment of expectations with actual performance

Setting up an OSCE

• Design
• Evaluate
• Logistics

Standardised, compartmentalised (i.e. not holistic), variables removed / controlled, predetermined standards, variety of methods

Competencies broken into components

Is at the “shows how” part of Miller’s pyramd (Miller, 1990, The assessment of clinical skills, Academic Medicine, 65; S63-S67)

Design an OSCE, using the following guidelines:

• Summative assessment for undergraduate students
• Communication skill
• Objective
• Instructions (student, examiner, standardised patient)
• Score sheet
• Equipment list

Criticise the OSCE stations of another group

Assessing clinical performance

Looked at using mini-CEX (clinical evaluation exercise)

Useful for formative assessment

Avoid making judgements too soon → your impression may change over time

Share

What is assessment?

“Defines for students what is important, what counts, how they will spend their times and how they will see themselves as learners. If you want to change student learning, then change methods of assessment” (Brown, Bull and Pendlebury)

Integrate assessment into teaching and learning in a way that includes students in the process

Feedback is an important part of assessment

• Should be frequent
• Must identify gaps and provide direction to help students close the gap
• Stimulates deep learning
• Feedback should be task-centred, not emotional / personal

Assessment linked to outcomes i.e. are we assessing what we say is important for our graduates to be able to do (alignment)

Students can be involved in self-assessment and peer feedback but it needs to be scaffolded / structured. Should be used for different reasons

Personal and professional development. Do we assess for good physio’s or good people? Peer assessment can be used to encourage formation of professional behaviour and interpersonal dimensions

Move away from a testing culture to an assessment culture i.e. away from an emphasis on procedures and products of assessment to emphasis a process of assessment → deep learning. Difference between “knowing what” and “knowing how”

Readings:

• Assessment practices that improve teaching and learning (Luckett & Sutherland)
• Assessment in medical education (Epstein)

Feedback as an educational tool

Start with the positive, then move on to negative. Also, feedback doesn’t have to even include a negative. We can also use it to highlight good work / understanding. As long as it is always given with the intention of moving the student forward. I would argue that splitting feedback into “positive” and “negative” might have little value anyway. Students will often latch onto the “negative” and forget anything positive you opened with. Can feedback be neutral, aiming only to highlight how the student can move forward?

Feedback should always be aimed at “closing the gap” between what is currently known and where the student should be i.e. it should always be formative

It should not be personal or have a value judgement assigned. The purpose is to identify a problem with a behaviour / skills / competency /etc. not with the person

Identify what they do know, what they don’t know and how to improve

Use feedback as an opportunity for self-assessment e.g. How do you feel about that? What do you think you could have done better? What did you do well?

Needs to be given as soon as is appropriate. Some feedback may be better given in private, or in a group, depending on the context

Be kind / sensitive / aware / empathic

Acknowledge if you are at fault e.g. arriving late for a feedback session / tutorial

Be aware of power relationships

Feedback should be continuous

Follow up on your feedback

The feedback should be written down at some stage, either by the student or teacher. Verbal feedback given in the moment may not be remembered later

It’s not only important to identify what students got wrong, but also to try and determine why they got it wrong

Curriculum development

Whose needs does the curriculum address? Ultimately, the community who will be served by the graduates of your course → the aims & objectives of the course must be aligned with community needs

Harden’s 10 Question model for curriculum design

“How is the detail of the curriculum communicated”, especially to first year students? Are students aware of how what they’re doing right now is relevant to the course they signed up for e.g. why am I studying physics, I want to be a sports physio?

“Blueprinting”

SPICES model of educational strategies:

• Student-centred ↔ Teacher centred
• Problem-based ↔ Information gathering
• Integrated ↔ Discipline-based
• Community-based ↔ Hospital-based
• Electives ↔ Standard programme
• Systematic ↔ Apprenticeship-based or Opportunistic

There are models that can be used to determine where a curriculum lies on each of the spectrums listed above

Is a curriculum set in stone, or is it dynamic?

“Diseases of the curriculum” (Abrahamson, 1960s)

You can’t teach everything (there’s just too much) but you can help students become self-directed learners, which is what clinicians are. We don’t have all the answers but we know how to find the answers that will help fill the gaps in our knowledge / understanding. If students think they’re supposed to know all the answers, then that’s an enormous burden to carry

How do we select students who had a poor secondary education but who have the potential to be good (great?) therapists / clinicians? How do you support them?

You should always look for competence in students, no matter what curriculum design you use

A discipline-based approach doesn’t challenge students to engage with the later stages of Bloom’s taxonomy

Problem-based learning helps to create a more authentic learning environment / experience and allows the integration of pre-clinical and clinical science

“The contribution of South African curricula to prepare health professionals for working in rural or under-served areas in South Africa: a peer review evaluation. SAMJ, 2011, 101:34-38

What happens to students when they leave? Should this be a question we consider during curriculum planning?

Learning:

• Informative
• Formative
• Transformative

“If you can’t explain it simply, you don’t understand it well enough” – Einstein

Share

Achievement: changed the way I think about the world (word = clinical education)

Challenge: helping others to see the world the way I do

Never give up…or Give up often? Come up with lots of ideas, some will be good, some not so good, some terrible. Test them all (even if only mentally) and throw away the ones that don’t stand up to being tested. Analogy with digital cameras and taking loads of photos because the cost is zero and you can delete the poor ones.

Find the underlying principle that can be generalised to many contexts i.e. details aren’t necessarily important

Why did I miss the diagnosis? (Bordage, 1999) → “Less is better” i.e. foundations are good to build on

We tend to focus on student behaviour, instead of their learning e.g. “students must attend class and pay attention”…but if you’re not interesting, then why should they attend? What is it about their presence that somehow ensures that “learning happens”? If they’re not going to listen to you (and if they can pass the exam without attending), then why do we place so much emphasis on their presence?

Teach an approach to discovery, rather than a list of things

Dual processing theory (a universal model of diagnostic reasoning)

• How do we reason through clinical problems?
• Make observations and identify signs / variables
• Query your existing database i.e. your pre-existing knowledge
• Identify associations between the observed variables and your own database i.e. pattern recognition
• If existing knowledge is insufficient, query an external database (e.g. patient / textbook)

If you go through the above process enough times, you “get experience” → add more patterns to your internal database

I’m sure the above process is more eloquently and comprehensively described elsewhere

“Dancing naked in the mind field” – title of a book that perfectly describes why I blog…putting my thoughts, reflections and experiences out there and by doing so, exposing myself while sharing.

Having a diagnosis frees you from having to think. This has implications for when you’re tired / stressed / pushed for time, in that in those circumstances you can’t think and so latch onto a diagnosis. Students experience the same thing when they’re looking for answers. Having the answer means they don’t have to think because thinking is hard and places a high demand on system resources.

There’s a strong emotional response / association with diagnoses that are made intuitively i.e. without an analytical reasoning process

Talking out loud externalises a reasoning process that is often obscured and hidden from the student

“Diagnostic error and clinical reasoning” (Norman & Eva, Medical Education, 2010)

“construct referenced” as it relates to feedback?

Black , P. & William, D. (1998). Assessment and classroom learning, Assessment in Education, 5, pp. 7–75

Readings:

• “Classroom assessment techniques” (Angelo & Cross)
• Rushton, A. (2005). Formative assessment : a key to deep learning ? Medical Teacher, 27(6), 509-513
• Nofziger, A. C., Naumburg, E. H., Davis, B. J., Mooney, C. J., & Epstein, R. M. (2010). Impact of Peer Assessment on the Professional Development of Medical Students : A Qualitative Study. Academic Medicine, 85(1), 140-147

*SAFRI (Southern Africa FAIMER Regional Institute); FAIMER (Foundation for Advancement of International Medical Education and Research)

**SAAHE (South African Association of Health Educators)

Share