Objective Structured Clinical Exams

This is the first draft of the next piece of content that I’ll be publishing in my Clinical Teacher app.

Abstract

The Objective Structured Clinical Examination was introduced as an assessment method that aimed to address some of the challenges that arose with the assessment of students’ competence in clinical skills. In a traditional clinical examination there are several interacting variables that can influence the outcome, including the student, the patient, and the examiner. In the structured clinical examination, two of the variables – the patient and the examiner – are more controlled, allowing for a more objective assessment of the student’s performance.

The OSCE is a performance-based assessment that can be used in both formative and summative situations. It is a versatile multipurpose tool that can be used to evaluate healthcare students in the clinical context, used to assess competency based on objective testing through direct observation. As an assessment method it is precise, objective, and reproducible which means that it allows consistent testing of students for a wide range of clinical skills. Unlike the traditional clinical exam, the OSCE could evaluate areas most critical to the performance of healthcare professionals such as communication skills and the ability to handle unpredictable patient behaviour. However, the OSCE is not inherently without fault and is only as good as the team implementing it. Care should be taken not to assume that the method is in itself valid, reliable or objective. In addition, the OSCE cannot be used as a measure of all things important in medical education and should be used in conjunction with other assessment tasks.

 

Introduction and background

The OSCE was developed in an attempt to address some of the challenges with the assessment of clinical competence that were prevalent at the time (Harden, Stevenson, Wilson-Downie & Wilson, 1975). These included problems with validity, reliability, objectivity and feasibility. In the standard clinical assessment at the time, the student’s performance was assessed by two examiners who observed them with a several patients. However, the patient and examiner selection meant that chance played too dominant a role in the examination, leading to variations in the outcome (ibid.). Thus there was a need for a more objective and structured approach to clinical examination. The OSCE assesses competencies that are based on objective testing through direct observation. It consists of several stations in which candidates must perform a variety of clinical tasks within a specified time period against predetermined criteria (Zayyan, 2011).

The OSCE is a method of assessment that is well-suited to formative assessment. It is a form of performance-based assessment, which means that a student must demonstrate the ability to perform a task under the direct observation of an examiner. Candidates get examined on predetermined criteria on the same or similar clinical scenario or tasks with marks written down against those criteria thus enabling recall, teaching audit and determination of standards.

 

Rationale for the OSCE

While the OSCE attempts to address issues of validity, reliability, objectivity and feasibility it should be noted that it cannot be all things to all people. It is practically impossible to have an assessment method that satisfies all the criteria of a good test in terms of validity and reliability. For example the OSCE cannot be used to measure students’ competence of characteristics like empathy, commitment to lifelong learning and care over time. These aspects of students’ competence should be assessed with other methods. Having said that, we should discuss the four important aspects of accurate assessment that inform the implementation of the OSCE (Barman, 2005).

Validity

Validity is a measure of how well an assessment task measures what it is supposed to measure, and may be regarded as the most important factor to be considered in an assessment. For a test of have a high level of validity, it must contain a representative sample of what students are expected to have achieved. For example, if the outcome of the assessment task is to say that the student is competent in performing a procedure, then the test must actually measure the student’s ability to perform the procedure. In addition, the OSCE tests a range of skills in isolation, which does not necessarily indicate their ability to perform the separate tasks as an integrated whole.

Reliability

Reliability is a measure of the stability of the test results over time and across sample items. In the OSCE reliability may be low if there are few stations and short timeframes. Other factors that influence reliability include unreliable “standardised” patients, personal scoring systems, patients, examiners and students who are fatigued, and noisy or disruptive assessment environments. The best way to improve the reliability of an OSCE is to have a high number of stations and to combine the outcomes with other methods of assessment.

Objectivity

The objectivity of the OSCE relies on the standardisation of the stations and the checklist method of scoring student performance, which theoretically means that every student will be assessed on the same task in the same way. However, there is evidence that inter-rater reliability can be low on the OSCE as well, meaning that there is still a bias present in the method. In order to reduce the effect of this bias, the OSCE should include more stations.

Feasibility

In the process of making the decision about whether or not to use the OSCE as an assessment method i.e. whether or not it is feasible, there are a number of factors to be considered. These include the number of students to be assessed, the number of examiners available, the physical space available for running the exam, and the associated cost of these factors. It is important to note that the OSCE is more time-consuming and more expensive in terms of human and material cost than other assessment methods, for example the structured oral examination. In addition, the time required for setting up the examination is greater than that needed in traditional assessment methods, which must be taken into account when making decisions about whether or not to use the OSCE.

 

Advantages of the OSCE format

The OSCE format allows for the direct observation of a student’s ability to engage with clinical ethics skills during a patient interaction. In addition, the OSCE can be used effectively to evaluate students’ communication skills, especially if standardised instruments for assessing this skills are used. In addition, it (Shumway & Harden, 2003; Chan, 2009):

  • Provides a uniform marking scheme for examiners and consistent examination scenarios for students, including pressure from patients.
  • Generates formative feedback for both the learners and the curriculum, whereby feedback that is gathered can improve students’ competency and enhance the quality of the learning experience.
  • Allows for more students to be be examined at any one time. For example, when a student is carrying out a procedure, another student who has already completed that stage may be answering the question at another station.
  • Provides for a more controlled setting because only two variables exist: the patient and the examiner.
  • Provides more insights about students’ clinical and interactive competencies.
  • Can be used to objectively assess other aspects of clinical expertise, such as physical examination skills, interpersonal skills, technical skills, problem-solving abilities, decision-making abilities, and patient treatment skills.
  • Student participation in an OSCE has a positive impact on learning because the students’ attention is focused on the acquisition of clinical skills that are directly relevant to clinical performance.

 

Preparation for an OSCE

The first thing to do when considering developing an OSCE is to ask what it is to be assessed. It is important to realise that OSCEs are not appropriate for assessing all aspects of competence. For example, knowledge is best assessed with a written exam.

The venue where the OSCE is going to take place must be carefully considered, especially if it needs to be booked in advance. If there are large numbers of students, it may be worthwhile to have multiple tracks running in different venues. The advantages are that there will be less noise and fewer distractions. If space is not an issue, having separate rooms for each station is preferable, although multiple stations in a single room with partitions is also reasonable. If you will have real patient assisting, note that you will need rooms for them to rest in (Bouriscot, 2005).

Be aware that you will need to contact and confirm external examiners well in advance of running the OSCE. Clinicians are busy and will needs lots of advance warning. It may be useful to provide a grid of dates and times that are available to give examiners the option of choosing sessions that are most suitable for them (ibid.).

One of the key factors in the success of using the OSCE for assessment is the use of either real or standardised patients. This is a component that adds confidence to the reliability of the outcomes. Standardised patients are the next best thing to working with live patients. They are usually volunteers or actors who are trained in the role playing of different psychological and physiological aspects of patients. Finding and training standardised patients is a significant aspect of preparing for an OSCE (Dent & Hardent, 2005).

If equipment is required, ensure that there are lists available at every station, highlighting what equipment should be present in order for the student to successfully complete the station. You should go through each station with the list the day before the OSCE to ensure that all equipment is present (Bouriscot, 2005).

Mark sheets to be used for the OSCE must be developed in advance. Each examiner at each station must be provided with an appropriate number of mark sheets for the students, including an estimation of spoilage. If there are going to be large numbers of students, it may be worthwhile developing mark sheets that can be electronically scanned. If results are to be manually entered, someone will need to ensure that they have been captured correctly (Bouriscot, 2005).

 

Developing scenarios for each station

The number of stations in an examination is dependent on a number of factors, including the number of students to be assessed, the range of skills and content areas to be covered, the time allocated to each station, the total time available for the examination and the facilities available to conduct the examination (Harden & Cairncross, 1980). Preparing the content for each station should begin well in advance so that others can review the stations and perhaps even complete a practice run before the event. It may happen that a scenario is good in theory but that logistical complications make it unrealistic to run in practice.

The following points are important to note when developing stations (Bouriscot, 2005):

  • Instructions to students must be clear so that they know exactly what is expected of them at each station
  • Similarly, instructions to examiners must also make it clear what is expected of them
  • The equipment required at each station should be identified
  • Marking schedule that identifies the important aspects of the skill being assessed
  • The duration of the station

Stations should be numbered so that there is less confusion for students who are moving between them, and also for examiners who will be marking at particular stations. Note that it is recommended to have one rest station for every 40 minutes of assessment (Bouriscot, 2005). Arrows, either on the floor or wall will help candidates move between stations and avoid any confusion about rotation.

While stations may be set up in any number of ways, one suggested format is for the student to rotate through two “types” of stations; a procedure station and a question station (Harden, Stevenson, Wilson-Downie & Wilson, 1975). There are two advantages to this approach. In the first place it reduces the effect of cueing, whereby the question that the student must answer is presented at the same time as the instruction for performing the procedure. The nature of the question may prompt the student towards the correct procedure. By using two stations, the candidate is presented with a problem to solve or an examination to be carried out without the questions that come later. When the student gets to the “question” station, they are then unable to go back to the previous station to change their response. Thus the questions do not provide a prompt for the examination. The second advantage of the station approach is that more students can be examined at any one time. While one student is performing, another student who has already completed that stage is answering the questions (ibid.).

 

Running an OSCE

It may be useful, if the venue is large, to have a map of the facility set up, including the location of specific stations. This can help determine early on which stations will be set up in which rooms, as well as determining the order of the exam. The number of available rooms will determine how many stations are possible, as well as how many tracks can be run simultaneously (and therefore how many times each track will need to be run). You will also need a space for subsequent groups of students to be sequestered while previous round of students are finishing. If the exam is going to continue for a long time, you may need an additional room for examiners and patients to rest and eat.

Students should be informed in advance how they will proceed from one station to another. For example, will one bell be used to signal the end of one station and the beginning of another. If the OSCE is formative in nature, or a practice round, will different buzzers be used to signal a period of feedback from the examiner? When the bell signalling the end of the station sounds, candidates usually have 1 minute to move to the next station and read the instructions before entering.

On the day of the exam, time should be allocated for registering students, directing them to stations, setting the time, indicating station changes (buzzers, bells, etc.), and assisting with both setting up final changes and dismantling stations. Each station must have the station number and instructions posted at the entrance, and standardised patients, examiners and candidates matched to the appropriate stations. Examines and patients should be set up at their stations sufficiently in advance of the starting time in order to review the checklists and prepare themselves adequately. It may be possible to have a dry run of the station in order to help the patient get into the role.

It is possible to use paper checklists or to capture the marks with handheld devices like iPads or smartphones (see Software later). The benefits of using digital capturing methods as opposed to paper checklists is that the data is already captured at the end of the examination, and feedback to students and the organisers can be provided more efficiently. If paper checklists are used, they must be collected at the end of the day and data captured manually.

Some of the common challenges that are experienced during the running of the OSCE include (Bouriscot, 2005):

  • Examiners not turning up – send reminders the week before and have reserves on standby
  • Standardised patients not turning up – have reserves on standby
  • Patients not turning up – remind them the day before, provide transport, plan for more patients than are needed
  • Patient discomfort with the temperature – ensure that the venue is warmed up or cooled down before the OSCE begins
  • Incorrect / missing equipment – check the equipment the day before, have spares available in case of equipment malfunction, batteries dying, etc.
  • Patients getting ill – have medical staff on hand
  • Student getting ill – take them somewhere nearby to lie down and recover

The above list demonstrates the range of complications that can arise during an OSCE. You should expect that things will go wrong and try and anticipate them. However, you should also be aware that there will always be room for improvement, which is why attention must be paid to evaluating the process. It is essential that the process be continually refined and improved based on student and staff feedback (Frantz, et al., 2013).

 

Marking of the OSCE

The marking scheme for the OSCE is intentional and objectively designed. It must be concise, well-focused and unambiguous, with the aim of discrimination between good and poor student performance. The marking scheme must therefore be cognisant of many possible choices and provide scores that are appropriate to each student performance (Zayyan, 2011).

The allocation of marks between the different parts of the examination should be determined in advance and will vary with, among other things, the seniority of the students. Thus, with junior students there will be more emphasis their technique and fewer marks will be awarded for the findings of their interpretation (Harden, Stevenson, Wilson Downie, Wilson, 1975).

The following example marking rubric for OSCE stations is taken from Chan (2009):

 

Excellent Proficient Average Poor
Diagnosis Able to give an excellent analysis and understanding on the patients’ problems and situations and applied medical knowledge to the clinical practice and determined the appropriate treatment. Able to demonstrate medical knowledge with a satisfactory analysis on the patients’ problems, and determined the appropriate treatment. Showed a basic analysis and knowledge on the patients’ problems, still provided the appropriate treatment. Only able to show minimal level of analysis and knowledge on the patients’ problems, unable to provide the appropriate treatment.
Problem-solving skills Able to manage the time to suggest and bring out appropriate solutions to problems; more than one solutions were provided; logical approach to seek for solutions was observed. Able to manage the time to bring out only one solution; logical flow was still observed but there was a lack of relevance of the flow. Still able to bring out one solution on time; logical flow was hardly observed. Failed to bring out any solution in specific time; logical flow was not observed.
Communication and interaction Able to get detail information needed for diagnosis; gave very clear and detail explanation and answers to patients; paid attention to patients’ responses and words. Able to get detail information needed for diagnosis; gave clear explanation and answers to patients; attempted but only paid some attention to patients’ responses and words. Only able to get basic information needed for diagnosis; attempted to give a clear explanation to patients but omitted some points; did not pay attention to patients’ responses and words. Failed to get information for diagnosis; gave ambiguous explanation to patients.
Clinical skills Perfectly performed the appropriate clinical procedures for every clinical tasks with no omission; no unnecessary procedure was done. Performed the required clinical procedures satisfactorily; committed a few minor mistakes or unnecessary procedure which did not affect the overall completion of the procedure. Performed the clinical procedures at an acceptable standard; committed some mistakes and some unnecessary procedures were done. Failed to carry out the necessary clinical procedures; committed lots of mistakes and misconception about operating clinical apparatus.

 

Common mistakes made by students during the OSCE

It may be helpful to guide students before the examination by helping them to understand what the OSCE is not (Medical Council of Canada, n.d.).

  • Not reading the instructions carefully – The student must elicit from the “patient” only the precise information that the question requires. Any additional or irrelevant information provided must not receive a mark.
  • Asking too many questions – Avoid asking too many questions, especially if the questions are disorganised and erratic, and seem aimed at hopefully stumbling across the few appropriate questions that are relevant to the task. The short period of time is designed to test candidates ability to elicit the most appropriate information from the patient.
  • Misinterpreting the instructions – This happens when candidates try to determine what the station is trying to test, rather than working through a clinically appropriate approach to the patient’s presenting complaint.
  • Using too many directed questions – Open-ended questions are helpful in this regard as they give the patient the opportunity to share more detailed information, while still leaving space for you to follow up with more directed questions.
  • Not listening to patients – Patients often report that candidates did not listen appropriately and therefore missed important information that was provided during the interview. In the case of using standardised patients, they may be trained to respond to an apparently indifferent candidate by withdrawing and providing less information.
  • Not explaining what you are doing in physical examination stations – The candidates may not explain what they are doing during the examination, leaving the examiner guessing as to what was intended, or whether the candidate observed a particular finding. By explaining what you see, hear and intend doing, you provide the examiner with context that helps them in scoring you appropriately.
  • Not providing enough direction in management stations – At stations that aim to assess the candidate’s management skills, they should provide clear instructions that will help you to improve their performance.
  • Missing the urgency of a patient problem – When the station is designed to assess clinical priorities, work through the priorities first and then come back later for additional information if this was not elicited earlier.
  • Talking too much – The time that the candidate spends with their patient should be used effectively in order to obtain the most relevant information. Candidates should avoid showing off with their vast knowledge base. Speak to the patient with courtesy and respect, eliciting relevant information.
  • Giving generic information – The candidate should avoid giving generic information that is of little value to the patient when it comes to making an informed decision.

 

Challenges with the OSCE

While the OSCE has many positive aspects, it should be noted that there are also many challenges when it comes to setting up and running them. The main critique against the OSCE is that it is very resource intensive but there are other disadvantages that include (Barman, 2005; Chan, 2009):

  • Requiring a lot of organisation. However, an argument can also be made that the increased preparation time occurs before the exam and allows for an examiners time to be used more efficiently.
  • Being expensive in terms of manpower, resources and time.
  • Discouraging students from looking at the patient as a whole.
  • Examining a narrow range of knowledge and skills and does not test for history-taking competency properly. Students only examine a number of different patients in isolation at each station instead of comprehensively examining a single patient.
  • Manual scoring of OSCE stations is time-consuming and increases the probability of mistakes.
  • It is nearly impossible to have children as standardised patients or patients with similar physical findings.

In addition, while being able to take a comprehensive history is an essential clinical skill, the time constraints necessary in an OSCE preclude this from being assessed. Similarly, because students’ skills are assessed in sections, it is difficult to make decisions regarding students’ ability to assess and manage patients holistically (Barman, 2005). Even if one were able to construct stations that assessed all aspects of clinical skills, it would only test those aspects in isolation rather than comprehensively integrating them all into a single demonstration. Linked to that, the OSCE also has a potentially negative impact on students’ learning because it contains multiple stations that sample isolated aspects of clinical medicine. The student may therefore prepare for the examination by compartmentalising the skills and not completely understanding the connection between them (Shumway & Harden, 2003). There also seems to be some evidence that while the OSCE is an appropriate method of assessment in undergraduate medical education, it is less well-suited for assessing the in-depth knowledge and skills of postgraduate students (Patil, 1993).

Challenges with reliability in the clinical examination may arise from the fact that different students are assessed on different patients and one may come across a temperamental patient who may help some students while obstructing others. In addition, test scores may not reflect students’ actual ability as repetitive demands may fatigue the student, patient or examiner. Students’ fatigue due to lengthy OSCEs may may affect their performance. Moreover, some students affect experience greater tension before and during examinations, as compared to other assessment methods. In spite of efforts to control patient and examiner variability, inaccuracies in judgment due to these effects remain. (Barman, 2005).

 

Software for managing an OSCE

There is an increasing range of software that assists with setting up and running an OSCE. These services often run a on a variety of mobile devices, offering portability and ease of use for examiners. One of the primary benefits of the using digital, instead of paper, scoring sheets is that the results are instantly available for analysis and for reporting to students. Examples of some of the available software include OSCE Online, OSCE Manager and eOSCE.

Selection_002

Ten OSCE pearls

The following list is taken from Dent & Harden (2005), and includes lessons learned from practical experiences of running OSCEs.

  1. Make all stations the same length, since rotating students through the stations means that you can’t have some students finishing before others.
  2. Linked stations require preparation. For example, if station 2 requires the student to follow up on what was done at station 1, then no student can begin at station 2. This means that a staggered start is required. In this case, one student would begin the exam before everyone else. Then, when the main exam begins, the student at station 1 will move to station 2. This student will finish one station before everyone else.
  3. Prepare additional standardised patients, and have additional examiners available to allow for unpredictable events detaining either one.
  4. Have backup equipment in case any of the exam equipment fails.
  5. Have staff available during the examination to maintain security and help students move between stations, especially those who are nervous at the beginning.
  6. If there is a missing student, move a sign labelled “missing student” to each station as the exam progresses. This will help avoid confusion when other students move into the unoccupied station by mistake.
  7. Remind students to remain in the exam room until the buzzer signals the end of the station, even if they have completed their task. This avoids having students standing around in the areas between rooms.
  8. Maintain exam security, especially when running the exam multiple times in series. Ensure that the first group of students are kept away from the second group.
  9. Make sure that the person keeping time and sounding the buzzer is well-prepared, as they have the potential to cause serious confusion among examiners and students. In addition, ensure that the buzzer can be heard throughout the exam venue.
  10. If the rotation has been compromised and people are confused, stop the exam before trying to sort out the problem. If a student has somehow missed a station, rather allow them the opportunity to return at the end and complete it then.

 

Take home points

  • The OSCE aims to improve the validity, reliability, objectivity and feasibility of assessing clinical competence in undergraduate medical students
  • The method is not without it’s challenges, which include the fact that it is resource intensive and therefore expensive
  • Factors which can play a role in reducing confidence in the test results include student, examiner and patient fatigue.
  • The best way to limit the influence of factors that negatively impact on the OSCE is to have a high number of stations.
  • Being well-prepared for the examination is the best way to ensure that it runs without problems. However, even when you are well-prepared, expect their to be challenges.
  • The following suggestions are presented to ensure a well-run OSCE:
    • Set an exam blueprint
    • Develop the station cases with checklists and rating scales
    • Recruit and train examiners
    • Recruit and train standardised patients
    • Plan space and equipment needs
    • Identify budgetary requirements
    • Prepare for last-minute emergencies

 

Conclusion

The use of the OSCE format for clinical examination has been shown to demonstrate improvements in reliability and validity of the assessment, allowing examiners to say with more confidence that students are proficient in the competencies that are tested. While OSCEs are considered to be more fair than other types of practical assessment, they do require significant investment in terms of finance, time and effort. However, these disadvantages are offset by the improvement in objectivity that emerge as a result of the approach.

 

Bibliography

Accepting student work as a gift

Selection_001A few months ago we invited a colleague from the institution to give a short presentation in my department, sharing some of her ideas around research. At some point in the session, she said “I offer this to you, because…”. I forget the rest of the sentence but what was striking to me was how it had begun. It really resonated with something I’d read earlier this year, from Ronald Barnett’s book “A will to learn: Being a student in an age of uncertainty“. From Barnett:

Here are gifts given without any hope of even a ‘thank-you’, yet this ‘gift-giving’ looks for some kind of return. The feedback may come late; the marks may not be as hoped, but the expectation of some return is carried in these gifts. The student’s offerings are gifts and deserve to be recognized as such, despite their hoped-for return.

….

The language that I have in mind is one of proffering, of tendering, of offering, of sharing, and of presenting and gifting. The pedagogical relationship may surely be understood in just these terms, as a setting of gift-giving that at least opens a space for mutual obligations attendant upon gift-giving.

….

In the pedagogical setting, the student engages in activities circumscribed by a curriculum. Those activities are implicitly judged to be worthwhile, for the curriculum has characteristically been formally sanctioned (typically through a university’s internal course validation procedures). However, those curricula activities are not just worthwhile in themselves for they are normally intended to lead somewhere. In that leading somewhere, there is something that emerges, whether it be the result of a laboratory experiment, a problem that has been solved, an essay that has been submitted or a design that has been created. These are pedagogical offerings.

….

Both the teacher and the taught put themselves forward, offer themselves, give themselves. They even, to some extent, exchange themselves.

I think that there is something incredibly powerful that happens when we begin to think about the work that the student submits (offers) as a gift. Something that they have given of themselves, a representation of the time, effort and thought they have put into a creative work. If we think about the student’s offering as a gift, surely it must change the way it is treated and the way we respond? How does feedback and assessment change if we think of them as responses to gifts? Or, as gifts themselves? Would our relationships with students change (be enhanced?) if we thought of their submissions and our feedback as mutual gifts, offered to each other as representations of who we are?

“Eleven hundred hours” – Poem by a student

For the past few years I’ve been asking my final year students to develop a learning portfolio as part of the ethics module I teach. Even though I encourage them to use different forms of knowledge representation, few of them take up the offer. However, every now and again someone submits something very different to the 2 page narrative. The student has given me permission to share her work here.

Its 11. She normally comes at 11.
I hope she forgets today.
She doesn’t care how I feel.
I’m always so tired.
The medication makes me drowsy.
The lines across her face I cannot even discern, my eyesight is failing.
My legs are weak.
I cannot feel my big toe.
She uses a toothpick, I cannot feel it, yet I know it hurts.
I have HIV, I know that.
Some days I cry
She doesn’t know
I’m not sure if I can trust her
I tell her all I want to do is sleep
She talks about exercise
I haven’t exercised a day in my life
My life is about surviving
Surviving the streets of Hanover Park
Protecting my family
Selling myself to support my family
She doesn’t know…
Its 11. She always comes at 11…

Its 11! The hour I despise.
Ms X is next on my patient list.
I wish she would open up.
I talk and talk and nothing gets through to her.
She’s demotivated and I’ve used all my weapons in my arsenal to help her
But its null en void.
I wish I could help her, but she needs to let me in.
Her body language pushes me away,
Never looking directly at me,
But help her I must.
And try and try again I will.
She thinks I don’t understand.
She thinks I cannot see the pain and suffering.
A hard woman is she.
Burdened. Troubled. Scourged.
Her barriers I need to break down, if only she lets her guard down.
I hope in vain that tomorrow will be a better day.
It’s 11! The hour I despise.

Assessing Clinical Competence with the Mini-CEX

This is the first draft of an article that I published in The Clinical Teacher mobile app.

Introduction

The assessment of clinical competence is an essential component of clinical education but is challenging because of the range of factors that can influence the outcome. Clinical teachers must be able to make valid and reliable judgements of students’ clinical ability, but this is complex. The more valid and reliable a test is, the longer and more complicated it is to administer. The mini Clinical Evaluation Exercise, or mini-CEX was developed in response to some of the challenges of the traditional clinical evaluation exercise (CEX) and has been found to be a feasible, valid and reliable tool for the assessment of clinical competence.

Assessment of competence

Competence in clinical practice is defined as the “the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individuals and communities being served” (Epstein & Hundert, 2002). The assessment of competence can take a range of forms in clinical education, but this article will only discuss competence around the physical examination of patients.

Teaching physical examination skills is a unique challenge in clinical education because of the many variables that impact on how it is conducted. Consider how each of the following factors plays a role in the quality of teaching and learning that happens; the teachers’ own clinical skills; trainees’ prior knowledge, skills and interest; availability of patients with the necessary findings; patient willingness to be examined by a group of doctors and trainees who may not have any impact on their clinical care; the physical environment which is usually less than comfortable; and trainee fatigue level. In addition, the session should be relevant to the student and have significant educational value, otherwise there is the risk that it will degenerate into a “show and tell” exercise (Ramani, 2008).

This article will demonstrate how the mini-CEX provides a structured way to achieve the following goals of clinical assessment (Epstein, 2007):

  • Optimise the capabilities of all learners and practitioners by providing motivation and direction for future learning
  • Protect the public by identifying incompetent physicians
  • Provide a basis for choosing applicants for advanced training

The mini-Clinical Evaluation Exercise

The mini-CEX is a method of assessing the clinical competence of students in an authentic clinical setting, while at the same time providing a structured means of giving feedback to improve performance. It involves the direct observation of a focused clinical encounter between a student and patient, followed immediately with structured feedback designed to improve practice. It was developed in response to the shortcomings of both the traditional bedside oral examination and initial clinical evaluation exercise (CEX) (Norcini, 2005).

In the mini-CEX, the student conducts a subjective and objective assessment of a patient, focusing on one aspect of the patients presentation, and finishing with a diagnosis and treatment plan. The clinician scores the students’ performance on a range of criteria using the structured form, and provides the student with feedback on their strengths and weaknesses. The clinician highlights an area that the student can improve on, and together they agree on an action the student can take that will help them in their development. This can include a case presentation at a later date, a written exercise that demonstrates clinical reasoning, or a literature search (Epstein, 2007).

The session is relatively short (about 15 minutes) and should be incorporated into the normal routine of training. Ideally, the student should be assessed in multiple clinical contexts by multiple clinicians, although it is up to the student to identify when and with whom they would like to be assessed (Norcini, 2005). Students should be observed at least four times by different assessors to get a reliable assessment of competence (Norcini & Burch, 2007). The mini-CEX is a feasible, valid and reliable assessment tool with high fidelity for the evaluation of clinical competence (Nair, et al., 2008).

The mini-CEX is a good example of a workplace-based assessment method that fulfils three requirements for facilitating learning (Norcini & Burch, 2007):

  1. The course content, expected competencies and assessment practices are aligned
  2. Feedback is provided either during or immediately after the assessment
  3. The assessment is used to direct learning towards desired outcomes

Structure of a mini-CEX form

Each of the competences in Table 1 below is assessed on 9-point scale where 1-3 are “unsatisfactory”, 4 is “marginal”, 5-6 are “satisfactory”, and 7-9 are “superior” (Norcini, et al., 2005). In addition to the competences documented below, there is also space for both student and assessor to record their experience of the assessment, indicating their satisfaction with the process, time taken for the encounter and experience of the assessor.

Table 1: Competencies and descriptors of the mini-CEX form

Competence

Descriptor of a Satisfactory Trainee

History taking

Facilitates patient’s telling of story, effectively uses appropriate questions to obtain accurate, adequate information, responds appropriately to verbal and non-verbal cues.

Physical exam

Follows efficient, logical sequence; examination appropriate to clinical problem, explains to patient; sensitive to patient’s comfort, modesty.

Professionalism

Shows respect, compassion, empathy, establishes trust; Attends to patient’s needs of comfort, respect, confidentiality. Behaves in an ethical manner, awareness of relevant legal frameworks. Aware of limitations.

Clinical judgement

Makes appropriate diagnosis and formulates a suitable management plan. Selectively orders/ performs appropriate diagnostic studies, considers risks, benefits.

Communication skill

Explores patient’s perspective, jargon free, open and honest, empathetic, agrees management plan/therapy with patient.

Organisation/efficiency

Prioritises; is timely. Succinct. Summarises.

Overall clinical care

Demonstrates satisfactory clinical judgment, synthesis, caring, effectiveness. Efficiency, appropriate use of resources, balances risks and benefits, awareness of own limitations.

Role of the assessor

The assessor does not need to have prior knowledge or experience with assessing the student, but should have some experience in the domain of expertise that the assessment is relevant for. The patient must be made aware that the mini-CEX is going to used to assess a student’s level of competence with them, and they should give consent for this to happen. It is important to note that the session should be led by the trainee, not the assessor (National Health Service, n.d.).

The assessor must also ensure that the patient and assessment task selected is an appropriate example of something that the student would reasonably be expected to be able to do. Remember that the mini-CEX is only an assessment of competence within a narrow scope of practice, and therefore only a focused task will be assessed. They should also record the complexity of the patient’s problem, as there is some evidence that assessors score students higher on cases of increased complexity (Norcini, 2005).

After the session has been completed, the assessor must give feedback to the student immediately, highlighting their strengths as well as areas in which they can improve. Together, clinician and student must agree on an educational action that the student can take in order to improve their practice. It is also recommended that assessors go on at least a basic workshop to be introduced to the mini-CEX. Informal discussion is likely to improve both the quality of the assessment and of the feedback to students (Norcini, 2005).

Advantages of the mini-CEX

The mini-CEX also has these other strengths:

  • It is used in the clinical context with real patients and clinician educators, as opposed to the Objective Structured Clinical Exam (OSCE), which uses standardised patients.
  • It can be used in a variety of clinical settings, including the hospital, outpatient clinic and trauma, and while it was designed to be administered in the medical field, it is equally useful for most health professionals. The broader range of clinical challenges improves the quality of the assessment and of the educational feedback that the student receives.
  • The assessment is carried out by a variety of clinicians, which improves the reliability and validity of the tool, but also provides a variety of educational feedback for the student. This is useful because clinicians will often have different ways of managing the same patient, and it helps for students to be aware of the fact that there is often no single “correct” way of managing a patient.
  • The assessment of competence is accompanied with real, practical suggestions for improvement. This improves the validity of the score given and provides constructive feedback that the student can use to improve their practice.
  • The process provides a complete and realistic clinical assessment, in that the student must gather and synthesise relevant information, identify the problem, develop a management plan and communicate the outcome.
  • It can be included in students’ portfolio as part of their collection of evidence of general competence
  • The mini-CEX encourages the student to focus on one aspect of the clinical presentation, allowing them to prioritise the diagnosis and management of the patient.

Challenges when using the mini-CEX

There is some evidence that assessor feedback in terms of developing a plan of action is often ignored, negating the educational component of the tool. In addition, many students often fail to reflect on the session and to provide any form of self-evaluation. It is therefore essential that faculty training is considered part of an integrated approach to improving students’ clinical competence, because the quality of the assessment is dependent on faculty skills in history and physical exam, demonstration, observation, assessment and feedback (Holmboe, et al., 2004a). Another point to be aware of when considering the use of the mini-CEX is that it doesn’t allow for the comprehensive assessment of a complete patient examination (Norcini, et al., 2003).

Practice points

  • The mini-CEX provides a structured format for the assessment of students’ clinical competence within a focused physical examination of a patient
  • It is a feasible, valid and reliable method of assessment when it is used by multiple assessors in multiple clinical contexts over a period of time
  • Completion of the focused physical examination should be followed immediately by the feedback session, which must include an activity that the student can engage in to improve their practice

Conclusion

The mini-CEX has been demonstrated to be a valid and reliable tool for the assessment of clinical competence. It should be administered by multiple assessors in multiple clinical contexts in order for it to achieve its maximum potential as a both an assessment and educational tool.

 

References and sources

Simon Barrie presentation on Graduate Attributes

“Curriculum renewal to achieve graduate learning outcomes: The challenge of assessment”
Prof Simon Barrie, Director of T&L, University of Sydney

Last week I had the opportunity to attend a presentation on graduate attributes and curriculum renewal by Prof Simon Barrie. The major point I took away from it was that we need to be thinking about how to change teaching and assessment practices to make sure that we’re graduating the kinds of students we say we want to. Here are the notes I took.

campuslife

Assessment is often a challenge when it comes to curriculum renewal. The things that are important (e.g. critical thinking) are hard to measure. Which is why we often don’t even try.

Curriculum is more powerful than simply looking at T&L, although bringing in T&L is an essential aspect of curriculum development. Is curriculum renewal just “busy bureaucracy”? It may begin with noble aims but it can degenerate into managerial traps. Curriculum renewal and graduate attributes (GA) should be seen as part of a transformative opportunity.

GA are complex “things” and need to be engaged with in complex ways

GA should be focused on checking that higher education is fulfilling it’s social role. UNESCO World Declaration on Higher Education: “higher education has given ample proof of it’s viability over the centuries and of its ability to change and induce change and progress in society”.

GA should be a starting point for a conversation about higher education. If they exist simply as a list of outcomes, then they haven’t achieved their purpose.

How is an institution’s mission embodied in the learning experiences of students and teaching experiences of teachers?

What is the “good” of university?

  • Personal benefit – work and living a rich and rewarding life
  • Public benefit – economy and prosperity, social good
  • The mix of intended “goods” can influence our descriptions of the sorts of graduates that universities should be producing and how they should be taught and assessed. But, the process of higher education is a “good” in itself. The act of learning can itself be a social good e.g. when students engage in collaborative projects that benefit the community.

Universities need to teach people how to think and to question the world we live in.

If you only talk to people like you about GA, you develop a very narrow perspective about what they are. Speaking to more varied people, you are exposed to multiple set of perspectives, which makes curriculum renewal much more powerful. We bring our own assumptions to the conversation. Don’t trust your assumptions. Engage with different stakeholders. Don’t have the discussion around outcomes, have it around the purpose and meaning of higher education.

A framework for thinking about GA: it is complex and not “one size fits all”. Not all GA are at the same “level”, there are different types of “understand”, which means different types of assessment and teaching methods.

  • Precursor: approach it as a remedial function, “if only we got the right students”
  • Complementary: everybody needs “complementary” skills that are useful but not integral to domain-specific knowledge
  • Translation: applied knowledge in an intentional way, should be able to use knowledge, translating classroom knowledge into real world application, changing the way we think about the discipline
  • Enabling: need to be able to work in conditions of uncertainty, the world is unknowable, how to navigate uncertainty, develop a way of being in the world, about openness, going beyond the discipline to enable new ways of learning (difficult to pin down and difficult to teach, and assess, hard to measure)

The above ways of “understanding” are all radically different, yet many are put on the same level and taught and assessed in the same way. Policies and implementation needs to acknowledge that GA are different.

Gibbons: knowledge brought into the world and made real

The way we talk about knowledge can make it more or less powerful. Having a certain stance or attitude towards knowledge will affect how you teach and assess.

What is the link, if any, between the discipline specific lists and institutional / national higher education lists?

The National GAP – Graduate Attribute Project

What are the assessment tasks in a range of disciplines that generate convincing evidence of the achievement of graduate learning outcomes? What are the assurance processes trusted by disciplines in relation to those assessment tasks and judgments? Assessing and assuring graduate learning outcomes (AAGLO project). Here are the summary findings of the project.

Assessment for learning and not assessment of learning.

Coherent development and assessment of programme-level graduate learning outcomes requires an institutional and discipline statement of outcomes. Foundation skills? Translation attributes? Enabling attributes and dispositions? Traditional or contemporary conceptions of knowledge?

Assessment not only drives learning but also drives teaching.

  • Communication skills – Privileged
  • Information literacy – Privileged
  • Research and inquiry – Privileged
  • Ethical social professional understandings – Ignored (present in the lists, but not assessed)
  • Personal intellectual autonomy – Ignored (present in the lists, but not assessed)

Features of effective assessment practices:

  • Assessment for learning
  • Interconnected, multi-component, connected to other assessment, staged, not isolated
  • Authentic (about the real world), relevant (personally to the student), roles of students and assessors
  • Standards-based with effective communication of criteria, assessment for GA can’t be norm-referenced, must be standards-based
  • Involve multiple decision makers – including students
  • Programme level coherence, not just an isolated assessment but exists in relation to the programme

The above only works as evidence to support learning if it is coupled with quality assurance

  • Quality of task
  • Quality of judgment (calibration prior to assessment, and consensus afterwards)
  • Confidence

There is a need for programme-level assessment. Assessment is usually focused at a module level. There’s no need to assess on a module level if your programme level is effective. You can then do things like have assessments that cross modules and are carried through different year levels.

How does a university curriculum, teaching and learning effectively measure the achievement of learning outcomes? In order to achieve certain types of outcomes, we need to give them certain types of learning experiences.

Peter Knights “wicked competencies”: you can’t fake wickedness – it’s got to be the real thing, messy, challenging and consequential problems.

The outcomes can’t be used to differentiate programmes, so use teaching and learning methods and experiences to differentiate.

Stop teaching content. Use content as a framework to teach other things e.g. critical thinking, communication, social responsibility

5 lessons:

  1. Set the right (wicked) goals collaboratively
  2. Make a signature pedagogy for complex GA part of the 5 year plan
  3. Develop policies and procedures to encourage and reward staff
  4. Identify and provide sources of data that support curriculum renewal, rather than shut down conversations about curriculum
  5. Provide resources and change strategies to support curriculum renewal conversations

Teaching GA is “not someone else’s problem”, it needs to be integrated into discipline-specific teaching.

Be aware that this conversation is very much focused on “university” or “academic” learning, and ignores the many different ways of being and thinking that exist outside the university. How is Higher Education connecting with the outside world? Is there a conversation between us and everyone else?

We try to shape students into a mold of what we imagine they should be. We don’t really acknowledge their unique characteristics and embrace their potential contribution to the learning relationship?

We (academics) are also often removed from where we want our students to be. Think about critical thinking, inquiry-based learning, collaboration, embracing multiple perspectives. Is that how we learn? Our organisational culture drives us away from the GA we say we want our students to have.

Resources

Workplace-based assessment

Yesterday I attended a workshop / seminar on workplace-based assessment given by John Norcini, president of FAIMER and creator of the mini-CEX. Here are the notes I took.

Methods
Summative (“acquired learning” that’s dominated assessment) and formative (feedback that helps to learn, assessment for learning)

The methods below into the workplace, require observation and feedback

Portfolios (“collection of measures”) are workplace-based / encounter-based and must include observation of the encounter and procedures, with a patient record audit i.e. 360 degree assessment. Trainee evaluated on the contents of the portfolio. The training programme maintains the portfolio, but the trainee may be expected to contribute to it.

“Tick box”-type assessment isn’t necessarily a problem, it depends on how faculty observe and assess the tasks on the list.

Other: medical knowledge test

The following assessment methods are all authentic, in the sense that they need to be based in the real world, and assesses students on what they are actually doing, not what they do in an “exam situation”.

Mini-CEX
Assessor observes a trainee during a brief (5-10 min) patient encounter, and evaluates trainee on a few aspects /dimensions of the encounter. Assessor then provides feedback. Ideally should be different patients, different assessors, different aspects. Should take 10-15 minutes.

Direct observation of procedural skills (DOPS)
10-15 exercise, faculty observe a patient encounter, emphasis on procedures, assessor rates along a no. of dimentsions, assessor then provides feedback.

Chart stimulated recall
Assessor reviews a patient record where trainee makes notes. Discussion centred on the trainee’s notes, and rates things like diagnoses, planning, Rx, etc. Has an oral exam with trainee, asking questions around clinical reasoning based on the notes. Takes 10-15 minutes, and should be over multiple encounters. Must use actual patient records → validity / authentic.

360 degree evaluation
Trainee nominates peers, faculty, patients, self, etc. who then evaluate the trainee. Everyone fills out the same form, which assesses clinical and generic skills. Trainee is given self-ratings, assessor ratings, mean ratings. Discrepency forms a foundation for discussion around the misconceptions. Good to assess teamwork, communication, interpersonal skills, etc.

There are forms available for these tasks, but in reality, since it’s formative, you can make up a form that makes sense for your own profession. These assessments are meant to be brief, almost informal, encounters. They should happen as part of the working process, not scheduled as part of an “evaluation” process. This should also not replace a more comprehensive, in-depth evaluation. They may also be more appropriate for more advanced trainees, and undergrad students may be better served with a “tick-list”-type assessment tool, since they’re still learning what to do.

Don’t aim for objectivity, aim for consensus. Aggregating subjective judgements brings us to what we’re calling “objective”. We can’t remove subjectivity, even in the most rigorous MCQs, as it’s human beings that make choices about what to include, etc. So, objectivity, is actually impossible to achieve. But consensus can be achieved.

For these methods, you can make the trainee responsible for the process (i.e. they can’t progress / complete without doing all the tasks), so the trainee decides which records, when it takes place, who will assess. This creates an obvious bias. Or, faculty can drive the process, in which case it often doesn’t get done.

Why are workplace methods good for learning?
Good evidence that trainees are not observed often during their learning i.e. lack of formative assessment during the programme. Medical students are often observed for less than 10% of their time in the clinical settings. If the trainees aren’t being observed and getting feedback related to that performance.

WPBA is crtical for learning and have a significant influence on achievement. One of the 4 major factors that influence learning is feedback, which counts for massive effect sizes in learning. Feedback alone is often effective in creating achievement in 70% of studies. Feedback is based on observation. Good feedback is often about providing sensitive information to individuals, which can be challenging in a group. Positive feedback given early in training can have long-lasting effects, and can be given safely in groups.

Feedback given by different professions, at different levels, is a good thing for trainees. So, observation of procedures, etc. should be done by a variety of people, in a variety of contexts. People should be targeted for feedback, based on the type of feedback they’re most appropriate to give i.e. to give feedback on what they do best. So, it’s fine for a physio to give feedback on a doctor’s performance, but it might be about teamwork ability, rather than medical knowledge.

Giving feedback is different from giving comments. Feedback creates a pathway to improvement of learning, whereas comments might just make students feel better for a short period of time.

Types of training

Massed – many people together for a short period of time, is intense, is faster, results in higher levels of confidence among trainees, and greater satisfaction

Spaced – many people, spread out over time, results in longer retention and better performance

Retrieval of information or a perfomance enhances learning. Learning isn’t about information going in, it’s also about how to retrieve information. Testing forces retrieval. Regular repetition of a performance leads to better performance of a task.

Faculty don’t agree with direct observation of performance, on the quality of the performance. So, you need to have several observations.
All patients are different, so you have to have observations of several patients.
The time frame for a long-case assessment is unreasonable in the real world, so assessment should be within a time frame that is authentic.

WPBA focuses on formative assessment, requires observation and feedback, directs and cretes learning, responds to the problems of traditional clinical assessment.

Rating students on a scale of unsatisfactory, satisfactory, etc. is formative and doesn’t carry the weight as the weight of a pass / fail, summative assessment. We also need to make sure that dimensions of the assessment are commonly defined or understood, and that faculty expectations for the assessment are the same.

Assessment forms should be modified to suit the context it is to be used in.

Gobal vs. check list assessments
Mini-CEX is a type of global i.e. it’s a judgement based on a global perception of the trainee. Our assessments are more global assessments. The descriptions of behaviours / dimensions are meant to indicate assessors with what they should be thinking about during the assessment.
A check list is a list of behaviours, and when the behaviour occurs, the trainee gets a tick.
Our assessment forms were mixing the two types of form, which may be why there were problems.

Faculty development should aim to “surface disagreement”, because that is how you generate discussion.

Conducting the encounter

  • Be prepared and have goals for the session
  • Put youself into the right posotion
  • Minimise external interruptions
  • Avoid intrusions

Characteristics of effective faculty development programmes (Skeff, 1997) – link to PDF

Faculty training / workshops are essential to prepare faculty to use the tools. It makes them more comfortable, as well as more stringent with students. If you’re not confident in your own ability, you tend to give students the benefit of the doubt. Workshops can be used to change role model behaviours.

Feedback

  • Addressees three aspects: Where am I going? How am I going? Where to next?
  • Four areas that feedback can focus on: task, process, self-regulation, self as a person (this last point is rarely effective, and should be avoided, therefore feedback must focus on behaviour, not on the person)
  • Response to feedback is influenced by the trainees level of achievement, their culture, perceptions of the accuracy of the feedback, perceptions of credbility and trustworthiness of the assessor, perceptions of the usefulness of the feedback
  • Technique of the assessor influences the impact that the feedback has: establish appropriate interpersonal climate, appropriate location, elicit trainees feelings and thoughts, focus on observed behaviours, be non judgemental, be specific, offer right amount of feedback (avoid overwhelming), suggestions for improvement
  • Provide an action plan and close the loop by getting student to submit something

Novice student: emphasis feedback on the task / product / outcome
Intermediate student: emphasise specific processes related to the task / performance
Advanced student: emphasise global process that extends beyond this specific situation e.g. self-regulation, self-assessment.

Necessary to “close-the-loop” so give students something to do i.e. an action plan that requires the student to go away and do something concrete that aims to improve an aspect of their performance.

Asking students what their impressions of the task were, is a good way to set up self-regulation / self-assessment by the student.

Student relf-report on something like confidence may be valid, but student self-report on competence is probably not, because students are not good judges of their own competence.

Summary
Provide an assessment of strengths and weaknesses, enable learner reaction, encourage self-assessment, develop an aciton plan.

Quality assurance in assessment (this aspect of the workshop conducted by Dr. Marietjie de Villiers)

Coming to a consensual definition:

  • External auditors (extrinsic) vs self-regulated (intrinsic)
  • Developing consensus as to what is being assessed, how, etc. i.e. developing outcomes
  • Including all role players / stakeholders
  • Aligning outcomes, content, teaching strategies, assessment i.e. are we using the right tools for the job?
  • “How can I do this better?”
  • Accountability (e.g. defending a grade you’ve given) and responsibility
  • There are logistical aspects to quality assurance i.e. beaurocracy and logistics
  • A quality assurance framework may feel like a lot of work when everything is going smoothly, but it’s an essential “safety net” when something goes wrong
  • Quality assurance has no value if it’s just “busy work” – it’s only when it’s used to change practice, that it has value
  • Often supported with a legal framework

Some quality assurance practices by today’s participants:

  • Regular review of assessment practices and outcomes can identify trends that may not be visible at the “gound level”.
  • Problems identified should lead to changes in practice.
  • Train students how to prepare for clinical assessments. Doesn’t mean that we should coach them, but prepare them for what to expect.
  • Student feedback can also be valuable, especially if they have insight into the process.
  • Set boundaries, or constraints on the assessment so that people are aware that you’re assessing something specific, in a specific context.
  • Try to link every procedure / skill to a reference, so that every student will refer back to the same source of information.
  • Simulating a context is not the same as using the actual context.
  • Quality assurance is a long-term process, constantly being reviewed and adapted.
  • Logistical problems with very large student groups require some creativity in assessment, as well as the evaluation of the assessment.
  • Discuss the assessment with all participating assessors to ensure some level of consensus re. expectations, at a pre-exam meeting. Also have a post-exam meeting to discuss outcomes and discrepencies.
  • Include external examiners in the assessment process. These external examiners should be practicing clinicians.

When running a workshop, getting input from external (perceived to be objective) people can give what you’re trying to do an air of credibility that may be missing, especially if you’re presenting to peers / colleagues.

2 principles:
Don’t aim for objectivity, aim for consensus
Multiple sources of input can improve the quality of the assessment

2 practical things:
Get input from internal and external sources when developing assessment tasks
Provide a standard source for procedures / skills so that all students can work from the same perspective

Article on work based assessment from BMJ

Peer review of teaching

Introduction
Peer review is a form of evaluation designed to provide feedback to teachers about their professional practice. The standard method of evaluating teaching is to ask students at the end of a module or course, for their feedback on the lecturers performance. While student feedback does have value, it also has limitations. For example, students are often not qualified to determine a lecturers knowledge base or understanding of course content. They may also lack the skills to identify appropriate levels of difficulty of the assessment tasks, as well as the appropriateness of the learning objectives as they relate to the overall curriculum.

Peer review of your teaching practice should be performed over time on different occasions, by different colleagues. This will create a more reliable measure of your teaching practice, as it goes some way towards eliminating bias. Effective peer evaluation should incorporate input from multiple sources. These can include the peer review from colleagues, but should also integrate feedback from students, personal reflection (as might be obtained from a teaching portfolio), and a review of student work. Students in particular can provide input on their perceptions of the classroom instruction process, outside-classroom interactions, and their satisfaction with the lecturer’s ability to mentor them. This integration of input from various sources allows for a more comprehensive, holistic view of teaching practice.

Why you should consider using peer review
Peer review of your teaching practice has several benefits. These include;

  • The opportunity to learn from others’ perspectives
  • Being exposed to new ideas
  • New staff learning from more experienced colleagues

However, you should also be aware of some possible pitfalls. These include bias when the observer has beliefs about teaching practice that aren’t consistent with your own, and a lack of validity (when used summatively) if it is viewed as an independent indicator of teaching ability.

Peer review process guidelines
Now that you have a form that is relevant for your context, you need to implement it. Think of the process as a collaborative one, rather than an assessment. Before you begin, you should decide what class is going to be observed, and who will observe you. It may be difficult but try and choose someone who can help identify areas in which you can improve, rather than someone you feel safe with. It may be easier asking a friend but you may find that they don’t give you the objective evaluation you need to develop your practice. The same goes for the activity you choose. You may want to go with a module you’re comfortable with and that you know well, but this doesn’t allow you much chance to improve. Instead, try to use the process as opportunity to challenge yourself.

Before the activity begins, you should meet with the observer so that you can discuss what you will be doing during the session. The aim of this is to provide some context for them to work within. For example, you may discuss the goal of your session, specific objectives you wish to achieve, the teaching strategies you will use to achieve the objective, how you will measure this achievement, as well as any concerns you would like the observer to take note of. An activity outline that they can keep may help to remind them what you’re going to try and do during your teaching activity.

The observer should arrive 10 minutes before the class begins, as arriving late is to model poor behaviour to students. They should be briefly introduced to the students, and their role explained. Finally, the observer should not ask questions during the activity, as this may detract from the process and invalidate the outcome. If the activity will go on for more than an hour, decide beforehand which components the observer will stay for.

After the teaching activity you should de-brief with the observer. This can either be done immediately, or after a short period of reflection. The advantage of doing it immediately is that everything is fresh in everyone’s minds, but which doesn’t allow time for both parties to reflect on the process. Whether you choose to do it immediately or after a reflective period, this session is where the observer can report on their observations for further discussion. This session should be led by the person who was observed, rather than be in the form of post-mortem by the observer.

Following the discussion, it is essential to decide on a set of actions that you can take in order to move forward and use the review process as a means of improving your practice. Without setting objectives for improvement based on the feedback, and then taking action to achieve those objectives, there is little point in the peer review process. Finally, the completed evaluation form, professional development objectives, and plan of action should be archived in your teaching portfolio.

Click on the image below for an example peer review form

Hints and tips

  • Peer review has been identified as one way in which teaching practices can be improved through objective feedback from colleagues
  • In order to gain the maximum benefit from peer review, there are processes that can be followed, rather than taking an ad hoc approach
  • Peer review should always be followed by a plan of action. Without acting on the feedback, the process is little more than an administrative exercise

References and sources
Peer Review of Teaching for Promotion Applications: Peer Observation of Classroom Teaching. Information, Protocols and Observation Form for Internal Peer Review Team.

Babbie, E. & Mouton, J. (2006). The Practice of Social Research. Oxford University Press. ISBN: 0195718542.

Brent, R. & Felder, R.M. (2004). A Protocol for Peer Review of Teaching. Proceedings of the 2004 American Society for Engineering Education Annual Conference & Exposition.

Butcher, C. Davies, C. and Highton, M. (2006). Designing Learning. From Module Outline to Effective Teaching. Routledge. ISBN: 9780415380300.

Graduate Attributes (2006). Curtin University of Technology.

Peer Observation Guidelines and Recommendations. University of Minnesota, Center for Teaching and Learning.

Additional reading and resources
Classroom Observation Instruments. University of Minnesota, Center for Teaching and Learning. (a list of instruments that you can use in your own teaching practice)

McKenzie, J. & Parker, N. (2011). Peer review in online and blended learning environments. Report from the Australian Learning and Teaching Council.

Harris, K., Farrell, K., Bell, M., Devlin, M. & James, R. (2008). Peer Review of Teaching in Australian Higher Education. A handbook to support institutions in developing and embedding effective policies and practices. Centre for the Study of Higher Education. ISBN 9780734040459.

Resources on Peer Observation and Review. University of Minnesota, Center for Teaching and Learning.

Jan Herrington’s model of Authentic learning

A few days ago I met with my supervisor  to discuss my research plan for the year. She suggested I look into Jan Herrington’s work on authentic learning so I thought I’d make some notes here as I familiarize myself with it.

To begin with, there are 9 elements of authentic learning (I believe that in designing our blended module we’ve managed to cover most of these elements. I’ll write that process up another time):

  1. Provide authentic contexts that reflect the way the knowledge will be used in real life
  2. Provide authentic tasks and activities
  3. Provide access to expert performances and the modelling of processes
  4. Provide multiple roles and perspectives
  5. Support collaborative construction of knowledge
  6. Promote reflection to enable abstractions to be formed
  7. Promote articulation to enable tacit knowledge to be made explicit
  8. Provide coaching and scaffolding by the teacher at critical times
  9. Provide for authentic assessment of learning within the tasks

The above elements are non-sequential.

“Authentic activities” don’t necessarily mean “real”, as in constructed in the real-world (e.g. internship), only that they are realistic tasks that enable students to behave as they would in the real-world.

Here are 10 characteristics of authentic activities (Reeves, Herrington & Oliver, 2002). Again, I believe that we’ve designed learning activities and tasks that conform – in general – to these principles. It’s affirming to see that our design choices are being validated as we move forward. In short, authentic tasks:

  1. Have real-world relevance i.e. they match real-world tasks
  2. Are ill-defined (students must define tasks and sub-tasks in order to complete the activity) i.e. there are multiple interpretations of both the problem and the solution
  3. Are complex and must be explored over a sustained period of time i.e. days, weeks and months, rather than minutes or hours
  4. Provide opportunities to examine the task from different perspectives, using a variety of resources i.e. there isn’t a single answer that is the “best” one. Multiple resources requires that students differentiate between relevant / irrelevant information
  5. Provide opportunities to collaborate should be inherent i.e. are integral to the task
  6. Provide opportunities to reflect i.e. students must be able to make choices and reflect on those choices
  7. Must be integrated and applied across different subject areas and lead beyond domain-specific outcomes i.e. they encourage interdisciplinary perspectives and enable diverse roles and expertise
  8. Seamlessly integrated with assessment i.e. the assessment tasks reflect real-world assessment, rather than separate assessment removed from the task
  9. Result in a finished product, rather than as preparation for something else
  10. Allow for competing solutions and diversity of outcome i.e. the outcomes can have multiple solutions that are original, rather than a single “correct” response

Design principles for authentic e-learning (Herrington, 2006)

“Authentic learning” places the task as the central focus for authentic activity, and is grounded in part in the situated cognition model (Brown et al, 1989) i.e. meaningful learning will only occur when it happens in the social and physical context in which it is to be used.

“How can situated theories be operationalized?” (Brown & Duguid, 1993, 10). Herrington (2006) suggests that the “9 elements” framework can be used to design online, technology-based learning environments based on theories of situated learning.

The most successful online learning environments:

  • Emphasised education as a process, rather than a product
  • Did not seek to provide real experiences but to provide a “cognitive realism”
  • Accept the need to assist students to develop in a completely new way

There is a tendency when using online learning environments to focus on the information processing features of computers and the internet. There is rarely an understanding of the complex nature of learning in unfamiliar contexts in which tasks are “ill-defined”.

The “physical fidelity” (how real it is) of the material is less important than the extent to which the activity promotes “realistic problem-solving processes” i.e. it’s cognitive realism. “The physical reality of the learning situation is of less importance that the characteristics of the task design, and the engagement of students in the learning environment” (Herrington, Oliver, & Reeves, 2003a).

Learners may need to be assisted in coming to terms with the fact that the simulated reality of their task is in fact, an authentic learning environment. It may call for their “willing suspension of disbelief” (Herrington, 2006).

There is a need for design-based research into the efficacy of authentic learning to better understand the affordances and challenges of the approach.

An instructional design framework for authentic learning environments (Herrington & Oliver, 2000)
One of the difficulties with higher education is teaching concepts, etc. in a decontextualised situation, and then expecting the students / graduates to apply what they’ve learned in another situation. This is probably one of the biggest challenges in clinical education, with people being “unable to access relevant knowledge for solving problems”

“Information is stored as facts, rather than as tools (Bransford, Sherwood, Hasselbring, Kinzer & Williams, 1990). When knowledge and context are separated, knowledge is seen by learners as a product of education, rather than a tool to be used within dynamic, real-world situations. Situated learning is a model that encourages the learning of knowledge in contexts that reflect the way in which the knowledge is to be used (Collins, 1988).

Useful tables and checklists on pg. 4-6 and pg. 8-10 of Herrington & Oliver, 2000. An instructional design framework for authentic learning environments
An “ill-defined” problem isn’t prescriptive, lacks boundaries, doesn’t provide guiding questions and doesn’t break the global task into sub-tasks. Students are expected to figure out those components on their own. We’re beginning by providing boundaries and structure. As we move through subsequent cases, the facilitators will withdraw structure and guidance, until by the end of the module, students are setting their own, personal objectives. Students should define the pathway and the steps they need to take.

Situated learning seems to be an effective teaching model with trying to guide the learning of an appropriately complex task i.e. advanced knowledge acquisition

Students benefit from the opportunity to articulate, scaffold and reflect on activities with a partner. When these opportunities are not explicitly described, students may seek it covertly.

Students often perceive a void between theory and practice, viewing theory as relatively unimportant (jumping through hoops, in the case of our students…busy-work with no real benefit other than passing theory exams) and the practical component as all-important. They appreciate the blurring of boundaries between the two domains.

The authentic activity should present a new situation for which the students have no answer, nor for which they have a set of procedures for obtaining an answer i.e. it should be complex and the solution uncertain.

Herrington & Reeves (2003). Patterns of engagement in authentic online learning environments

There seems to be an initial reluctance to immerse oneself in the online learning environment, possibly owing to the lack of realism from contexts that are not perfect simulations of the real-world. Students may need to be encouraged to suspend their disbelief  (pg. 2). They must agree to go along with an interpretation of the world that has been created.
Once the student has accepted the presented interpretation of the world, it is only internal inconsistency that causes dissonance. Other challenges occur when students perceive the environment as being non-academic, non-rigorous, a waste of time, and unnecessary for effective learning (which may well be the case if they perceive “effective learning” as sitting passively in a classroom trying to memorise content)
Be aware that the designer of the online space may present an interpretation of the world that is not shared with everyone i.e. it is one person’s view of what the real world is like.
A willing suspension of disbelief can be likened to engagement: “…when we are able to give ourselves over to a representational action, comfortably and unambiguously. It involves a kind of complexity” (Laurel, 1993, 115). It isn’t necessary to try and perfectly simulate the real-world, only that the representation is close enough to get students engaged e.g. the quality / realism  of images doesn’t have to be perfect, as long as it enables students to get the idea.
Many students find the shift to a new learning paradigm uncomfortable. If students are not self-motivated, if they are accustomed to teacher-centred modes of instruction and if they dislike the lack of direct supervision, they may resist. They may also be uncomfortable with the increased freedom they have i.e. there is less teacher-specified content, fewer teacher-constructed objectives, and almost no teacher-led activities. On some occasions, students may feel that they are not being taught, and may express this with anger and frustration.
The facilitator is vital in terms of presenting the representation in a way that encourages engagement, much like an actor in a play must convince the audience that what is happening on the stage is “real”. Without that acceptance, you would not enjoy the play, just as the student won’t perceive the value of the learning experience.
Students need to be given the time and space to make mistakes. They will begin by working inefficiently, but the expectation is that efficiency increases over time.
We need to “humanise” the online learning experience with compassion, empathy and open-mindedness.

References

  • Bransford, J.D., Sherwood, R.D., Hasselbring, T.S., Kinzer, C.K., & Williams, S.M. (1990). Anchored instruction: Why we need it and how technology can help. In D. Nix & R. Spiro (Eds.), Cognition, education and multimedia: Exploring ideas in high technology (pp. 115-141). Hillsdale, NJ: Lawrence Erlbaum
  • Brown, J.S., & Duguid, P. (1993). Stolen knowledge. Educational Technology, 33(3), 10-15
  • Brown, J.S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32-42
  • Collins, A. (1988). Cognitive apprenticeship and instructional technology (Technical Report 6899): BBN Labs Inc., Cambridge, MA
  • Herrington, J. (2006). Authentic e-learning in higher education: Design principles for authentic learning environments and tasks, World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education, Chesapeake, Va
  • Herrington, J., & Oliver, R. (2000). An instructional design framework for authentic learning environments. Educational Technology Research and Development, 48(3), 23-48
  • Herrington, J., Oliver, R., & Reeves, T.C. (2003a). ‘Cognitive realism’ in online authentic learning environments. In D. Lassner & C. McNaught (Eds.), EdMedia World Conference on Educational
  • Herrington, J., & Reeves, T. C. (2003). Patterns of engagement in authentic online learning environments. Australian Journal of Educational Technology, 19(1), 59-71
  • Laurel, B. (1993). Computers as theatre. Reading, MA: Addison-Wesley
  • Reeves, T. C., Herrington, J., & Oliver, R. (2002). Authentic activities and online learning. HERDSA (pp. 562-567)

Twitter Weekly Updates for 2012-01-09

Teaching and learning workshop at Mont Fleur

Photo taken while on a short walk during the retreat.

A few weeks ago I spent 3 days at Mont Fleur near Stellenbosch, on a teaching and learning retreat. Next year we’re going to be restructuring 2 of our modules as part of a curriculum review, and I’ll be studying the process as part of my PhD. That part of the project will also form a case study for an NRF-funded, inter-institutional study on the use of emerging technologies in South African higher education.

I used the workshop as an opportunity to develop some of the ideas for how the module will change (more on that in another post), and these are the notes I took during the workshop. Most of what I was writing was specific to the module I was working with, so these notes are the more generic ones that might be useful for others.

————————

Content determines what we teach, but not how we teach. But it should be the outcomes that determine the content?

“Planning” for learning

Teaching is intended to make learning possible / there is an intended relationship between teaching and learning

Learning = a recombination of old and new material in order to create personal meaning. Students bring their own experience from the world that we can use to create a scaffold upon which to add new knowledge

We teach what we usually believe is important for them to know

What (and how) we teach is often constrained by external factors:

  • Amount of content
  • Time in which to cover the content (this is not the same as “creating personal meaning”)

We think of content as a series of discrete chunks of an unspecified whole, without much thought given to the relative importance of each topic as it relates to other topics, or about the nature of the relationships between topics

How do we make choices between what to include and exclude?

  • Focus on knowledge structuring
  • What are the key concepts that are at the heart of the module?
  • What are the relationships between the concepts?
  • This marks a shift from dis-embedded facts to inter-related concepts
  • This is how we organise knowledge in the discipline

Task: map the knowledge structure of your module

“Organising knowledge” in the classroom is problematic because knowledge isn’t organised in our brains in the same way that we organise it for students / on a piece of paper. We assign content to discrete categories to make it easier for students to understand / add it to their pre-existing scaffolds, but that’s not how it exists in minds.

Scientific method (our students do a basic physics course in which this method is emphasised, yet they don’t transfer this knowledge to patient assessment):

  1. Observe something
  2. Construct an hypothesis
  3. Test the hypothesis
  4. Is the outcome new knowledge / expected?

Task: create a teaching activity (try to do something different) that is aligned with a major concept in the module, and also includes graduate attributes and learning outcomes. Can I do the poetry concept? What about gaming? Learners are in control of the environment, mastering the task is a symbol of valued status within the group, a game is a demarcated learning activity with set tasks that the learner has to master in order to proceed, feedback is built in, games can be time and resource constrained

The activity should include the following points:

  • Align assessment with outcomes and teaching and learning activities (SOLO taxonomy – Structured Observation of Learning Outcomes)
  • Select a range of assessment tools
  • Justify the choice of these tools
  • Explain and defend marks and weightings
  • Meet the criteria for reliability and validity
  • Create appropriate rubrics

Assessment must be aligned with learning outcomes and modular content. It provides students with opportunities to show that they can do what is expected of them. Assessment currently highlights what students don’t know, rather than emphasising what they can do, and looking for ways to build on that strength to fill in the gaps.

Learning is about what the student does, not what the teacher does.

How do you create observable outcomes?

The activity / doing of the activity is important

As a teacher:

  • What type of feedback do you give?
  • When do you give it?
  • What happens to it?
  • Does it lead to improved learning?

Graduate attributes ↔ Learning outcomes ↔ Assessment criteria ↔ T&L activities ↔ Assessment tasks ↔ Assessment strategy

Assessment defines what students regard as important, how they spend their time and how they come to see themselves as individuals (Brown, 2001; in Irons, 2008: 11)

Self-assessment is potentially useful, although it should be low-stakes

Use a range of well-designed assessment tasks to address all of the Intended Learning Outcomes (ILOs) for your module. This will help to provide evidence to teachers of the students competence / understanding

In general quantitative assessment uses marks while qualitative assessment uses rubrics

Checklist for a rubric:

  • Do the categories reflect the major learning objectives?
  • Are there distinct levels which are assigned names and mark values?
  • Are the descriptions clear? Are they on a continuum and allow for student growth?
  • Is the language clear and easy for students to understand?
  • Is it easy for the teacher to use?
  • Can the rubric be used to evaluate the work? Can it be used for assessing needs? Can students easily identify growth areas needed?

Evaluation:

  • What were you evaluating and why?
  • When was the evaluation conducted?
  • What was positive / negative about the evaluation?
  • What changes did you make as a result of the feedback you received?

Evaluation is an objective process in which data is collected, collated and analysed to produce information or judgements on which decisions for practice change can be based

Course evaluation can be:

  • Teacher focused – for improvement of teaching practice
  • Learner focused – determine whether the course outcomes were achieved

Evaluation be conducted at any time, depending on the purpose:

  • At the beginning to establish prior knowledge (diagnostic)
  • In the middle to check understanding (formative) e.g. think-pair-share, clickers, minute paper, blogs, reflective writing
  • At the end to determine the effectiveness of the course / to determine whether outcomes have been achieved (summative) e.g. questionnaires, interviews, debriefing sessions, tests

Obtaining information:

  • Feedback from students
  • Peer review of teaching
  • Self-evaluation

References

  • Knight (n.d.). A briefing on key concepts: Formative and summative, criterion and norm-referenced assessment
  • Morgan (2008). The Course Improvement Flowchart: A description of a tool and process for the evaluation of university teaching