Problem based learning: transitioning to an online / hybrid learning environment

A few weeks ago I attended a short presentation by Prof. Meena Iyer from Missouri University. Prof. Iyer spoke about how she moved her PBL module from using a traditional, mainly face-to-face approach, to an online / hybrid approach. Here are my notes.

—————————-

“All life is problem solving” – Karl Popper

How do we get students to think like professionals in the field?
How do we foster group interaction in online spaces?
How do I assess learning in online spaces?

PBL addresses the content issue, as well as enhancing critical thinking through the collaborative solving of authentic, real-world problems

Mismatch:

  • PBL → solving problems is the tool, learning is the goal
  • Traditional → content is the tool, problem solving is the goal

PBL is all unstructured (but it can be scaffolded), and there’s not necessarily a right/wrong answer

Six steps to problem solving (IDEALS):

  • Identify the problem (What is the real question we are facing?)
  • Define the context (What are the facts that frame this problem?)
  • Enumerate the choices (What are the plausible actions?)
  • Analyse the options (What is the best course of action?)
  • List reasons explicitly (why is this the best course of action?)
  • Self-correct (What did we miss?)

The problem should be authentic and appealing (a mystery to solve)
Clearly outline expectations for each step of the process

Why move from face-to-face to online?

  • In F2F, you can only move forward at the speed of the slowest learner
  • Significant time requirements for F2F
  • Identify…can be anonymous online → fewer preconceived biases among students

Challenges:

  • How do you transition F2F to online
  • What tools are appropriate / feasible / viable / affordable?
  • How do you do collaborative work when everyone is online at different times?

Format:

  • Cases are presented in multiple formats / media
  • Introductory week to familiarise students with online environment. In addition to learning the content and critical thinking, students also have to learn about PBL
  • Scenarios are released in 2 stages over a 2 week period
  • Scenarios are accompanied by a set of probing questions to stimulate discussion
  • Teacher provides support during the discussions
  • Students must also design their own case
  • Assessment is based on content and depth
  • Wiki used for question / answer. Each student must answer each of the questions, each answer must be different i.e. must add to what has already been added (this means that the question can’t just be a knowledge question)
  • Discussion boards are used for students to dissect the cases (All and Group)
  • Each group assesses their own knowledge base, and define what the gaps are, and therefore what they need to find out (who provides the links to the resources, or can students use any resources?)
  • At least 3 posts per student, including: Summarise and question one citation; Answer another students’ question; Follow up any discussion on their own posts
  • Reading assignment: written, critial appraisal of a published article relevant to the case study. This summary must be posted online.

Important for students to learn how to share information in supportive environments

Assessment:

  • What parts of the process need to be assessed?
  • What parts can be graded as a group?
  • What needs to be submitted for individual assessment?
  • What are the time constraints for the grading?
  • How do you balance grading workload with the need to externally motivate student performance?
  • There is also a syllabus quiz to ensure the students actually know the content

Design:

  • Make the problem compelling
  • Outline expectations
  • The problem analysis should relate to the professoinal field
  • As student proficiency develops, withdraw support
  • Use learning issues to encourage EBP
  • Ensure that solution development is based on critical appraisal

Resources

  • Barrows, HS (1996). Problem based learning in medicine and beyond: a brief overview. New directions for teaching and learning
  • Barrows HS & Tamblyn, RM (1980). Problem based learning: an approach to medicla education. New York, Springer Pub. Co.
  • Hmelo-Silver, C (2004). Problem based learning: what and how do students learn? Educational Psychology Review, 16(3)
  • www.criticalthinking.org

 

The role of clinicians in student assessment

A few weeks ago I was at a workshop attended mainly by clinicians who are involved in student supervision. One of the questions asked was about the role of clinicians in student assessment. I thought it was worth writing a short note about the general feeling in the room, and my own thoughts on the matter.

First of all, we don’t allow clinicians to sit in on the formal assessment (i.e. examinations) of our students, for reasons of objectivity. We feel that in the past, the nature of personal relationships between students and clinicians has either positively or negatively affected assessment outcomes for the students. In fact, one clinician was bemoaning the fact that a student had passed an exam, even though the same student had performed poorly throughout the block. Not that having university staff completely removes bias but we feel that we’re more able to view the assessment without letting personal feelings impact the outcome. In addition, having not seen the student during the course of the placement, we can’t know how the student has performed over the previous month or so and since the exam is purely an indication of performance on the day, we are better able to make unbiased decisions.

I also need to make a clear distinction between the role of the clinician in summative and formative assessment. Clinicians certainly have a role in formative assessment as it relates to teaching. And that’s the key for me. Because of their daily experience on wards and with patients, clinicians have an incredibly valuable role to play in students’ clinical development. I would even argue that their impact is as (if not more) valuable than the role of the academic physiotherapist, for the reasons mentioned above.

However, when it comes to summative assessment i.e. exams, I don’t think that clinicians should be involved at all and not for the reasons I presented in the second paragraph above. The reason I don’t think that clinicians have a role to play in summative assessment is that it’s the university that provides the certification. We are accountable for making the decision of students’ competence and so we should have the final say in examination proceedings. Secondly, the HPCSA sets the curriculum and to some extent, the professional learning outcomes. Clinicians are not familiar with the curriculum, modules or in most cases, the specific learning outcomes of the Clinical Practice module. This is why I don’t think that clinicians should sit in on student exams.

 

CHEC course: teaching and learning (day 3)

Image taken from YMCA cultural diversity office

Yesterday’s CHEC session was presented by Jeff Jawitz from UCT, who looked at tools for addressing diversity in the South African university classroom. I’ve seen Jeff present before at conferences and he’s got a really relaxed way of introducing and working with often highly controversial topics, like race and gender. I was especially excited to have the opportunity to learn more from him during this session. Here are my notes.

———————–

In what ways are students diverse? Which of these matters?

There are many different differences, and any one of these might be highly significant for one person, but insignificant for everyone else → we can’t take all of these into account when we’re working with groups. Yet, we must be aware of all of the differences nonetheless

No single aspect of diversity addresses all of the issues

What does diversity mean in a South African context?

Diversity enriches the classroom

Learning styles (e.g. Felder-Silverman) can be used to change teaching practice to take diversity into account, rather than categorising students. Bear in mind that the most aspects of diversity in education deal with the issue of cognitive diversity i.e. ways of learning, but there are others e.g. language

Language can be used to communicate effectively, but also to engage deeply with the academic discipline. These are two different things and can be developed in different ways (See Cummins, 1996)

“Every time a student sits down to write for us, he has to learn to speak our language, to speak as we do, to try on the particular ways of knowing, selecting, evaluating, reporting, concluding and arguing that define the discourse of our community. He must learn to speak our language. Or he must dare to speak it or to carry off the bluff, since speaking and writing will most certainly be required long beofre the skill is learned” – Bartholomae, 1985, 134-135

Discourses are ways of being in the world, which integrate words, acts, values, beliefs, attitudes and social identities, as well as gestures, glances, body positions and clothes…a sort of identify kit” – James Gee

Where do discourses come from (Gee, 1996, p.137)?

  • Primary – acquired early in life within the socio-cultural setting of the family
  • Secondary – learned / taught as part of socialisations within schools, religious communities and other local, state and national groups

How do discourses cause discomfort among others?

Socio-cultural dimensions of diversity:

  • Race
  • Gender
  • Class (“income diversity”)
  • Religion/culture

What resources do students need in order to complete a task?

How comfortable are you using race as a descriptor?

What are the problems with using race as a descriptor?

What is the value of using race as a descriptor?

“Am I a racist if I think about race in my courses? Shouldn’t I treat all my students equally?” – Milner, 2003, p.176

How does one address the significant difference in retention and graduation rates between black and white students at university in South Africa without reference to race?

When discussing diversity in the classroom, it’s as much about who we are  (i.e. teachers) in that discussion

Authority doesn’t only come from what you know, but also from what you look like. The notion of authority has huge racial overtones in South Africa

Knowledge in the blood” – Jansen

“Race reflection” – Milner, 2003

  • How might my race influence my work as a teacher?
  • How might my students’ racial experiences influence their work with me? What does it mean for a young black student who has never even had a conversation with an adult white male, to be told to come and see the teacher anytime, when that teacher is an adult white male?
  • How do I negotiate the power structure around race in my class to allow students to feel a sense of worth?
  • Am I willing to speak about the injustice of racism in conservative spaces?

“We are a nation struggling to come out of our history”

References

  • Bartholomae, D (1985). “Inventing the university”, in Rose M (ed), When a writer can’t write: studies in writer’s block and other composing precess problems.
  • Cummins J (1996). Negotiating identities: education for empowerment in a diverse society
  • Felder, RM (1993). Reaching the second tier – learning and teaching styles in college science education. Journal of college science teaching, 23(5):286-290
  • Gee J (1996). Social linguistics and literacies: ideology in discourse
  • Milner HR (2003). Teacher reflection and race in cultural contexts: history, meaning and methods of teaching. Theory into practice, 42(3):174-180

Additional resources

Applying theoretical concepts to clinical practice

Concept map about concept mapping taken from IHMC website

I just finished giving feedback to my students on the concept mapping assignment they’re busy with. It’s the first time I’ve used concept mapping in an assignment and in addition to the students’ learning, I’m also  trying to see if it helps me figure out what they really understand about applying the theory we cover in class to clinical contexts. They’re really struggling with what seem to be basic ideas, highlighting the fact that maybe the ideas aren’t so basic after all. I have to remind myself that clinical reasoning is a skill that takes many years to develop through reflection and isn’t really something I can “teach”. Or is it?

For this assignment I wanted the the students to set a learning objective for themselves (I gave examples of how to do this, including using SMART principles of goal setting). They also needed to highlight a particular clinical problem that they wanted to explore and how they would use concepts from the Movement Science module to do this. They needed to describe a clinical scenario / patient presentation and use it to identify the problem they wanted to explore. From that short presentation, they should derive a list of keywords that would become the main concepts for the concept map.

Here’s a list of the most common problems I found after reviewing their initial drafts:

  • Many of them lacked alignment between the patient presentation, the learning objective, keyword / propositions and the final concept map
  • Many of the learning objectives were vague. They really found it hard to design appropriate learning objectives, which meant that their whole assignment was muddled
  • There were two processes going on in the students’ minds: patient management, and their own learning. This assignment was about student learning, but most of the students were focused on patient management. This was especially clear in the learning objective and actual maps they created, which all had a clinical focus on the interventions they would use to treat the patient, rather than the learning concepts they would apply
  • Most of the students created hierarchical maps which failed to identify complex relationships between concepts

After going through their initial drafts, I had another session with them to go through the feedback I’d given and providing more examples of what I expected from them. This assignment is proving far more difficult for the students than I’d expected. However, I’m not sure if it’s because they can’t apply theoretical concepts to clinical scenarios, or if they just don’t have a good understanding of how to create concept maps. I think that they’re having difficulty thinking in terms of relationships between concepts. The maps they’ve been drawing are appropriate in terms of the interventions they’d choose to manage their patients, but the students can’t seem to transfer the concepts from the classroom into clinical contexts.

They’re used to memorising the content because that’s how we assess them i.e. our assessments are knowledge-based. Then they go into clinical contexts and almost have to re-learn the theory again in the clinical environment. There doesn’t seem to be much transfer going on, in terms of moving knowledge from the classroom context to the clinical one. I haven’t researched this yet, but I wonder what sort of graduate we’d get if we scrapped classroom teaching altogether and just did everything on the wards and in the clinics? I understand the logistical issues of an apprentice-based approach to teaching large groups but if we didn’t have classroom time at all, maybe it’d be possible?

Sharing? Collaboration? No thanks

Last week I took our third year students to see a demonstration of the management of a patient with spinal cord injury as part of the Movement Science module that I teach. I noticed that during the demonstration many of them were taking pretty comprehensive notes, and thought that this would be a great opportunity to use a collaborative writing platform to create something useful for everyone in the class.

I proposed the following to them the next day:

  • I’d set up a shared online workspace, either using a wiki or Google Docs and create the document structure so that they’d just have to fill in the spaces from their notes
  • We’d use class time so that this wouldn’t be regarded as extra work
  • I highlighted the benefits i.e. additions to their individual notes from other students, adding multimedia e.g. video and images to enhance understanding, linking out to external sources to strengthen the evidence base, error correction by the group and myself, and creating a potentially useful resource for anyone else in the world

Their response…no thanks. It wasn’t even up for discussion. I found out that they didn’t even planning on typing up their notes, even after I’d pointed out the digital notes are searchable, expandable and shareable. They told me that if they wanted to share with their friends then they’d just photocopy the notes.

These aren’t selfish students, and they’re not limited by access to technology. They just don’t see that sharing in this context has any value for them as individuals, and that’s where I think the problem lies. They think that sharing doesn’t benefit them in the context of their learning (or studying as they call it, which I think is a fundamental problem in itself). They told me that they are connected but only in their social lives. They regarded studying as that thing they do in the classroom, and that learning comes from studying.

I also got the sense that they believe in some way that this is a zero sum game, in the sense that the notes they have will give them some kind of competitive advantage over other students in the class, thereby increasing the odds that they’ll get a higher mark. What it is they’re competing for is unclear. I wonder if grading is somehow related? Grading sets up a system of ranking and competition, not of sharing and collaboration. From that point of view, sharing knowledge is only good if it doesn’t impact on my own position in the ranking system. If you get a higher mark than me, it pushes me down the list. If sharing is seen as a zero-sum game in which your success impacts negatively on my success, then sharing isn’t a good strategy.

Anyway, I was pretty disappointed because I believe that sharing and collaboration has enormous potential for learning. What do I do…force them to share in the hope that they’ll see the light? Even if I design collaborative assignments that requires a sharing component, as long as they see it as work, I’m not sure that it’ll change their thinking.

SAAHE conference, 2011 – day 3

Today was the last day of the SAAHE conference. Coming as it did immediately after a week of the SAFRI programme, I can’t say I’m not glad it’s over. It’s been an amazing experience though, mostly as a result of the wonderful health educators I’ve been fortunate enough to meet. Here are the last of the notes I took during the conference.

Simulation-based asessment: challenges and opportunities by Jack Boulet

Simulations used for summative and formative assessment, as well as curriculum assessment and patient safety

Need to know what health practitioners actually do, (i.e. procedurally), as opposed to what they know

Simulated (standardised) patients are good for some things but not others e.g. trauma

Performance measures:

  • Link measures to scenario events
  • Focus on observable behaviour
  • Incorporate multiple measures from different sources

Types of scores:

  • Explicit process
  • Implicit process
  • Explicit outcome

Checklist for assessing acute scare skills

  • Certain actions are more important than others
  • Sequence and timing are important

Checklists reward thoroughness

Training and quality assurance are important when it comes to assessment and ratings

Developing reliable and valid scoring systems is difficult

Important to identify and minimise errors of measurement

Peer review is essentially about getting a high number of opinions that over time will average out to be an accurate measure

Cases (simluations) are “vehicles” to measure skills

  • Who are the target examinees?
  • Specificity
  • Difficulty
  • Essential manoeuvres and questions?
  • Sampling from a domain (identify the domain)

Predictive validity” – Even with simulation, it’s difficult to establish predictive value → performance in the real world

Challenges:

  • Cost
  • Logistics
  • Setting standards
  • Interdisciplinary skills (e.g. measure doctor-patient interaction but not doctor-nurse interaction)
  • Integration

What is the societal cost of having providers with inadequate knowledge and skills?

It’s more interesting to measure how people lose skills / competence over time, than to measure how they acquire skills

Inferences concering competence are dependent on linking scores to performance criteria

How can we best use technology as part of current clinical / educational efforts?

  • Electronic portfolios
  • Online testing
  • Combined methods

Use of advanced technology to increase fidelity e.g. virtual reality, haptic systems

Students change behaviour when they know how they’re being assessed e.g. with checklists

Simulation studies using confederates (can this be done with students and “broken” equipment?)

Good teamwork is easier to recognise than it is to define


Making use of adult learning theories by David Taylor

Attended a workshop yesterday, which covered much of the same content

Behaviourism – consequences drive actions

Picking up the rules of a community”, “learning the rules of the game”, “what does it mean to be?

How do you learn “enough of the words” to make sense of the discipline?

Exploration of a model based on Kolb’s learning cycle:

Elaborate” new knowledge → consider all propositions and discard ones that are irrelevant, experts navigate this path quickly

  • Work out the most likely resources to refine possibilities
  • Actively participate in the activity
  • Refine the information into a hypothesis

Reflecting / organising:

  • Test – retest the hypothesis
  • Organise information into a “story” that makes sense to them
  • Teachers need to provide cognitive structures upon which students can build → scaffolding
  • Encourage reflection-in-action / reflect while doing

Feedback:

  • Students needs to articulate prior knowledge
  • Assessment is a form of feedback
  • Feedback can only be given when students have articulated / exposed their understanding
  • Teachers must be open to accept (and to act on) feedback from students

Reflect / consolidate:

  • Take on board the feedback
  • Reflection in the light of new knowledge and the learning process
  • Evaluate personal responsibility for learning
  • Teacher needs to provide opportunities for the learner to rehearse / apply new knowledge i.e. encourage reflection-on-action

Dissonance:

  • Using a challenge to help students make a conceptual leap by identifying / proposing an alternative concept that they had not considered
  • Is dissonance a way to help students move through Vygotsky’s ZPD?
  • Mezirow – “learning is a disorienting dilemma”
  • Can be created by manipulating”
    • Resources: should be appropriate, sufficient and relevant
    • Motivation (Knowles):
      • Intrinsic: adults learn because they need to know, have a self-concept as a learner, have life experiences, readiness to learn, orientated towards learning
      • Extrinsic: programme / curriculum, community of practice
    • Stage of development (Perry): Duality (“right and wrong answers and the teacher knows what’s right”) → multiplicity (“comfortable that in any given situation, there’s more than one answer, and that context is what matters most” – comfort in dealing with uncertainty)
    • Style of learning (Entwistle, Biggs): strategic, deep, surface ← how do you measure which of these is happening?

CoP (Wenger):

  • We don’t live or work (learn) in a vacuum
  • Everyone is part of a community
  • We only learn in community (does this mean that it’s impossible to learn independently?)
  • We develop as part of that community


Perceptions and experiences on community engagement as part of learning Student sessions. Points below taken from a variety of student presentations

Train of hope (Phulophepa)

Service learning, research and volunteerism: providing support to about 87 organisations in the area, entirely student run → builds confidence and experience, receive certificates for work done which are valued by future employers

Move from community service to community development

External evaluation bring accountability to projects

Most common health problem encountered by medical students on community-based learning placements is drug abuse

Community dynamics:

  • lack of medication
  • non-compliance
  • traditional healers
  • religion and beliefs
  • social problems

Patients’ stories are often heartbreaking

The patient is more than the illness”

Poor of the poorest”

Medical doctors and sangomas (traditional healers) have areas of overlapping practices and principles

As long as patients are living in this world, they are appreciated as human beings”

Challenge students to move out of their comfort zones

Community-based projects count for a very small percentage of the coursework grade, yet it takes an enormous amount of personal commitment and time, and is supposedly valued by educators

Language is a significant problem for student-patient interaction

Students conflicted when we tell them to think outside the box, but then have expectations for them to “do it by the book”

Does the institution learn as much as the students from the community experiences? When students report back to the institution, what changes do they experience as / if the institution responds? Does the institution respond?

SAAHE conference, 2011 – day 2

Social accountability: the mark of excellence in health professional education by Charles Boelen

Moved from interrogation → an assertion

What is a health professional?

Flexner’s report changed the scope of medical education by introducing a scientific background, especially in education

Medical education should be patriotic (Flexner)

Educating → health professionals → for a strategy → to meet people’s needs (must begin with identifying people’s needs); the 4 layers are not necessarily closely correlated

There are many influences on the process, besides the input we provide in our HEIs

What are people’s needs and values today? What will they be in 20 years time? We’re training professionals for today and tomorrow

Not teaching for health professionals, but change agents

Social accountability: the obligation to direct their education, research and service activities towards addressing the priority health concerns of the community, region and / or nation they have the mandate to serve. The priority health concerns are to be identified jointly by governments, health care organisations, health professionals and the public”

Priority health concerns:

  • quality (ideally, we provide the best to everyone)
  • equity (who are the most vulnerable?)
  • cost-benefit (resources are fixed and we have to work within them)
  • relevance (what are the most relevant problems to address?)

As educators, we can’t achieve the values above on our own, we need to work with others

Functions of a school:

  • Education
  • Research
  • Service

Stakeholders (Boelen, 2000, Towards unity for health, WHO):

  • Policy makers
  • Health professions
  • Academic institutions
  • Health managers
  • Communities

Social accountability (evaluate impact of commitments)

Social responsiveness (more explicitly aware, use data to drive action)

Social responsibility (awareness)

We should accompany our graduates when they leave our schools

Be critical of the health system

Conceptualisation of professional (role of the school) → production of professional (educational interventions) → usability of professional (society’s satisfaction). Using a commercial model to explore ideas

Pay more attention before beginning training, as well as afterwards

Boelen & Woollard, 2009. The CPU model: conceptualisation-production-usability. Medical Education, 43: 887-894

The concept of social accountability helps us to answer the question: “Why are we doing all of this?”

Global consensus for social accountability of medical schools (www.healthsocialaccountability.org)

A socaily accountable medical shool should:

  • respond to current and future health needs
  • reorient education, research and service accordingly
  • strengthen governance and partnership with other stakeholders
  • use evaluation and accreditation to determine impact

Preparing for the future:

  • ethics → impact
  • democracy → transparency
  • globalisation → competition
  • public support → synergy

What is our common purpose?”

 

Developing enterprising health care professionals. How should we meet the challenge? By Prof. Deborah Murdoch-Eaton

Enterprise:

  • making the most of opportunities
  • identifying areas for improvement
  • developing innovative solutions
  • implementing and refining strategies

Health professionals will always work in a business, whether private or public. A business is only as good as it’s employees

Need confidence to implement changes

Reference to “Health professionals for a new century. Frenk et al, The Lancet”

Provoke” global reforms in all health professions

Core role of universities as social institutions

Need to be effective in an ever-changing environment and knowledge-base

Change in accordance with local needs

Graduate attributes = set of core skills that are not discipline-specific, “meta-skils”

  • Communication
  • Higher order reasoning
  • Critical thinking
  • Ability to use technology

Students should not all fit into the “same box”, need to develop those who can lead and innovate

Structured (and repeated) opportunities to practice skills

Incorporate self-reflection and appraisal of own abilities, provide evidence of own learning / experiences

Soft skills:

  • teamwork
  • creativity
  • project planning
  • presentation skills
  • IT skills
  • time management
  • networking
  • negotiating
  • leadership

Be enterprising in teaching style and assessment

Embedding of enterprise:

  • strands
  • blocks / modules
  • extra-curricular

How do we colleagues to collaborate?

What are the persuasive arguments?

Fit for (whose) purpose?

Empower students to help retain them in the country

Need to give guidance and time to think and reflect deeply, not related to “competence-driven outcomes”

Values, cultures and ethics are hard to measure

new knowledge is regulated by measuring it against exising scholarship through the process of peer review, rather than the extent to which it meets the needs of those external to the field”

Teach students to ask relevant questions, empower them to become change agents

 

Blackboard training initiative by A Botha

Staff training using Blackboard within institution

(People think that) Blackboard can:

  • support good teaching practice
  • support learning styles
  • encourage collaboration
  • keep abreast of learning technologies

Staff were overwhelmed and reported needing extra time for training

Trying to improve quality of people trained, rather than getting numbers through the system

The problem with LMSs is that they don’t take the student into account, the focus is on the lecturer, the content and the course

What is the relationship between activity on the LMS and the throughput?

 

E-learning: student’s perspective by G Sinombe

Used to increase access to course materials and information, increase contact and participation in courses and enhance variety of learning styles ← blended approach

Courses (reading material, notes, assignments, etc.) uploaded ← not e-learning (by whatever definition you choose to use for “e-learning”)

Student responses differed, some attempted to use, some didn’t try at all

Why do some students not try? Study aimed to determine student perceptions and challenges that impacted on their use

Student responses:

  • good tool
  • good for slow learners
  • enhances communication between lecturer and student
  • flexibility in time and place
  • easy access to courses
  • hard to find unoccupied computer
  • network slow, when available
  • hard to access off-campus
  • helps me to share ideas outside of the classroom
  • enhances sharing of ideas

Is “e-learning” necessary? How does it improve performance? This has to be taken in context…what is the context in which you want to use it?

 

Students’ approaches to learning disciplines in an integrated curriculum by D Manning

How many students achieve pass marks while scoring below the minimum in individual subjects?

In how many disciplines are passing students scoring below the pass mark

Students going into clinical years with substantial gaps in their knowledge

What is going to be done about this?

Is there a need for discipline based subminimums and consideration of logistical solutions

 

Ready for the catwalk? By what criteria should a new model be judged? By F Cilliers

How does assessment influence learning? Cilliers et al, AHSE, 2010, December

One or more sources of impact, through one or more mechanisms, leading to one or more learning effects

Test the model in a different context to what it was designed for i.e. model developed in theory, then evaluated in clinical practice

Quality of learning” ← how was this defined?

Interaction with preceptors:

  • regular accountability
  • personal consequences
  • emotional valence

Preparing out of a fear of humiliation, rather than a desire for understanding. Just wanted to “survive” the ward round

“You will learn because you don’t want to continue to live in fear”

Tyrants” and “teddy bears” ← categories of supervisors

You don’t want to disappoint the “teddy bears”, go out of your way to participate, gather more information

But they are still motivated by others, rather than for themselves. Did any students report a desire for understanding based on what it would mean for their own clinical practice?

Teddy bears” = scope and safety to ask questions and explore areas of personal interest

 

Reviewing assessment to promote medical student engagement in basic sciences to cure and care better by C Brand

Curricula are living, self-organising organic systems

De-load” some of the course content

Threshold concepts = conceptual gateway that opens up previously inaccessible domains

All change begins with conversation

 

Introduction of a new clinical assessment: did it achieve it’s aim? by L Green-Thompson

Students arrive in the clinical environment unprepared

Introduced an observed examination (mini-CEX) and then evaluated student perceptions of the exam

Landscape of assessment”

Students reported that the clinical exam was a valuable experience: “a good opportunity to see how far I had come”

Assessments are theoretical events, rather than clinical events”

 

Workshop: Making use of adult learning theories by Dr. David Taylor

Theories not mutually exclusive, each have something to recommend them

Instrumental learning theories:

  • Behaviourist → stimulus – response (e.g. Skinner)
  • Cognitive → mental process not behaviour (e.g. Piaget, constructivism)
  • Experiential → behaviour in practice (e.g. Kolb)

Humanistic theories:

  • Andragogy → need, readiness, motivation (e.g. Knowles)
  • Self-directed learning → what about the social context?
  • Transformative learning → disorienting dilemma that drives learning, context, critical reflection (e.g. Mezirow)

Situated cognition (e.g. communities of practice):

  • Learning and thinking are social
  • Structured by tools available
  • Dependant on situation
  • Knowing is supported by doing (e.g. Wenger)

How difficult it is to “get into” a new discipline. Need to acquire an understanding of the profession before you can “be in it”

The challenge for a learner is to find out enough about the subject to enable learning

Expectancy valence theory: low expectation of success results in poor motivation for learn, unless rewards are overwhelming → can’t give students tasks that they can’t achieve

  • You need to recognise what you already know
  • You need to know how the new knowledge fits in
  • Dialogue makes it easier

Feedback will drive future learning

What are the responsibilities of the learner and teacher respectively in each phase of a learning cycle?

Dissonance (putting the cat among the pigeons):

  • Teacher’s responsibility to challenge student to move outside of their comfort zone
  • Learner needs to be open to being challenged, need to identify their own gaps, pre-conceived perspectives impact on worldview

Critical friendship”

Teachers don’t need to be responsible for the answers, but they need to enable the discovery

Hypotheses that can’t be tested are useless (Poppper)

It’s possible to be critically reflective on your own but feedback is important for error correction → create your own dissonance

Laurillard: feedback is a continuous process that occurs as part of a 2-way dialogue (“conversation”)

Feedback sandwich” 🙂

SAFRI 2011 (session 2) – day 3

Began today with a session on workplace-based learning, spent some time “developing a model” for learning a new skill after actually trying to learn (what was for many) a new skill…spinning a top. My group came up with the following model which, truth be told was obviously based on Kolb’s learning cycle.

Reviewed educational model of Teunissen et al (2007)

Models are fluid frameworks that help to develop understanding, not algorithms that need to be followed

Readings:

  • Teunissen et al, (2007). Attending doctors’ perspectives on how residents learn. Medical Education, 41: 1050-1058
  • Teunissen et al (2007). How residents learn: qualitative evidence for the pivotal role of clinical activities. Medical Education, 41: 763-770

 

One minute preceptor / 5-step micro-skills

Has led to modest improvements in teaching skills (Furney et al, 2001), has been found to be equal to or better than traditional methods of clinical teaching in time-constrained environments e.g. bedside, ward rounds (Aagaard et al, 2004).

The one-minute preceptor is a framework (Neher et al, 1992):

  1. Get a commitment: statement of understanding / intent, should be verbal (“What do you think?”)
  2. Probe for supporting evidence: question student for further depth / detail i.e. probe the statement (“Why do you say that?”)
  3. Reinforce what was done well: provide feedback on appropriate behaviour / performance
  4. Give guidance about errors or omissions: error correction
  5. Teach a general principle: extrapolate the situation / event to more general terms
  6. Conclusion: end with clear steps for moving forward

Readings:

  • Furney et al, 2001. Teaching the one-minute preceptor: a randomised controlled trial. Journal of General Internal Medicine, 16: 620-624
  • Aagaard et al, 2004. Effectiveness of the one-minute preceptor model for diagnosing the patient and the learner: proof of concept. Academic Medicine, 79: 42-49
  • Neher et al, 1992. A five-step “microskills” model of clinical teaching. Journal of the American Board of Family Practice, 5: 419-242

 

Portfolios

What is a portfolio? A collection of learning objects / experiences, aggregated over time, directing at documenting the achievement of developmental objectives. Demonstrates progress. Must be measurable although this is hard to do.

Short session on designing portfolio tasks, asked to design a task for something we currently teach. This is a useful framework for an assignment. I chose a task for Movement Science:

  • What – To understand the relationships between concepts in movement science and apply those concepts to clinical practice (challenging for students)
  • Activity – Develop a concept map of an activity (e.g. observed movement of a patient with a movement disorder), using concepts from module to explore / explain the activity
  • Where – Clinical setting / patient encounter
  • How – Short patient history, with an associated concept map
  • Reflection – Each link on the concept map must have a note explaining the observed movement / activity, it’s relation to the module concept, and be linked to the patient history
  • Measure – concept maps can demonstrate understanding of relationships between concepts (theory) and their application to practice (patient presentation)

Type of learning taking place in above example:

  • Clinical reasoning
  • Professionalism (i.e. note taking and patient presentation)
  • Reflective practice
  • Interpreting data

Assessment is a snapshot of learning

Students worry that they didn’t learn something that you might ask them in an assessment

Be strategic in what you’re going to measure

Portfolios are about “doing”, there should be consideration of patient management involved

Portfolio assessment is often not feasible in a resource-constrained environment as it can be labour intensive. The document should serve as an indicator to guide assessment of the student. It must assess something different to what is already being assessed.

Assess reasoning. Knowledge is better assessed with other methods e.g. MCQ

Portfolios are not just another thing to do

 

Reflection

Reflection raises awareness → exploration of alternative methods

Asked to do a short reflection on personal / professional development since beginning with SAFRI programme:

“My SAFRI project began as a study that would look (in retrospect) quite superficially at the use of a social network to development clinical / ethical reasoning skills.

Over time it became clear that the method involved little more than describing “what students did”, which I found deeply unsatisfying. “Is there more to this?”

I began looking into theoretical frameworks that could help to structure the research. I’m now analysing the data with a structure (i.e. theoretical framework) that is helping me develop a deeper understanding of the process.

This has played a role I changing how I think about research. I find myself questioning a lot more in other parts of my life…”Why is it like that?”

We discussed the challenges of evaluating personal (and often deeply personal) reflections. Giving marks clouds the issue, as students then write what they think you want to hear in order to get a better mark, in contrast to sharing honest, open, authentic experiences.

It’s important to link their reflections with expected outcomes

In the community block that I supervise, students share and discuss their reflections in a group. I do this because I think it’s important to hear what others are going through (i.e. to “normalise” the experiences and emotional responses). I understand that this can change the reflections that students produce i.e. will they be honest and open if they know the reflection will be shared? I’ve found that students give still give honest reflections when they realise that the feedback is non-judgemental and that it can help them to move to deeper understanding

Does reflecting actually change practice? Is there a difference between what they say and what they do?

SAFRI 2011 (session 2) – day 2

“Teach” a group of colleagues about “Assessment”. Here are some notes I took in preparation for a 5 minute teaching session

What is assessment?

“Defines for students what is important, what counts, how they will spend their times and how they will see themselves as learners. If you want to change student learning, then change methods of assessment” (Brown, Bull and Pendlebury)

Integrate assessment into teaching and learning in a way that includes students in the process

Feedback is an important part of assessment

  • Should be frequent
  • Must identify gaps and provide direction to help students close the gap
  • Stimulates deep learning
  • Feedback should be task-centred, not emotional / personal

Assessment linked to outcomes i.e. are we assessing what we say is important for our graduates to be able to do (alignment)

Students can be involved in self-assessment and peer feedback but it needs to be scaffolded / structured. Should be used for different reasons

Personal and professional development. Do we assess for good physio’s or good people? Peer assessment can be used to encourage formation of professional behaviour and interpersonal dimensions

Move away from a testing culture to an assessment culture i.e. away from an emphasis on procedures and products of assessment to emphasis a process of assessment → deep learning. Difference between “knowing what” and “knowing how”

Readings:

  • Assessment practices that improve teaching and learning (Luckett & Sutherland)
  • Assessment in medical education (Epstein)

Feedback as an educational tool

Start with the positive, then move on to negative. Also, feedback doesn’t have to even include a negative. We can also use it to highlight good work / understanding. As long as it is always given with the intention of moving the student forward. I would argue that splitting feedback into “positive” and “negative” might have little value anyway. Students will often latch onto the “negative” and forget anything positive you opened with. Can feedback be neutral, aiming only to highlight how the student can move forward?

Feedback should always be aimed at “closing the gap” between what is currently known and where the student should be i.e. it should always be formative

It should not be personal or have a value judgement assigned. The purpose is to identify a problem with a behaviour / skills / competency /etc. not with the person

Identify what they do know, what they don’t know and how to improve

Use feedback as an opportunity for self-assessment e.g. How do you feel about that? What do you think you could have done better? What did you do well?

Needs to be given as soon as is appropriate. Some feedback may be better given in private, or in a group, depending on the context

Be kind / sensitive / aware / empathic

Acknowledge if you are at fault e.g. arriving late for a feedback session / tutorial

Be aware of power relationships

Feedback should be continuous

Follow up on your feedback

The feedback should be written down at some stage, either by the student or teacher. Verbal feedback given in the moment may not be remembered later

It’s not only important to identify what students got wrong, but also to try and determine why they got it wrong

 

Curriculum development

Whose needs does the curriculum address? Ultimately, the community who will be served by the graduates of your course → the aims & objectives of the course must be aligned with community needs

Harden’s 10 Question model for curriculum design

“How is the detail of the curriculum communicated”, especially to first year students? Are students aware of how what they’re doing right now is relevant to the course they signed up for e.g. why am I studying physics, I want to be a sports physio?

“Blueprinting”

SPICES model of educational strategies:

  • Student-centred ↔ Teacher centred
  • Problem-based ↔ Information gathering
  • Integrated ↔ Discipline-based
  • Community-based ↔ Hospital-based
  • Electives ↔ Standard programme
  • Systematic ↔ Apprenticeship-based or Opportunistic

There are models that can be used to determine where a curriculum lies on each of the spectrums listed above

Is a curriculum set in stone, or is it dynamic?

“Diseases of the curriculum” (Abrahamson, 1960s)

You can’t teach everything (there’s just too much) but you can help students become self-directed learners, which is what clinicians are. We don’t have all the answers but we know how to find the answers that will help fill the gaps in our knowledge / understanding. If students think they’re supposed to know all the answers, then that’s an enormous burden to carry

How do we select students who had a poor secondary education but who have the potential to be good (great?) therapists / clinicians? How do you support them?

You should always look for competence in students, no matter what curriculum design you use

A discipline-based approach doesn’t challenge students to engage with the later stages of Bloom’s taxonomy

Problem-based learning helps to create a more authentic learning environment / experience and allows the integration of pre-clinical and clinical science

“The contribution of South African curricula to prepare health professionals for working in rural or under-served areas in South Africa: a peer review evaluation. SAMJ, 2011, 101:34-38

What happens to students when they leave? Should this be a question we consider during curriculum planning?

Learning:

  • Informative
  • Formative
  • Transformative

“If you can’t explain it simply, you don’t understand it well enough” – Einstein

SAFRI 2011 (session 2) – day 1

A few days ago we began the second SAFRI* session of 2011, which will lead into the SAAHE conference** later in the week. Every day I take notes and will try to put them up as we go along bearing in mind that a lot of what we do is workshop-based. The notes are a combination of points given by presenters, and my own reflections that were sparked by something that someone said. My thoughts are in italics.

Achievement: changed the way I think about the world (word = clinical education)

Challenge: helping others to see the world the way I do

Never give up…or Give up often? Come up with lots of ideas, some will be good, some not so good, some terrible. Test them all (even if only mentally) and throw away the ones that don’t stand up to being tested. Analogy with digital cameras and taking loads of photos because the cost is zero and you can delete the poor ones.

Find the underlying principle that can be generalised to many contexts i.e. details aren’t necessarily important

Why did I miss the diagnosis? (Bordage, 1999) → “Less is better” i.e. foundations are good to build on

We tend to focus on student behaviour, instead of their learning e.g. “students must attend class and pay attention”…but if you’re not interesting, then why should they attend? What is it about their presence that somehow ensures that “learning happens”? If they’re not going to listen to you (and if they can pass the exam without attending), then why do we place so much emphasis on their presence?

Teach an approach to discovery, rather than a list of things

Dual processing theory (a universal model of diagnostic reasoning)

  • How do we reason through clinical problems?
  • Make observations and identify signs / variables
  • Query your existing database i.e. your pre-existing knowledge
  • Identify associations between the observed variables and your own database i.e. pattern recognition
  • If existing knowledge is insufficient, query an external database (e.g. patient / textbook)

If you go through the above process enough times, you “get experience” → add more patterns to your internal database

I’m sure the above process is more eloquently and comprehensively described elsewhere

Dancing naked in the mind field” – title of a book that perfectly describes why I blog…putting my thoughts, reflections and experiences out there and by doing so, exposing myself while sharing.

Having a diagnosis frees you from having to think. This has implications for when you’re tired / stressed / pushed for time, in that in those circumstances you can’t think and so latch onto a diagnosis. Students experience the same thing when they’re looking for answers. Having the answer means they don’t have to think because thinking is hard and places a high demand on system resources.

There’s a strong emotional response / association with diagnoses that are made intuitively i.e. without an analytical reasoning process

Talking out loud externalises a reasoning process that is often obscured and hidden from the student

“Diagnostic error and clinical reasoning” (Norman & Eva, Medical Education, 2010)

“construct referenced” as it relates to feedback?

Black , P. & William, D. (1998). Assessment and classroom learning, Assessment in Education, 5, pp. 7–75

Readings:

  • “Classroom assessment techniques” (Angelo & Cross)
  • Rushton, A. (2005). Formative assessment : a key to deep learning ? Medical Teacher, 27(6), 509-513
  • Nofziger, A. C., Naumburg, E. H., Davis, B. J., Mooney, C. J., & Epstein, R. M. (2010). Impact of Peer Assessment on the Professional Development of Medical Students : A Qualitative Study. Academic Medicine, 85(1), 140-147

*SAFRI (Southern Africa FAIMER Regional Institute); FAIMER (Foundation for Advancement of International Medical Education and Research)

**SAAHE (South African Association of Health Educators)