The role of clinicians in student assessment

A few weeks ago I was at a workshop attended mainly by clinicians who are involved in student supervision. One of the questions asked was about the role of clinicians in student assessment. I thought it was worth writing a short note about the general feeling in the room, and my own thoughts on the matter.

First of all, we don’t allow clinicians to sit in on the formal assessment (i.e. examinations) of our students, for reasons of objectivity. We feel that in the past, the nature of personal relationships between students and clinicians has either positively or negatively affected assessment outcomes for the students. In fact, one clinician was bemoaning the fact that a student had passed an exam, even though the same student had performed poorly throughout the block. Not that having university staff completely removes bias but we feel that we’re more able to view the assessment without letting personal feelings impact the outcome. In addition, having not seen the student during the course of the placement, we can’t know how the student has performed over the previous month or so and since the exam is purely an indication of performance on the day, we are better able to make unbiased decisions.

I also need to make a clear distinction between the role of the clinician in summative and formative assessment. Clinicians certainly have a role in formative assessment as it relates to teaching. And that’s the key for me. Because of their daily experience on wards and with patients, clinicians have an incredibly valuable role to play in students’ clinical development. I would even argue that their impact is as (if not more) valuable than the role of the academic physiotherapist, for the reasons mentioned above.

However, when it comes to summative assessment i.e. exams, I don’t think that clinicians should be involved at all and not for the reasons I presented in the second paragraph above. The reason I don’t think that clinicians have a role to play in summative assessment is that it’s the university that provides the certification. We are accountable for making the decision of students’ competence and so we should have the final say in examination proceedings. Secondly, the HPCSA sets the curriculum and to some extent, the professional learning outcomes. Clinicians are not familiar with the curriculum, modules or in most cases, the specific learning outcomes of the Clinical Practice module. This is why I don’t think that clinicians should sit in on student exams.

 

Posted to Diigo 08/21/2011

    • “ ‘Thunks’ are deceptively tricky little questions that ‘make your brain go ouch!’

      Ian’s examples of ‘Thunks’ include:

      • ‘Is a broken down car parked?’
      • ‘If you read a newspaper in the newsagents without paying for it, is it stealing?’
      • ‘Do all polo’s taste the same?’
      • ‘Can you be proud of someone you have never met?’
      • ‘Is a hole a thing?’
    • Active Learning-I ask students to use their prior experiences (interests, hobbies, coursework) in a service learning project for my class, using what they know, and connections they have been able to make, to benefit others. They propose their own projects.


      This idea has all 5 components:
      a. Incorporating peer feedback into a course.
      b. By giving students the opportunity to learn how to give feedback to classmates’ works-in-progress, this motivates students to perform at the same or higher level or their peers in a non-threatening way.
      c. It is also an activity which reinforces learning of the subject matter (active & challenging).
      d. It can create a sense of support and community because the students are helping each other improve their own works (and grades!)
      e. And peer feedback allows students to gain an appreciation for classmates’ efforts and a glimpse of the instructor or professor’s role.

    • When students are stressed by the lesson, find a way to show how you (the teacher) messed up learning the topic
    • First day of class: When introducing class & trying to establish rapport, have the students say where they see themselves in 3 to 5 years
    • I give several smaller “low-risk” assignments in addition to “high-risk” mid-terms and finals
    • Share with my students some of my personal stories as a student, my stories and failures
    • I teach a 3rd year/level class and on the first day of a particular subject, in this case hydraulics, I give them an exam of what they should know from 1st year. I eventually let them take it home to complete it on their own
    • review some student work in class in front of the class, inviting student input. It challenges many students to work harder on their own work
    • I give an exam which I grade and return as soon as feasible to the students. I identify the learning outcomes that most students missed and offer students another test that focuses ONLY on those learning outcomes using different test items. They know they have a chance to improve their exam grades with the subsequent test score.

    • apply newly acquired knowledge to themselves or someone they are familiar with
    • In many sciences, students are concerned about memorizing information (ex. Periodic Table). I tell them that we never used to memorize an atlas, but we learn how to use it as a tool/reference
    • At the end of each lecture topic, students hand it a slip of paper with two items: “Got it”-what they understood and “don’t get it”-what they don’t understand. Once a week, results are posted on Moodle and explanations given for the “don’t get it” items
    • Collaborative learning: I found that students learn better and smooth out their rough edges when they combine conflicting analysis to produce a bounded unit of learning
    • When composing homework assignments, I combine questions of varying difficulty. I ultimately include a problem or two beyond the difficulty required for my course and offer extra credit for solving these problems. The number of students that choose to step up to the challenge is so incredibly refreshing and motivating
    • Have students take responsibility for their learning by applying lesson concepts to their occupations, field of expertise, and personal experience
    • The student gets to choose a topic they feel they could teach the class (for 15 minutes), they become the expert with certain guidelines to follow. Motivation—they are the “star” for 15 minutes. Active learning—they research. Task—they choose the topic. Community—they all practice with each other to get feedback before their 15 minute presentation. Holistic—they learn all types of things; respect, confidence, professionalism, body language, etc.
    • In a writing course, students receive each others drafts throughout the semester and one by one the whole class peer reviews the drafts. Students learn from other student papers and gain critical feedback on their own paper. In other words, every paper is read by every student, and every student must provide feedback.
    • Start each class with a “hook”—something that is contextual and related to the day’s concepts—provides relevancy and captures interest and involvement
    • I engage students to choose a concept from the course and teach a segment of the class.
    • Take students’ pictures 1st day of class and memorize their names. Call students by name from the 2nd class on. Use their names frequently. This instills community and aids in engagement because students cannot hide
    • Allow students to text you with questions
    • I am a student—I am tired (flight came in late). I am hungry (I missed lunch), I can’t concentrate (my daughter keeps texting me). How will you engage me? Tell me a story, a story with characters facing shocking/new or interesting challenges. Then ask me about it, allowing me to be a little grumpy, but encouraging me to continue to participate and reward my efforts. I will cheer up and feel transformed by your class session
    • “Self-correcting exams”: Students are able to alter test answers after the fact—through additional research/working with other students—and resubmit exam responses for additional partial credit.
      • Can the Ethics oral exam use a similar approach? After the exam students go and write a short reflective essay on what was discussed during the exam?
    • Clarify how knowledge that is being taught is important to students’ future work and/or life. Students will value only what they believe they need to know and will use in work/life
    • Problem based learning in groups. Assigning a facilitator in the group and then pick an idea that there isn’t a right answer. You can use concept mapping. First, students present the problem. Second, they find resources and list them. Third, they investigate solutions to the problem and list them. Fourth, they identify the three best solutions. Last, they pick the best one and provide the reason
    • I ask my students (every week), why are you here? I note week one and compare it with the final week. Typical observation: growing ambition.

    • Motivation and Value: I get my students to stop thinking like a student, rather think like a teacher (or a professional in their field) to see the value of the activities and assignments in the course
    • People respond to genuine concern for their well-being
    • Active Learning: I occasionally divide the assigned chapter up among the students, and give everyone 10 minutes to come up with a 1 minute presentation to the class. The students have at least read a few pages, and they talk to each other instead of listening to me
    • When a student engages in course dialogue, I write a note thanking them as their classmates and I enjoyed their participation and how important their questions and comments are to educators and their students. I also include that I hope they continue to share in all courses. This helps encourage participation, especially from those who seldom do
    • Recognize and sustain the “cool” in everyone and challenge the students to move from “pretend cool” to “cool.”
    • Remove fear/anxiety and increase expectations of success by: allowing students to redo/resubmit work or problems for regarding/assessment. They work and learn from their mistakes, but are not punished for their mistakes.
    • Living Concept Maps: During a case study, have the students in groups of 4-5, actively create a concept map as the case study is unfolding—adapting and changing it as new data becomes available. At the end, share the concept maps and see how different/similar they are and explore those.
    • the selection and use of technologies for teaching and learning is driven as much by context and values and beliefs as by hard scientific evidence or rigorous theory
    • There are deep philosophical, technical and pragmatic challenges in trying to provide a model or set of models flexible but practical enough to handle the huge range of factors involved
    • theories and beliefs about education will influence strongly the choice and use of different technologies
    • it is a mistake to focus solely on the educational characteristics of technologies. There are social, organizational, cost and accessibility issues also to be considered
    • what is best done face-to-face and what online, and in what contexts? What is the role of the human teacher, and can/should/will he/she be replaced by technology?

Posted to Diigo 09/19/2010

    • Survey results indicate that the majority of college students prefer courses that offer podcasts over those that do not
    • Students cite convenience, flexibility, and positive impact on learning as the main reasons to have recorded lectures (Nagel, 2008; Fernandez, Simo, & Sallan, 2009)
    • Lecture capture provides additional resources for students: archived lectures, tutorials for lab work, demonstrations of difficult concepts and complex procedures, and presentations by guest speakers;

      • allows students to review material at their own pace and convenience (Coghlan et al., 2007);

      • offers students more flexibility in note-taking;

      • makes time for active learning during class by having the lecture available for viewing before the class meetings (Lund, 2008);

      • allows students to catch up with a missed lecture;

      • offers another tool for student learning projects (e.g., student-generated podcasts for interviewing locals and sharing with peers in a study-abroad program).

    • Although many anecdotal reports about lecture capture circulate in higher education, only a few notable evaluation studies have investigated its impact on student learning
    • No noticeable impact on students’ class attendance
    • In one study, students explained that they continue to attend lecture because it offers opportunities for interaction in a structured learning environment (Copley, 2007)
    • Offering podcasts of lectures has the potential to improve students’ mastery of course material
    • Undergraduate students have reported in focus groups and surveys that podcasts helped them stay focused on the course, made learning more fun and informal, supported independent learning, and enabled deep engagement with course material (Edirisingha & Salmon, 2007; Duke University, 2005)
    • several perceived benefits of lecture capture technology, including the opportunity to make up a missed class, the convenience of watching lectures on demand, and the potential for increased retention of class materials (Veeramani & Bradley, 2008)
    • when students do use podcasts, they use them for specific reasons and at specific times in a course of study
    • Students report that they appreciate the flexibility of accessing podcasts anywhere and anytime (Fernandez, Simo, & Sallan, 2009; Winterbottom, 2007)
    • students who viewed video presentations of lectures with slides tended to back up and repeat slides containing information that they had difficulty understanding (Dey, Burn, & Gerdes, 2009)
    • Although students may appreciate podcasts, this technology tool should not replace traditional learning resources but, instead, should serve as a complement to them (Fernandez, Simo, & Sallan, 2009)
    • studies show that lecture recordings (audio or video) are used mainly to prepare for exams and review course material in order to gain a better understanding of complex topics
    • students usually view podcasts shortly after a lecture has occurred and in the few days before an exam
    • It is important not to assume that all students possess the same technology skills and have had equal exposure and access to technology
    • students without experience and comfort using techonogy may be disadvantaged when class materials must be retrieved from the Web
    • If the podcast will include students’ questions or responses, you should ask students to sign a consent form when the podcast audience is broader than the class itself
    • The availability of lectures in podcast form may change students’ learning behaviors as well as their expectations about the use of class time
    • Since students take fewer or summary style notes in courses using lecture capture (Brotherton & Abowd, 2004), they have more time to process course material on the spot, which may lead them to ask more questions and want more interactivity during lecture
    • Students may, therefore, expect the format of lectures to shift from a process of information transfer to a more student-centered and interactive format
    • As a result, instructors may need to devote more time to in-class activities that enable students to practice skills, think critically about material, and apply what they’ve heard in lecture to grappling with real-world problems (McKenzie, 2008)
    • Recommendations for Using Lecture Capture Effectively
    • 1. Before you start, make sure that you have clear goals for podcasting lectures and the time to prepare them consistently throughout the entire semester

    • 2.Once you decide to podcast, make time to experiment with recording quality. Poor sound quality may prevent students from using the resource
    • 3. Make podcasts available as soon as possible after a lecture
    • 4. If you require students to listen to podcasts before lecture, provide them with content-related questions or other learning activities
    • 5. When podcasts are assigned in advance of a class meeting, use class time for interactive discussion, student-centered learning activities, or demonstrations to complement and build on podcast content
    • 6. Before making podcasts available, be sure that all of your students have access to and are comfortable using devices to download and play podcasts
    • 7. When appropriate, make reference to podcasts during lectures or when responding to students’ questions so that students will be more likely to use them
    • 8. Make accessing and using podcasts easy and fast by providing detailed instructions for downloading and ensuring that the file format is compatible with common media-playing devices
    • 9.Provide students with a clear explanation of instructional goals
    • 10. Draft an evaluation plan for your lecture capture project to investigate what did and didn’t work for you and your students

Some thoughts on assessment in the Ethics module

This post is a result of a few short notes I made on my phone a few months ago, and then forgot about. I came across it the other day and thought I’d put it up here.

The way we currently assess students has no equivalent in practice. For some of our modules (e.g. Management), it makes more sense to allow students to write the test in a computer lab where they have a set time limit but can find what they need online. When would anyone of us be expected to know how to write up a business plan without being able to consult other sources? This would allow us to highlight in another way, that memorising content is not an appropriate strategy to be an effective physiotherapist in the real world. In case anyone is wondering, I’m not suggesting that students don’t need to be able to recall facts…they will obviously still need to memorise some stuff, particularly in the clinical components of the curriculum (e.g. you need to know what hypertonus is before you can address it). But there are definitely some areas of the curriculum where memorising facts is an terrible idea.

The “Professional Ethics” module is one of these areas. Currently assessed in a written examination format, this module has so many alternative approaches that could be more realistic and effective, including small group discussions (my current preference). Students would have to prepare in advance, familiarising themselves with the main concepts that the exam will cover (which would be provided to them beforehand). The exam could take place over a 30 minute period with the examiner guiding the conversation through a set number of topics that cover various important themes in professional ethics and ethical reasoning. Each group of students would certainly have a different conversation, so consistency would be hard to determine e.g. did all groups actually have the same exam?. A marking rubric is one way that could be used to ensure that each student is assessed on a range of the same themes as every other student e.g. participation, comprehension, communication skills, conflict resolution, etc.

I imagine a series of a few of these small group discussions taking place over the course of the year, potentially with different formats and objectives. There would be no summative assessment at the end of the year, only a few formative assessments during. I”m going to start putting together a proposal for the department to change the way the professional ethics module is assessed.

Any feedback or comments around this idea would be most welcome.

Assessment in an outcomes based curriculum

I attended a seminar / short course on campus yesterday, presented by Prof. Chrissie Boughey from Rhodes University. She spoke about the role of assessment in curriculum development and the link between teaching and assessing. Here are the notes I took.

Assessment is the most important factor in improving learning because we get back what we test. Therefore assessment is acknowledged as a driver of the quality of learning.

Currently, most assessment tasks encourage the reproduction of content, whereas we should rather be looking for the production of new knowledge (the analyse, evaluate and create parts of Bloom’s top level cognitive processes).

Practical exercise: Pick a course / module / subject you currently teach (Professional Ethics for Physiotherapists), think about how you assess it (Assignment, Test, Self-study, Guided reflection, Written exam) and finally, what you think you’re assessing (Critical thinking / Analysis around ethical dilemmas in healthcare, Application of theory to clinical practice). I went on to identify the following problems with assessment in the current module:

  • I have difficulty assigning a quantitative grade to what is generally a qualitative concept
  • There is little scope in the current assessment structure for a creative approach

This led to a discussion about formal university structures that determine things like, how subjects will be assessed, as well as the regimes of teaching and learning (“we do it this way because this is the way it’s always been done”). Do they remove your autonomy? It made me wonder what our university official assessment policy is.

Construct validity: Are we using assessment to asses something other than what we say we’re assessing? If so, what are we actually assessing?

There was also a question about whether or not we could / should asses only what’s been formally covered in class. How do you / should you asses knowledge that is self-taught? We could for example, measure the process of learning, rather than the product. I made a point that in certain areas of what I teach, I no longer assign a grade to an individual peice of work and rather give a mark for the progress that the student has made, based on feedback and group discussion in that area.

Outcomes based assessment / criterion referenced assessment

  1. Uses the principle of ALIGNMENT (aligning learning outcomes, passing criteria, assessment)
  2. Is assessing what students should be able to do
  3. “Design down” is possible when you have standardised exit level outcomes (we do, prescribed by the HPCSA)
  4. The actual criteria are able to be observed and are not a guess at a mental process, “this is what I need to see in order to know that the student can do it”
  5. Choosing the assessment tasks answers the question “How will I provide opportunities for students to demonstrate what I need to see?” When this is the starting point, it knocks everything else out of alignment
  6. You need space for students / teachers to engage with the course content and to negotiate meaning or understanding of the course requirements, “Where can they demonstrate competence?”

Criteria are negotiable and form the basis of assessment. They should be public, which makes educators accountable.

When designing outcomes, the process should be fluid and dynamic.

Had an interesting conversation about the priviliged place of writing in assessment. What about other expressions of competence? Since speech is the primary form of communication (we learn to speak before we learn to write), we find it easier to convey ideas through conversation, as it includes other cues that we use to construct meaning. Writing is a more difficult form because we lack visual (and other) cues. Drafting is one way that constructing meaning through writing could be made easier. The other point I thought was interesting was that academic writing is communal (drafting, editors, reviewers all provide a feedback mechanism that isn’t as fluid as speech, but is helpful nonetheless), but we often don’t allow students to write communally.

Outcomes based assessment focusses on providing students with multiple opportunities to practice what they need to do, and the provision of feedback on that practice (formative). Eventually, students must demonstrate achievement (summative).

We should only assign marks when we evaluate performace against the course outcomes.

Finally, in thinking about the written exam as a form of assessment, we identified these characteristics:

  • It is isolated and individual
  • There is a time constraint
  • There is pressure to pass or fail

None of these characteristics are present in general physiotherapy practice. We can always ask a colleage / go to the literature for assistance. There is no constraint to have the patient fully rehabilitated by any set time, and there are no pass or fail criteria.

If assessment is a method we use to determine competence to perform a given task, and the way we asses isn’t related to the task physio students will one day perform, are we assessing them appropriately?

Note: the practical outcomes of this session will include the following:

  • Changing the final assessment of the Ethics module from a written exam to a portfolio presentation
  • Rewriting the learning outcomes of the module descriptors at this year’s planning meeting
  • Evaluating the criteria I use to mark my assignments to better reflect the module outcomes