assessment conference education learning research students teaching

SAAHE conference, 2011 – day 3

Today was the last day of the SAAHE conference. Coming as it did immediately after a week of the SAFRI programme, I can’t say I’m not glad it’s over. It’s been an amazing experience though, mostly as a result of the wonderful health educators I’ve been fortunate enough to meet. Here are the last of the notes I took during the conference.

Simulation-based asessment: challenges and opportunities by Jack Boulet

Simulations used for summative and formative assessment, as well as curriculum assessment and patient safety

Need to know what health practitioners actually do, (i.e. procedurally), as opposed to what they know

Simulated (standardised) patients are good for some things but not others e.g. trauma

Performance measures:

  • Link measures to scenario events
  • Focus on observable behaviour
  • Incorporate multiple measures from different sources

Types of scores:

  • Explicit process
  • Implicit process
  • Explicit outcome

Checklist for assessing acute scare skills

  • Certain actions are more important than others
  • Sequence and timing are important

Checklists reward thoroughness

Training and quality assurance are important when it comes to assessment and ratings

Developing reliable and valid scoring systems is difficult

Important to identify and minimise errors of measurement

Peer review is essentially about getting a high number of opinions that over time will average out to be an accurate measure

Cases (simluations) are “vehicles” to measure skills

  • Who are the target examinees?
  • Specificity
  • Difficulty
  • Essential manoeuvres and questions?
  • Sampling from a domain (identify the domain)

Predictive validity” – Even with simulation, it’s difficult to establish predictive value → performance in the real world


  • Cost
  • Logistics
  • Setting standards
  • Interdisciplinary skills (e.g. measure doctor-patient interaction but not doctor-nurse interaction)
  • Integration

What is the societal cost of having providers with inadequate knowledge and skills?

It’s more interesting to measure how people lose skills / competence over time, than to measure how they acquire skills

Inferences concering competence are dependent on linking scores to performance criteria

How can we best use technology as part of current clinical / educational efforts?

  • Electronic portfolios
  • Online testing
  • Combined methods

Use of advanced technology to increase fidelity e.g. virtual reality, haptic systems

Students change behaviour when they know how they’re being assessed e.g. with checklists

Simulation studies using confederates (can this be done with students and “broken” equipment?)

Good teamwork is easier to recognise than it is to define

Making use of adult learning theories by David Taylor

Attended a workshop yesterday, which covered much of the same content

Behaviourism – consequences drive actions

Picking up the rules of a community”, “learning the rules of the game”, “what does it mean to be?

How do you learn “enough of the words” to make sense of the discipline?

Exploration of a model based on Kolb’s learning cycle:

Elaborate” new knowledge → consider all propositions and discard ones that are irrelevant, experts navigate this path quickly

  • Work out the most likely resources to refine possibilities
  • Actively participate in the activity
  • Refine the information into a hypothesis

Reflecting / organising:

  • Test – retest the hypothesis
  • Organise information into a “story” that makes sense to them
  • Teachers need to provide cognitive structures upon which students can build → scaffolding
  • Encourage reflection-in-action / reflect while doing


  • Students needs to articulate prior knowledge
  • Assessment is a form of feedback
  • Feedback can only be given when students have articulated / exposed their understanding
  • Teachers must be open to accept (and to act on) feedback from students

Reflect / consolidate:

  • Take on board the feedback
  • Reflection in the light of new knowledge and the learning process
  • Evaluate personal responsibility for learning
  • Teacher needs to provide opportunities for the learner to rehearse / apply new knowledge i.e. encourage reflection-on-action


  • Using a challenge to help students make a conceptual leap by identifying / proposing an alternative concept that they had not considered
  • Is dissonance a way to help students move through Vygotsky’s ZPD?
  • Mezirow – “learning is a disorienting dilemma”
  • Can be created by manipulating”
    • Resources: should be appropriate, sufficient and relevant
    • Motivation (Knowles):
      • Intrinsic: adults learn because they need to know, have a self-concept as a learner, have life experiences, readiness to learn, orientated towards learning
      • Extrinsic: programme / curriculum, community of practice
    • Stage of development (Perry): Duality (“right and wrong answers and the teacher knows what’s right”) → multiplicity (“comfortable that in any given situation, there’s more than one answer, and that context is what matters most” – comfort in dealing with uncertainty)
    • Style of learning (Entwistle, Biggs): strategic, deep, surface ← how do you measure which of these is happening?

CoP (Wenger):

  • We don’t live or work (learn) in a vacuum
  • Everyone is part of a community
  • We only learn in community (does this mean that it’s impossible to learn independently?)
  • We develop as part of that community

Perceptions and experiences on community engagement as part of learning Student sessions. Points below taken from a variety of student presentations

Train of hope (Phulophepa)

Service learning, research and volunteerism: providing support to about 87 organisations in the area, entirely student run → builds confidence and experience, receive certificates for work done which are valued by future employers

Move from community service to community development

External evaluation bring accountability to projects

Most common health problem encountered by medical students on community-based learning placements is drug abuse

Community dynamics:

  • lack of medication
  • non-compliance
  • traditional healers
  • religion and beliefs
  • social problems

Patients’ stories are often heartbreaking

The patient is more than the illness”

Poor of the poorest”

Medical doctors and sangomas (traditional healers) have areas of overlapping practices and principles

As long as patients are living in this world, they are appreciated as human beings”

Challenge students to move out of their comfort zones

Community-based projects count for a very small percentage of the coursework grade, yet it takes an enormous amount of personal commitment and time, and is supposedly valued by educators

Language is a significant problem for student-patient interaction

Students conflicted when we tell them to think outside the box, but then have expectations for them to “do it by the book”

Does the institution learn as much as the students from the community experiences? When students report back to the institution, what changes do they experience as / if the institution responds? Does the institution respond?


Posted to Diigo 08/12/2010

  • Simply labelling a group as a community neither ensures that it functions as one, nor that it is a beneficial, cohesive unit in which learning will take place readily
  • Riel and Polin  propose a typology of three ‘distinct but overlapping forms of learning within communities’
      1. Task-based: groups of people working together intently over a limited period of time on a particular product
      2. Practice-based: usually based around a profession or discipline, these groups focus on sharing and developing good practice
      3. Knowledge-based: here the focus is on the use and reuse of knowledge in a never ending cycle
  • These typologies to be useful in identifying perhaps extreme examples of communities, however in the majority of cases examples of working communities for learning are a hybrid of all three types
    • Effectively collaborating in joint activities, producing useful outputs related to practice and helping to advance the state-of-the art in domain knowledge
    • Categories
      twitter feed

      Twitter Weekly Updates for 2010-08-09

      twitter feed

      Twitter Weekly Updates for 2010-08-02

      learning social media

      Facilitating Communities of Practice in the Network Era

      Two days ago I was lucky enough to get the opportunity to attend a workshop at UCT facilitated by Nancy White (1), who co-wrote Digital habitats: stewarding technology for communities (2) with Etienne Wenger and John David Smith. Presentation slides from the workshop can be downloaded / viewed here. Unfortunately, I could only stay for the morning session, so my notes are  incomplete and they may be incorrect. If you attended the workshop and would like to extend these notes, please add your comments below.

      The workshop started with the Human Spectrogram, “a group face to face exercise to help surface similarities and differences in a group, help people to get to know each other and to do something together that is active. Other knowledge sharing toolkits can be found here.
      Communities collectively accept responsibility for the behaviour of others in the community
      Community is about purpose and specifically, shared purpose
      Reciprocity is very important in communities, although not necessarily with the same person who shared with you
      Leading / facilitating CoP will often require improvisation / innovation
      “Community indicator = sign of life: asking questions / showing something that delves deeper into what the community is or wants to be. It can vary by community, and should be reflective of the community
      Use of metaphor can be evocative. If you’re too explicit, you can turn people off because they may think they know what you’re talking about, and therefore miss what you actually want them to do / think about. Whereas, using something that’s open to interpretation, or more abstract will stimulate discussion or reflection in the community.
      Invitation to participate is essential. Invites can be in different forms:
      • Discovery (can be serendipitous)
      • Explicit invitation (this can take multiple forms)
      Game mechanics (Amy Jo Kim) → games stimulate interest and engagement with content (3) (4) (I explored this idea a little bit last year when I was thinking about the use of gaming in physiotherapy)

      Websites are not communities, people are communities
      Howard Rheingold’s book “The Virtual Community: Homesteading on the Electronic Frontier” is a good exploration of online communities
      How do you stop communities from fragmenting?
      Facilitating online and offline communities is always evolving because the environment is always changing
      A “CoP” perspective is not the same as defining if something is a CoP. CPD is an appropriate framework to explore communities:
      • Community (a group of people who can be named)
      • Practice (intent, talking about something in order to do it better
      • Domain (what we care about, shared interest, purpose)
      Don’t change all 3 of the above at once, for fear of destabilising the CoP
      CoP is not a binary thing. It can exist on continuum between is and isn’t
      Small groups are adaptable, don’t have to negotiate (as much) in order to change, can be flexible
      Institutional(?) / online interfaces are not usually designed for small groups, multiple small groups can scale out to large groups.
      • Me (individual): individual, identity, interest, trajectory, consciousness, confidence level, risk tolerance, styles, emotion
      • We (community): bounded, members who you know, group identity, shared interest, human centred, distinct power/trust dynamics, forward movement, strong blocking, statis, attention to maintenance, language
      • Many (networks): boundaryless, fuzzy, intersecting interests, “object centred sociality” (Jyri Engstrom), flows around blocks, less cohesion, distributed power/trust, change
      People trust people around the content they produce. Blogs and referral systems can establish relationships around “objects” / content. This can be scary for people who are used to creating relationships around personal interactions. This has implications for how we use content to attract and engage with people. Communities are not about curating or archiving content, but for providing channels for sharing content and facilitating relationships.
      There is a difference between a network and a community, and depending on your objectives, you may have to make a conscious decision about which one you want to develop towards. Networks of Practice is a concept used to explore the areas where network theory and CoP intersect (5).
      • Network – a lot of people know a lot of people, but they don’t all know each other. There are loose ties (link downloads article PDF) (Granovetter), it can scale beyond your ability to facilitate the group
      • Community – you know people more intimately, there is meaningful connection (but can also be present in networks)
      Blogging and communities – Lilia Efimova
      Dunbar’s number = 150 (how many stable social relationships we can manage)
      People have to actively engage of their own accord without the community being “done to them”
      Are we inward-facing or outward-facing in our department? Who are we looking to connect with / influence?
      Legitimate peripheral participation i.e. lurking in online groups. Are they part of a network or a community? It can be argued either way. This is a big part of online social networks → community or network?
      1. Interview with Nancy White by George Siemens for the World Conference on Educational Multimedia, Hypermedia and Telecommunications
      2. Online companion to Digital Habitats:  Stewarding technology for communities
      3. Stuart, B. (2006). How game mechanics can make your app more fun – a blog post looking at some of Amy Jo Kim’s work
      4. Putting the fun in functional – presentation by Amy Jo Kim on Slideshare
      5. Knowledge Networks: Innovation through Communities of Practice
      6. Granovetter, M. (1983). The strength of weak ties: A network theory revisited. Sociological Theory, 1, 201-233 (link downloads PDF)

      Posted to Diigo 05/16/2010

        • “PLEs are more a methodology or an approach to technology enhanced learning than an application.”
        • the introduction of e-learning led to a reverse in pedagogic innovation
        • I tend to think that knowledge is best shared and developed through communities of practice. Communities of practice as Etienne Wenger says are based on a shared repertoire of communal resources (routines, sensibilities, artefacts, vocabulary, styles, etc.) that members have developed over time.
        • One of the challenges faced by traditional education is found in the very goal of its existence (second only to its role as providing value statements through accreditation): to present bounded structures of knowledge in order for others to learn a discipline. Or put another way, schools and universities help students make sense of a discipline.
        • There are two elements under consideration:
        • 1. The curriculum itself
        • This is what learners must learrn.
        • Curriculum/content is created and disseminated through research and publication. This content then forms the basis of instruction. Nothing new here, with the exception of the argument that the scholarly publication process is too slow.
        • It’s this content that most people see as the important part of education.
        • We make learners do all sorts of fun things to get this to happen: cases, problem-based learning, games/simulations, lectures, podcasts, tests, eportfolios, and so on.
        • 2. The framework of sensemaking
        • Should the educator provide a formed narrative of coherence? Or should learners be tasked with this? Should the educator create a fully bounded content structure? Or should the content interaction opportunities be more fluid? And what about interaction? Should it be under the control of educators? Or should learners self-organanize as they deem worthwhile? This is where education truly begins to change. Tweaking content creating and delivery models is perhaps a start. But it’s not transformative.
        • It means the conversation is more chaotic. It means that we’re always missing something. Everyone is. Some important conversation, somewhere, is being overlooked. Why is that so discomforting?
        • We expect the academy to be a place that provide clarity, a path forward
        • When we then step into a course and discover the conversation is distributed and that the expected frameworks for telling us what to think don’t exist, we get disoriented.

          But isnt’ that life?

        • Isn’t that how real learning occurs? In business? In our personal lives? Who actually possesses a framework fo sensemaking in advance of encountering novel problems? Or who can rely on the “narrative of coherence” provided in advance of becoming a parent?
        • We can’t manage it all. We must choose. As we move through this course, we’ll focus more on what it means to choose – i.e. what types of networks we want and need to build. For now, realizing that our ability to make sense is under our control.
        • Where is the learning in this? The learning exists in the process of forming and navigating networks. Some sources we filter. Some thinkers we value greatly. Others we ignore.
        • This paper described experimentation in the development of distributed online courses and in software – particularly, the personal learning environment – that support the formation of connections between the far-flung pieces of such courses
        • suggests a pedagogy of participation rather than retention, and even suggests distributed and locally-based forms of evaluation and assessment
        • The intent of such systems is to to facilitate the conversation and interaction around episodic learning events in a distributed environment, transforming them from elements in a linear flow-based design to free-floating objects in an environment
        • In addition to providing an engaging and immersive environment for student learning, substantially improving motivation and interaction with the learning material, games and simulations are able to support learning in complex environments, offering a subtlety simple instruction-based or lecture-based learning cannot offer. (Squire, 2005)
        • games and simulations fall into a category similar to lectures and presentations in that they involve statically designed learning objectives and strategies. (Amory & Seagram, 2003)
        • we often do not know what it is we want to teach the student. (Caine & Caine, 1997) Today’s environment is variable, which means situations – and hence, fact – change fluidly. One day Pluto is a planet, the next day it is not. One day Czechoslovakia is a country, the next day it is not. One day capitalism is the unassailable foundation for our economic system, the next day, following a market collapse, it is not. Moreover, today’s environment is complex. The relations between variables cannot be described or even predicted.
        • learners themselves are changing
        • It has even been suggested that our interactions with modern communication technologies change the way we think. Even if we reject such descriptions as students as overly broad and inaccurate generalizations – and there is good reason for doing so – it is nonetheless the case that the needs, capabilities and interests of the target audience is rapidly shifting and changing
        • It is not merely to create a network into which to situate episodic learning, but rather, to create a network that learns and thus adapts and reshapes itself based on those conversations and interactions. (Downes, 2007)
        • the best we can manage is to teach students how to learn, and to encourage them to manage their own learning thereafter
        • how we learn itself is something that changes, and cannot be precisely taught
        • For this reason, we need to see the educational system itself as adaptive rather than merely prescriptive
        • knowledge exists in the minds of the members or participants, and this knowledge is derived from their direct (and recent) experience in the field
        • In addition, the need for content and support emerges from conversations among the participants. These interactions are able to reveal not only what company commanders know, but also what they don’t know (and need to know). The interaction, in other words, meets and addresses an objection often put of self-directed learners, that they don’t know what they need to know. (Clayson, 2005)
        • The core of a social networking technology is the capacity to create links between members in a community – to create, in other words, social networks
        • Topics, for example, are not assigned centrally, but are instead created by individuals ‘tagging’ certain content with terms or categories they choose themselves. (Barsky & Purdon, 2006) Each person’s social network on a social networking site, moreover, is unique; there is no definitive grouping of people, only a clustering of people with more or less similar interests.
        • users jump from service to service, creating (and discarding) new identities as needed. A typical web user may have multiple ‘home pages’ – their personal blog, their photo page on Flickr or Picassa, their Google Reader account, shared documents through Zoho, their video page on YouTube, their Twitter account, their profiles, on Facebook, MySpace and LinkedIn, their Wikipedia login, their email accounts, and (often least) their university LMS login. While real friendships and communities develop through this mélange, loyalty to online sites and services is limited and fleeting. (O’Brien, 2007)
        • The idea of the personal learning environment is that it performs many of the functions of a content management system and of a social network system but from the perspective of the individual rather than the community or the institution. (Attwell, 2006) Hence, the PLE may be understood as the intersection of the multiple home pages employed by any given individual. In the first instance, the PLE is a concept, rather than an application – it is the idea that a person’s web presence can be distributed. (Attwell, Graham Attwell: “Knowledge is best shared and developed through communities of practice”, 2007)
        • Because there were so many people contributing to the course, and because the content of the course actually shifted and varied according to participation and input into the course, it was necessary to emphasize to students that their role in the course was not to attempt to assimilate all course content. This was neither possible nor desirable. Rather, students were told that their role was to select and sample course content, pursuing areas of interest, reading related material from both within and outside the course, and then to contribute their unique perspective based on this reading. (Siemens, Where does the learning occur??, 2008)
        • we are currently seeing experimentation in the development of distributed online courses and in software – particularly, the personal learning environment – that support the formation of connections between the far-flung pieces of such courses
        • In the PLE project being undertaken by the National Research Council, the functionality of the PLE is depicted in four major stages: to aggregate, that is, to collect content from the individual’s and other online content service providers, where aggregation includes elements of recommendation, data mining and automated metadata extraction ; to remix, or to organize content from several different sources in different ways, including through automated clustering; to repurpose, or edit, localize, or otherwise modify or create new content; and to feed forward, or send the content to subscribers and other web services, either via RSS syndication, email, Twitter, or other relevant services. (Downes, Theory of Learning Networks, 2004)
        • the PLE is not a recreation of the capabilities of the learning management system, but rather, a learning network

      Posted to Diigo 05/05/2010

        • a Community of Practice (CoP) was defined by Wenger as a tightly knit group of colleagues engaged in a shared practice (within the same or similar firm or organization); who meet regularly (usually face-to-face) and share common understandings and language. The CoP members support each other, collaborate in problem solving and generally making sense of a shared world view.
        • a Network of Practice as a distributed aggregation of members who share some common interests and values, but their correspondence and especially face to face meetings occur much less often or not at all. Leadership and activities in a NoP are emergent and usually informal. NoP members interact sporadically and develop their network in an informal and spontaneous manner that is occasioned through blogs, social software based communities, perhaps a face-to-face or online conference, newsgroup, mailing list or other shared social networking interactions. Membership in a NoP is voluntary, usually open, often transitory and likely many of the NOP members are strangers to each other

      Posted to Diigo 05/03/2010