SAAHE podcast on building a career in HPE

In addition to the In Beta podcast that I host with Ben Ellis (@bendotellis), I’m also involved with a podcast series on health professions education with the South African Association of Health Educators (SAAHE). I’ve just published a conversation with Vanessa Burch, one of the leading South African scholars in this area.

You can listen to this conversation (and earlier ones) by searching for “SAAHE” in your podcast app, subscribing and then downloading the episode. Alternatively, listen online at http://saahe.org.za/2019/06/8-building-a-career-in-hpe-with-vanessa-burch/.

In this wide-ranging conversation, Vanessa and I discuss her 25 years in health professions education and research. We look at the changes that have taken place in the domain over the past 5-10 years and how this has impacted the opportunities available for South African health professions educators in the early stages of their careers. We talk about developing the confidence to approach people you may want to work with, from the days when you had to be physically present at a conference workshop, to explore novel ways to connect with colleagues in a networked world. We discuss Vanessa’s role in establishing the Southern African FAIMER Regional Institute (SAFRI), as well as the African Journal of Health Professions Education (AJHPE) and what we might consider when presented with opportunities to drive change in the profession.

Vanessa has a National Excellence in Teaching and Learning Award from the Council of Higher Education and the Higher Education Learning and Teaching Association of South Africa (HELTASA), and holds a Teaching at University (TAU) fellowship from the Council for Higher Education of South Africa. She is a Deputy Editor at the journal Medical Education, and Associate Editor of Advances in Health Sciences Education. Vanessa was Professor and Chair of Clinical Medicine at the University of Cape Town from 2008-2018in health and is currently Honorary Professor of Medicine at UCT. She works as an educational consultant to the Colleges of Medicine of South Africa.

What does scholarship sound like?

Creative work is scholarly work

The Specialist Committee recognises the importance of both formal academic research and creative outputs for the research cultures in many departments, as well as for individual researchers; it thus aims to give equal value to theoretical/empirical research (i.e. historical, theoretical, analytic, sociological, economic, etc. studies from an arts perspective) and creative work (i.e. in cases where the output is the result of a demonstrable process of investigation through the processes of making art.); the latter category of outputs is treated as fully equivalent to other types of research output, but in all cases credit is only given to those outputs which demonstrate quality and have a potential for impact and longevity.

The South African National Research Foundation has recently shared guidelines for the recognition of creative scholarly outputs, which serves to broaden the concept of what kind of work can be regarded – and importantly, recognised – as “scholarly”. The guidelines suggest that the creative work could include (among others):

  • Non-conventional academic activities related to creative work and performance: Catalogues, programmes, and other supporting documentation describing the results of arts research in combination with the works themselves;
  • In Drama and theatre: scripts or other texts for performances and the direction of and design (lighting, sound, sets, costumes, properties, etc.) for live presentations as well as for films, videos and other types of media presentation; this also applies to any other non-textual public output (e.g. puppetry, animated films, etc.), provided they can be shown to have entered the public domain;

I’m going to talk about podcasts as scholarly outputs because I’m currently involved in three podcast projects; In Beta (conversations about physiotherapy education), SAAHE health professions educators (conversations about educational research in the health professions), and a new project to document the history of the physiotherapy department at the University of the Western Cape.

These podcasts take up a lot of time; time that I’m not spending writing the articles that are the primary form of intellectual capital in academia and I wondered, in the light of the new guidelines from the NRF, if a podcast could be considered to be a scholarly output. There are other reasons for why we may want to consider recognising podcasts as scholarly outputs:

  1. They increase access for academics who are doing interesting work but who, for legitimate reasons, may not be willing to write an academic paper.
  2. They increase diversity in the academic domain because they can be (should be?) published in the language of preference of the hosts.
  3. They reduce the dominance of the PDF for knowledge distribution, which could only be a good thing.
  4. Conversations among academics is a legitimate form of knowledge creation, as new ideas emerge from the interactions between people (like, for example, in a focus group discussion).
  5. Podcasts – if they are well-produced – are likely to have a wider audience than academic papers.
  6. Audio gives an audience another layer of interesting-ness when compared to reading a PDF.
  7. Academic podcasts may make scholarship less boring (although, to be honest, we’re talking about academics, so I’m not convinced with this one).

What do we mean by “scholarship”?

Most people think of scholarly work as the research article (and probably the conference presentation) but there’s no reason that the article/PDF should remain the primary form of recognised scholarly output. It also requires that anyone wanting to contribute to a scholarly conversation must learn the following:

  • “Academic writing” – the specific grammar and syntax we expect from our writers.
  • Article structure – usually, the IMRAD format (Introduction, Methods, Results and Discussion).
  • Journals – where to submit, who is most likely to publish, what journals cater for which audiences.
  • Research process – I’m a big fan of the scientific method but sometimes it’s enough for a new idea to be shared without it first having to be shown to be “true”.

Instead of expecting people to first learn the traditions and formal structures that we’ve accepted as the baseline reality for sharing scholarly work, what if we just asked what scholarship is? Instead of defining “scholarship” as “research paper/conference presentation”, what if we started with what scholarship is considered to be and then see what maps onto that? From Wikipedia:

The scholarly method or scholarship is the body of principles and practices used by scholars to make their claims about the subject as valid and trustworthy as possible and to make them known to the scholarly public… Scholarship…is creative, can be documented, can be replicated or elaborated, and is peer-reviewed.

So there’s nothing about publishing PDFs in journals as part of this definition of scholarship. What about the practice of doing scholarly work? I’m going to use Boyer’s model of scholarship, not because it’s the best but because it is relatively common and not very controversial. Boyer includes four categories of scholarly work (note that this is not a series of progressions that one has to move through in order to reach the last category…each category is a form of scholarship on its own):

  • Scholarship of discovery: what is usually considered to be basic research or the search for new knowledge.
  • Scholarship of integration: where we aim to give meaning to isolated facts that consider them in context; it aims to ask what the findings of discovery mean.
  • Scholarship of application: the use of new knowledge to solve problems that we care about.
  • Scholarship of teaching: the examination of how teaching new knowledge can both educate motivate those in the discipline; it is bout sharing what is learned.

Here are each of Boyer’s categories with reference to podcasts:

  • Discovery (advancing knowledge): Can we argue that knowledge can be advanced through conversation? Is there something Gestalt in a conversation where a new whole can be an emergent property of the constituent parts? How is a podcast conversation any different to a focus group discussion where the cohort is a sample with specific characteristics of interest?
  • Integration (synthesis of knowledge): Can the editing and production of a podcast, using the conversation as the raw data, be integrated with other knowledge in order to add new levels of explanation and critique? This could either be in the audio file or as show notes. Could podcast guests be from different disciplines, exploring a topic from different perspectives?
  • Application/engagement (applied knowledge): Can we use emergent knowledge from the podcast to do something new in the world? Can we take what is learned from the initial conversation, which may have been modified and integrated with other forms of knowledge (in multiple formats e.g. text, images, video), and apply it to a problem that we care about?
  • Teaching (openly shared knowledge): Can we, after listening to a podcast and applied what we learned, share what was done, as well as the result, with others in order that the process (methods) and outcomes (results) can be evaluated by our peers?

This may not be a definitive conclusion to the question of whether podcasts could be regarded as scholarly work but at the very least, it suggests that it’s something we could consider. If you accept that a podcast might be regarded as scholarly we can then ask how we might go about formally recognising it as such.

Workflow to distribute scholarly work

I’m going to use an academic, peer-reviewed, traditional journal (or at least, the principle of one) to explore a workflow that we can use to get a sense of how a podcast could be formally recognised as scholarly work. We first need to note that a journal has two primary functions:

  1. Accreditation, which is usually a result of the journals peer review process, and their brand/history/legacy. The New England Journal of Medicine is a recognised “accreditor” of scholarly work, not because there is anything special about the journal but simply because it is the New England Journal of Medicine. Their reputation is enough for us to trust them when they say that the ideas presented in a piece of work have been tested through peer review and has not been found wanting.
  2. Distribution, which in the past meant printing those ideas on paper and literally shipping them around the world. Today, this distribution function has changed to Discoverability; the journal does what it can to make sure your article can be found by search engines, and if you’re the New England Journal of Medicine you don’t need to do much because Google will do your quality signalling for you by surfacing your articles above others. Theefore, ournals host content and try to increase the chances that we can find it, and the distribution function has largely been taken over by us (because we share articles on behalf of the journals).

By separating out the functions of a journal we can see that it’s possible for a journal to accredit work that it does not necessarily have to host itself. We could have a journal that is asked to accredit a piece of work i.e. signal to readers (or in our case, listeners) that the work has passed some set of criteria that we use to describe it as “scholarly”.

What might this workflow look like? Since I’m trying to show how podcasts could be accredited within the constraints of the existing system of journal publications, I’m going to stick to a traditional process as closely as possible, even though I think that this makes the process unnecessarily complicated, especially when you think about what needs to happen following the peer review. Here is what I think the accreditation process could look like:

  1. Create a podcast episode (this is basically a FGD) on a topic of interest where guests discuss a question or a problem that their community of peers recognises as valid. This could be done by a call to the community for topics of interest.
  2. Edit the podcast, including additional resources and comments as show notes. The podcast creators could even include further comments and analysis, either before, during or after the initial recorded conversation. The audio includes the raw data (the recorded conversation), real-time analysis and critique by participants, discussion of potential applications of the emergent knowledge, and conclusion (maybe via post-recording reflection and analysis).
  3. Publish the episode on any podcast-hosting platform. The episode is now in the public domain.
  4. Submit a link to the episode to a journal, which embeds the podcast episode as a post (“article”) along with a short description of what it includes (like an abstract), a description of the process of creation (like the methods), the outcome of the conversation (like a conclusion), and a list of additional reading (like a reference list).
  5. The journal begins the process of accrediting the podcast by allocating peer reviewers, whose reviews are published alongside the embedded podcast in the journal.
  6. Reviewers review the “methods”, “conclusions”, “references” and knowledge claims of the podcast guests, add comments to the post, and highlight the limitations of the episode. The show notes include a description of the process, participants, additional readings, DOI, etc. This could be where the process ends; the journal has used peer review to assign a measure of “quality” to the episode and does not attempt to make a judgement on “value” (which is what journals do when they reject submissions). It is left to the listener to decide if the podcast has value for them.
  7. The following points are included for completeness as they follow a traditional iterative process following peer review. I don’t think these steps are necessary but are only included to map the workflow onto a process that most authors will be familiar with:
    1. The podcast creators make some changes to the audio file, perhaps by including new analysis and comments in the episode, or maybe by adding new information to the textual component of the episode (i.e. the show notes).
    2. The new episode is released. This re-publication of the episode would need to be classified as an entirely different version since the original episode would have been downloaded and shared to networks. An updated version would, therefore, need a new URL, a new page on the podcast hosting service, etc.

In the example workflow above, the journal never hosts the audio file and does not “publish” the podcast. It includes an embedded version of the episode, the show notes (which include the problem under discussion, the participants and their bios, an analysis of the conversation, and a list of references), as well as the full peer reviews. Readers/listeners then decide on the “importance” of the episode and whether or not to assign value to it. In other words, the readers/listeners decide what work is valuable, rather than the peer reviewers or the journal.

In summary, I’ve tried to describe why podcasts are potentially a useful format for creating and sharing the production of new knowledge, presented a framework for determining if a podcast could be considered to be scholarly, and described the workflow and some practical implications of an accreditation process using a traditional journal.

Read: Academic research

Many people think that “ivory tower” intellectuals make little difference in the world. But some of the highest impact people in history have been researchers, and if you have good personal fit with academic research, we think it can be one of the highest-impact paths…In this article, we’ll cover why we think a career in academia has the potential to be very high impact in the right circumstances, how to figure out whether this option is for you, and how to maximise the impact you can have as an academic.

Whittlestone, J. (2018). Academic research. 80,000 hours.

This article is an incredibly deep dive into the relative benefits and disadvantages of an academic career. Whether you’re in the early stages of thinking about starting on an academic career or are already a full professor, you may find some very valuable context for planning your next move. If you think of your career as something that happens to you, then this article may be a good place to start thinking differently about the rest of your working life.

80 000 hours is an organisation that aims to help as many people as possible lead high impact careers. The idea is that you have about 80 000 working hours in your career, which means that your choice of career is one of the biggest decisions you’ll ever make. Therefore, it’s worth spending a bit of time figuring out how to use that time for good. Their About page has links to many more resources that I’ve found very useful for thinking about my career as an academic.

Emotions and assessment: considerations for rater‐based judgements of entrustment

We identify and discuss three different interpretations of the influence of raters’ emotions during assessments: (i) emotions lead to biased decision making; (ii) emotions contribute random noise to assessment, and (iii) emotions constitute legitimate sources of information that contribute to assessment decisions. We discuss these three interpretations in terms of areas for future research and implications for assessment.

Source: Gomez‐Garibello, C. and Young, M. (2018), Emotions and assessment: considerations for rater‐based judgements of entrustment. Med Educ, 52: 254-262. doi:10.1111/medu.13476

When are we going to stop thinking that assessment – of any kind – is objective? As soon as you’re making a decision (about what question to ask, the mode of response, the weighting of the item, etc.) you’re making a subjective choice about the signal you’re sending to students about what you value. If the student considers you to be a proxy of the profession/institution, then you’re subconsciously signalling the values of the profession/institution.

If you’re interested in the topic of subjectivity in assessment, you may be interested in two of our In Beta episodes:

You are your Metadata: Identification and Obfuscation of Social Media Users using Metadata Information

We spend a lot of time focusing on the content of messaging systems as a means of identifying people but it looks like the metadata encoded alongside the content may be just as important when it comes to de-anonymising the data. This wasn’t always a problem because it’s hard to analyse multivariate relationships in large sets of data, especially when we don’t really know what we’re looking for. It turns out that machine learning algorithms are very good at finding patterns that we don’t have to explicitly define, which means we need to think carefully about what is included in the data we share.

This may also have implications for the publication of data sets that researchers are under pressure to include in their final publications. How long before we need to ensure that metadata – as well as names – are scrubbed from the data sets?

We also found that data obfuscation is hard and ineffective for this type of data: even after perturbing 60% of the training data, it is still possible to classify users with an accuracy higher than 95%. These results have strong implications in terms of the design of metadata obfuscation strategies, for example for data set release, not only for Twitter, but, more generally, for most social media platforms.

Source: [1803.10133v1] You are your Metadata: Identification and Obfuscation of Social Media Users using Metadata Information

OpenPhysio | A new physiotherapy education journal

I’m really excited to announce a new project that I’ve been working on together with the folks at Physiopedia. Today we’re launching an open access, peer reviewed journal with a focus on physiotherapy education, with a few features that we think are pretty innovative in the academic publishing space. The journal is called OpenPhysio and represents what we think is a fundamental shift away from traditional ways of thinking about how we share knowledge.

Here are some of the ways we think the journal is different to more traditional publication channels:

  • Immediate publication. Your article is available to the public almost immediately after submission.
  • Peer review is open and transparent. Authors work together with peer reviewers, and the reviews and author responses are published alongside the final article, together with DOIs that make them citable objects.
  • You retain your intellectual property at no cost. OpenPhysio does not require you to transfer copyright to the journal, and there are no page fees for published articles.
  • Articles are first class internet citizens. Your articles can be enhanced with images, audio, tagging, hyperlinks, and video.

We’re still in the early stages of the project (we have no publications yet) and there’s a lot still to iron out, but we’ve decided to make it public nonetheless. This is in line with our broader thinking about publication, which is to share stuff early and then hash it out in the real world. We have Editorial and Advisory Boards and you can have a look at our policies around open access and peer review.

Now, before you write and tell me that there’s no such thing as physiotherapy education (you’d be right, by the way) we want to be clear that this is a journal aimed at physiotherapists with an emphasis on teaching and learning. it’s not about suggesting that the way physiotherapists learn is somehow different to how nurses, physicians and OTs learn. But we do think that there’s a space to explore our context in ways that may not translate well into other domains.

We want to encourage submissions from physios who are interested in learning more about teaching and learning, whether you’re supervising students or less-experienced colleagues in the clinical and community contexts, or if you’re an academic responsible for teaching in undergraduate and postgraduate classrooms. If you’re interested in teaching and learning in a physiotherapy context, we’d love it if you would consider OpenPhysio as a channel to share your ideas.

If you’d like to know more about the journal, please contact the Editor or visit the website.

Action research as liberation

"Paulo Freire" by Slobodan Dimitrov - Own work. Licensed under CC BY-SA 3.0 via Wikimedia Commons - https://commons.wikimedia.org/wiki/File:Paulo_Freire.jpg#/media/File:Paulo_Freire.jpg

Kemmis & Mctaggert’s (1990) definition of action research is that it is about improving the lives of people through transformation. It is an emancipatory approach to the research process that does as much for the participants as for the researchers. I’m busy reading Paulo Freire’s Pedagogy of the Oppressed, so the idea of a research process in the educational context as being a form of emancipation for students stands out. The idea that, through trying to learn more about learning and teaching, we can improve both, sits well with me.

“Education either functions as an instrument which is used to facilitate integration of the younger generation into the logic of the present system and bring about conformity or it becomes the practice of freedom, the means by which men and women deal critically and creatively with reality and discover how to participate in the transformation of their world.”

Paulo Freire (Pedagogy of the Oppressed)

 

Abstract for RCTs in educational research

There seems to have been a resurgence in calls for the use of systematic reviews and randomised controlled trials in educational research lately. There’s a lot to like (in my opinion) about RCTs in certain contexts because of how they are designed.  For example, when you want to figure out the effect of variable A on variable B, it’s a very useful approach because of the randomisation of the sample and the blinding of assessors and participants.

However, the method doesn’t translate well into most educational contexts for a variety of reasons, usually in the form of arguments for how RCTs in educational research are unethical and logistically difficult. I recently wrote a position paper with a colleague from Rhodes University where we looked at the argument against RCTs where we basically ignore the arguments just mentioned. We focus instead on how using an RCT pre-supposes an understanding of teaching and learning that is at odds with what we know about how learning happens. The article will be published soon in the African Journal of Health Professions Education. Here’s the abstract:

Randomised controlled trails (RCTs) are a valued research method in evidence-based practice in the medical and clinical settings. However, the use of RCTs is associated with a particular ontological and epistemological perspective that is situated within a positivist world view. It assumes that environments and variables can be controlled in order to establish cause-effect relationships. But current theories of learning suggest that knowledge is socially constructed, and that learning occurs in open systems which cannot be controlled and manipulated as would be required in a RCT. They recognise the importance and influence of context on learning, something that positivist research paradigms specifically aim to counter. We argue that RCTs are inappropriate in education research because they force us to take up ontological and epistemological positions within a technical rationalist framework that is at odds with current learning theory.

 

The CONSORT guidelines for systematic reviews of RCTs

When I was at the WCPT conference last year I came across the CONSORT guidelines for the publication of systematic reviews of RCTS, which I’d never heard of before. I made a note to look it up and finally got around to doing it. I thought would be quite helpful in planning and carrying out these kinds of research projects, so I’m sharing a few notes here.

CONSORT stands for Consolidated Standards of Reporting Trials and is an “evidence-based, minimum set of recommendations for reporting randomized trials”. In addition to the CONSORT statement, there is a checklist that can be used for evaluating the quality of reports of clinical trials. It is, in essence, a description of how to conduct and report on systematic reviews. If you’re interested in conducting systematic reviews of any trials, then this is definitely something to pay attention to.

consort-flow-diagram

Additional resources for the CONSORT guidelines

Digital literacy survey: Outcome of reliability testing

Earlier this year we started the International Ethics Project, a collaboration between physiotherapy departments from several countries who intend offering an online course in professional ethics to their undergraduate students. You can read more about the project here.

In June we started the process of developing a questionnaire that we can use to establish some baseline data on students’ levels of digital literacy. It’s taken a bit longer than expected but we’ve finally managed to complete the reliability testing of the questionnaire as part of a pilot study. Before we can begin planning the module and how it will be implemented we need to get a better understanding of how our population – drawn as they are from several countries from around the world – uses digital tools in the context of their learning practices. The results of the reliability study showed that most of the survey items had Kappa values between 0.5 – 0.6 (indicating moderate agreement); 0.7 – 0.8 (indicating strong agreement); or >0.8 (indicating almost perfect agreement). See this post on the project blog for more details on how the reliability testing was conducted.

Now that we have conducted quite a rigorous piloting of the questionnaire, we hope that this questionnaire might be useful for other health professional educators who are considering the use of digital tools in their classrooms. To this end we would like to report the results of this pilot, along with some preliminary results, at the ER-WCPT conference on 11-12 November, 2016 in Liverpool. We will therefore be submitting an abstract for the conference in the coming months.