What does scholarship sound like?

Creative work is scholarly work

The Specialist Committee recognises the importance of both formal academic research and creative outputs for the research cultures in many departments, as well as for individual researchers; it thus aims to give equal value to theoretical/empirical research (i.e. historical, theoretical, analytic, sociological, economic, etc. studies from an arts perspective) and creative work (i.e. in cases where the output is the result of a demonstrable process of investigation through the processes of making art.); the latter category of outputs is treated as fully equivalent to other types of research output, but in all cases credit is only given to those outputs which demonstrate quality and have a potential for impact and longevity.

The South African National Research Foundation has recently shared guidelines for the recognition of creative scholarly outputs, which serves to broaden the concept of what kind of work can be regarded – and importantly, recognised – as “scholarly”. The guidelines suggest that the creative work could include (among others):

  • Non-conventional academic activities related to creative work and performance: Catalogues, programmes, and other supporting documentation describing the results of arts research in combination with the works themselves;
  • In Drama and theatre: scripts or other texts for performances and the direction of and design (lighting, sound, sets, costumes, properties, etc.) for live presentations as well as for films, videos and other types of media presentation; this also applies to any other non-textual public output (e.g. puppetry, animated films, etc.), provided they can be shown to have entered the public domain;

I’m going to talk about podcasts as scholarly outputs because I’m currently involved in three podcast projects; In Beta (conversations about physiotherapy education), SAAHE health professions educators (conversations about educational research in the health professions), and a new project to document the history of the physiotherapy department at the University of the Western Cape.

These podcasts take up a lot of time; time that I’m not spending writing the articles that are the primary form of intellectual capital in academia and I wondered, in the light of the new guidelines from the NRF, if a podcast could be considered to be a scholarly output. There are other reasons for why we may want to consider recognising podcasts as scholarly outputs:

  1. They increase access for academics who are doing interesting work but who, for legitimate reasons, may not be willing to write an academic paper.
  2. They increase diversity in the academic domain because they can be (should be?) published in the language of preference of the hosts.
  3. They reduce the dominance of the PDF for knowledge distribution, which could only be a good thing.
  4. Conversations among academics is a legitimate form of knowledge creation, as new ideas emerge from the interactions between people (like, for example, in a focus group discussion).
  5. Podcasts – if they are well-produced – are likely to have a wider audience than academic papers.
  6. Audio gives an audience another layer of interesting-ness when compared to reading a PDF.
  7. Academic podcasts may make scholarship less boring (although, to be honest, we’re talking about academics, so I’m not convinced with this one).

What do we mean by “scholarship”?

Most people think of scholarly work as the research article (and probably the conference presentation) but there’s no reason that the article/PDF should remain the primary form of recognised scholarly output. It also requires that anyone wanting to contribute to a scholarly conversation must learn the following:

  • “Academic writing” – the specific grammar and syntax we expect from our writers.
  • Article structure – usually, the IMRAD format (Introduction, Methods, Results and Discussion).
  • Journals – where to submit, who is most likely to publish, what journals cater for which audiences.
  • Research process – I’m a big fan of the scientific method but sometimes it’s enough for a new idea to be shared without it first having to be shown to be “true”.

Instead of expecting people to first learn the traditions and formal structures that we’ve accepted as the baseline reality for sharing scholarly work, what if we just asked what scholarship is? Instead of defining “scholarship” as “research paper/conference presentation”, what if we started with what scholarship is considered to be and then see what maps onto that? From Wikipedia:

The scholarly method or scholarship is the body of principles and practices used by scholars to make their claims about the subject as valid and trustworthy as possible and to make them known to the scholarly public… Scholarship…is creative, can be documented, can be replicated or elaborated, and is peer-reviewed.

So there’s nothing about publishing PDFs in journals as part of this definition of scholarship. What about the practice of doing scholarly work? I’m going to use Boyer’s model of scholarship, not because it’s the best but because it is relatively common and not very controversial. Boyer includes four categories of scholarly work (note that this is not a series of progressions that one has to move through in order to reach the last category…each category is a form of scholarship on its own):

  • Scholarship of discovery: what is usually considered to be basic research or the search for new knowledge.
  • Scholarship of integration: where we aim to give meaning to isolated facts that consider them in context; it aims to ask what the findings of discovery mean.
  • Scholarship of application: the use of new knowledge to solve problems that we care about.
  • Scholarship of teaching: the examination of how teaching new knowledge can both educate motivate those in the discipline; it is bout sharing what is learned.

Here are each of Boyer’s categories with reference to podcasts:

  • Discovery (advancing knowledge): Can we argue that knowledge can be advanced through conversation? Is there something Gestalt in a conversation where a new whole can be an emergent property of the constituent parts? How is a podcast conversation any different to a focus group discussion where the cohort is a sample with specific characteristics of interest?
  • Integration (synthesis of knowledge): Can the editing and production of a podcast, using the conversation as the raw data, be integrated with other knowledge in order to add new levels of explanation and critique? This could either be in the audio file or as show notes. Could podcast guests be from different disciplines, exploring a topic from different perspectives?
  • Application/engagement (applied knowledge): Can we use emergent knowledge from the podcast to do something new in the world? Can we take what is learned from the initial conversation, which may have been modified and integrated with other forms of knowledge (in multiple formats e.g. text, images, video), and apply it to a problem that we care about?
  • Teaching (openly shared knowledge): Can we, after listening to a podcast and applied what we learned, share what was done, as well as the result, with others in order that the process (methods) and outcomes (results) can be evaluated by our peers?

This may not be a definitive conclusion to the question of whether podcasts could be regarded as scholarly work but at the very least, it suggests that it’s something we could consider. If you accept that a podcast might be regarded as scholarly we can then ask how we might go about formally recognising it as such.

Workflow to distribute scholarly work

I’m going to use an academic, peer-reviewed, traditional journal (or at least, the principle of one) to explore a workflow that we can use to get a sense of how a podcast could be formally recognised as scholarly work. We first need to note that a journal has two primary functions:

  1. Accreditation, which is usually a result of the journals peer review process, and their brand/history/legacy. The New England Journal of Medicine is a recognised “accreditor” of scholarly work, not because there is anything special about the journal but simply because it is the New England Journal of Medicine. Their reputation is enough for us to trust them when they say that the ideas presented in a piece of work have been tested through peer review and has not been found wanting.
  2. Distribution, which in the past meant printing those ideas on paper and literally shipping them around the world. Today, this distribution function has changed to Discoverability; the journal does what it can to make sure your article can be found by search engines, and if you’re the New England Journal of Medicine you don’t need to do much because Google will do your quality signalling for you by surfacing your articles above others. Theefore, ournals host content and try to increase the chances that we can find it, and the distribution function has largely been taken over by us (because we share articles on behalf of the journals).

By separating out the functions of a journal we can see that it’s possible for a journal to accredit work that it does not necessarily have to host itself. We could have a journal that is asked to accredit a piece of work i.e. signal to readers (or in our case, listeners) that the work has passed some set of criteria that we use to describe it as “scholarly”.

What might this workflow look like? Since I’m trying to show how podcasts could be accredited within the constraints of the existing system of journal publications, I’m going to stick to a traditional process as closely as possible, even though I think that this makes the process unnecessarily complicated, especially when you think about what needs to happen following the peer review. Here is what I think the accreditation process could look like:

  1. Create a podcast episode (this is basically a FGD) on a topic of interest where guests discuss a question or a problem that their community of peers recognises as valid. This could be done by a call to the community for topics of interest.
  2. Edit the podcast, including additional resources and comments as show notes. The podcast creators could even include further comments and analysis, either before, during or after the initial recorded conversation. The audio includes the raw data (the recorded conversation), real-time analysis and critique by participants, discussion of potential applications of the emergent knowledge, and conclusion (maybe via post-recording reflection and analysis).
  3. Publish the episode on any podcast-hosting platform. The episode is now in the public domain.
  4. Submit a link to the episode to a journal, which embeds the podcast episode as a post (“article”) along with a short description of what it includes (like an abstract), a description of the process of creation (like the methods), the outcome of the conversation (like a conclusion), and a list of additional reading (like a reference list).
  5. The journal begins the process of accrediting the podcast by allocating peer reviewers, whose reviews are published alongside the embedded podcast in the journal.
  6. Reviewers review the “methods”, “conclusions”, “references” and knowledge claims of the podcast guests, add comments to the post, and highlight the limitations of the episode. The show notes include a description of the process, participants, additional readings, DOI, etc. This could be where the process ends; the journal has used peer review to assign a measure of “quality” to the episode and does not attempt to make a judgement on “value” (which is what journals do when they reject submissions). It is left to the listener to decide if the podcast has value for them.
  7. The following points are included for completeness as they follow a traditional iterative process following peer review. I don’t think these steps are necessary but are only included to map the workflow onto a process that most authors will be familiar with:
    1. The podcast creators make some changes to the audio file, perhaps by including new analysis and comments in the episode, or maybe by adding new information to the textual component of the episode (i.e. the show notes).
    2. The new episode is released. This re-publication of the episode would need to be classified as an entirely different version since the original episode would have been downloaded and shared to networks. An updated version would, therefore, need a new URL, a new page on the podcast hosting service, etc.

In the example workflow above, the journal never hosts the audio file and does not “publish” the podcast. It includes an embedded version of the episode, the show notes (which include the problem under discussion, the participants and their bios, an analysis of the conversation, and a list of references), as well as the full peer reviews. Readers/listeners then decide on the “importance” of the episode and whether or not to assign value to it. In other words, the readers/listeners decide what work is valuable, rather than the peer reviewers or the journal.

In summary, I’ve tried to describe why podcasts are potentially a useful format for creating and sharing the production of new knowledge, presented a framework for determining if a podcast could be considered to be scholarly, and described the workflow and some practical implications of an accreditation process using a traditional journal.

OpenPhysio | A new physiotherapy education journal

I’m really excited to announce a new project that I’ve been working on together with the folks at Physiopedia. Today we’re launching an open access, peer reviewed journal with a focus on physiotherapy education, with a few features that we think are pretty innovative in the academic publishing space. The journal is called OpenPhysio and represents what we think is a fundamental shift away from traditional ways of thinking about how we share knowledge.

Here are some of the ways we think the journal is different to more traditional publication channels:

  • Immediate publication. Your article is available to the public almost immediately after submission.
  • Peer review is open and transparent. Authors work together with peer reviewers, and the reviews and author responses are published alongside the final article, together with DOIs that make them citable objects.
  • You retain your intellectual property at no cost. OpenPhysio does not require you to transfer copyright to the journal, and there are no page fees for published articles.
  • Articles are first class internet citizens. Your articles can be enhanced with images, audio, tagging, hyperlinks, and video.

We’re still in the early stages of the project (we have no publications yet) and there’s a lot still to iron out, but we’ve decided to make it public nonetheless. This is in line with our broader thinking about publication, which is to share stuff early and then hash it out in the real world. We have Editorial and Advisory Boards and you can have a look at our policies around open access and peer review.

Now, before you write and tell me that there’s no such thing as physiotherapy education (you’d be right, by the way) we want to be clear that this is a journal aimed at physiotherapists with an emphasis on teaching and learning. it’s not about suggesting that the way physiotherapists learn is somehow different to how nurses, physicians and OTs learn. But we do think that there’s a space to explore our context in ways that may not translate well into other domains.

We want to encourage submissions from physios who are interested in learning more about teaching and learning, whether you’re supervising students or less-experienced colleagues in the clinical and community contexts, or if you’re an academic responsible for teaching in undergraduate and postgraduate classrooms. If you’re interested in teaching and learning in a physiotherapy context, we’d love it if you would consider OpenPhysio as a channel to share your ideas.

If you’d like to know more about the journal, please contact the Editor or visit the website.

I enjoyed reading (December)

This post is really delayed, mainly because I took a break from blogging over December and January. I was starting to feel an “obligation to blog”, which is when I know that I need to step back a bit and take some time off. There’s nothing worse than writing because you feel you have to, rather than actually wanting to. Now that I’ve had a break, I find myself feeling excited at the prospect of blogging again, which is a much better place to be.

9 reasons why I am NOT a social constructivist (Donald Clark): Interesting critique of the concept of social constructivism as a theory that explains learning. To be honest, I’ll admit to having accepted the authenticity of the theory because it fits in with how I believe the world is. However, I haven’t been at all critical of it. In the spirit of adopting a more critical view of my beliefs, this was a very good post to read.

Educators nod sagely at the mention of ‘social constructivism’ confirming the current orthodoxy in learning theory. To be honest, I’m not even sure that social constructivism is an actual theory, in the sense that it’s verified, studied, understood and used as a deep, theoretical platform for action. For most, I sense, it’s a simple belief that learning is, well, ‘social’ and ‘constructed’. As collaborative learning is a la mode, the social bit is accepted without much reflection, despite its obvious flaws. Constructivism is trickier but appeals to those with a learner-centric disposition, who have a mental picture of ideas being built in the mind.

Going Beyond ‘Learning to Code’: Why 2014 is the Year of Web Literacy (Doug Belshaw): I like the idea of people having a sense of how technology works. As more and more of our lives become integrated with technology, isn’t it important to understand how it affects us? How are the decisions we make increasingly influenced by those who write the code of the applications and devices we use? Think about pacemakers that determine the frequency and regularity of your heartbeat. Wouldn’t you want to make sure that there are as few software bugs as possible? My interest in this topic is more related to the idea of open source software and the importance of ensuring that as much code as possible is open for review by an objective and independent community. Mozilla’s Web Literacy standard is one small aspect of developing competence in a range of skills that are increasingly relevant to our ability to interact with others in the world.

In this post I want to argue that learning to code is part of a larger landscape that we at Mozilla call ‘web literacy’. I see that landscape as being increasingly relevant in 2014 as we come to realise that “learn to code!” is too simplistic and de-contextualised to be a useful exhortation. Web Literacy, on the other hand, is reasonably well-defined as the skills and competencies required to read, write and participate effectively online. We’ve included ‘coding/scripting’ as just one part of a wider strand identified as ‘Building’ (i.e. writing) the web. Other competencies in this strand include ‘remixing’ and ‘composing for the web’.

Do What You Love: A Selfish and Misguided Message (Dean Shareski):

By keeping us focused on ourselves and our individual happiness, DWYL [Do What You Love] distracts us from the working conditions of others while validating our own choices and relieving us from obligations to all who labor, whether or not they love it. It is the secret handshake of the privileged and a worldview that disguises its elitism as noble self-betterment. According to this way of thinking, labor is not something one does for compensation, but an act of self-love. If profit doesn’t happen to follow, it is because the worker’s passion and determination were insufficient.

Academic publishers must sort out their outdated electronic submission and review processes (Dorothy Bishop):

My relationships with journals are rather like a bad marriage: a mixture of dependency and hatred. Part of the problem is that journal editors and academics often have a rather different view of the process. Scientific journals could not survive without academics. We do the research, often spending several years of our lives to produce a piece of work that is then distilled into one short paper, which the fond author invariably regards as a fascinating contribution to the field. But when we try to place our work in a journal, we find that it’s a buyer’s market: most journals are overwhelmed with more submitted papers than they can cope with, and rejection rates are high. So there is a total mismatch: we set out naively dreaming of journals leaping at the opportunity to secure our best work, only to be met with coldness and rejection.

Side note: The above post included a screenshot of this tweet, which I enjoyed.

Selection_001

Peer review of teaching

Introduction
Peer review is a form of evaluation designed to provide feedback to teachers about their professional practice. The standard method of evaluating teaching is to ask students at the end of a module or course, for their feedback on the lecturers performance. While student feedback does have value, it also has limitations. For example, students are often not qualified to determine a lecturers knowledge base or understanding of course content. They may also lack the skills to identify appropriate levels of difficulty of the assessment tasks, as well as the appropriateness of the learning objectives as they relate to the overall curriculum.

Peer review of your teaching practice should be performed over time on different occasions, by different colleagues. This will create a more reliable measure of your teaching practice, as it goes some way towards eliminating bias. Effective peer evaluation should incorporate input from multiple sources. These can include the peer review from colleagues, but should also integrate feedback from students, personal reflection (as might be obtained from a teaching portfolio), and a review of student work. Students in particular can provide input on their perceptions of the classroom instruction process, outside-classroom interactions, and their satisfaction with the lecturer’s ability to mentor them. This integration of input from various sources allows for a more comprehensive, holistic view of teaching practice.

Why you should consider using peer review
Peer review of your teaching practice has several benefits. These include;

  • The opportunity to learn from others’ perspectives
  • Being exposed to new ideas
  • New staff learning from more experienced colleagues

However, you should also be aware of some possible pitfalls. These include bias when the observer has beliefs about teaching practice that aren’t consistent with your own, and a lack of validity (when used summatively) if it is viewed as an independent indicator of teaching ability.

Peer review process guidelines
Now that you have a form that is relevant for your context, you need to implement it. Think of the process as a collaborative one, rather than an assessment. Before you begin, you should decide what class is going to be observed, and who will observe you. It may be difficult but try and choose someone who can help identify areas in which you can improve, rather than someone you feel safe with. It may be easier asking a friend but you may find that they don’t give you the objective evaluation you need to develop your practice. The same goes for the activity you choose. You may want to go with a module you’re comfortable with and that you know well, but this doesn’t allow you much chance to improve. Instead, try to use the process as opportunity to challenge yourself.

Before the activity begins, you should meet with the observer so that you can discuss what you will be doing during the session. The aim of this is to provide some context for them to work within. For example, you may discuss the goal of your session, specific objectives you wish to achieve, the teaching strategies you will use to achieve the objective, how you will measure this achievement, as well as any concerns you would like the observer to take note of. An activity outline that they can keep may help to remind them what you’re going to try and do during your teaching activity.

The observer should arrive 10 minutes before the class begins, as arriving late is to model poor behaviour to students. They should be briefly introduced to the students, and their role explained. Finally, the observer should not ask questions during the activity, as this may detract from the process and invalidate the outcome. If the activity will go on for more than an hour, decide beforehand which components the observer will stay for.

After the teaching activity you should de-brief with the observer. This can either be done immediately, or after a short period of reflection. The advantage of doing it immediately is that everything is fresh in everyone’s minds, but which doesn’t allow time for both parties to reflect on the process. Whether you choose to do it immediately or after a reflective period, this session is where the observer can report on their observations for further discussion. This session should be led by the person who was observed, rather than be in the form of post-mortem by the observer.

Following the discussion, it is essential to decide on a set of actions that you can take in order to move forward and use the review process as a means of improving your practice. Without setting objectives for improvement based on the feedback, and then taking action to achieve those objectives, there is little point in the peer review process. Finally, the completed evaluation form, professional development objectives, and plan of action should be archived in your teaching portfolio.

Click on the image below for an example peer review form

Hints and tips

  • Peer review has been identified as one way in which teaching practices can be improved through objective feedback from colleagues
  • In order to gain the maximum benefit from peer review, there are processes that can be followed, rather than taking an ad hoc approach
  • Peer review should always be followed by a plan of action. Without acting on the feedback, the process is little more than an administrative exercise

References and sources
Peer Review of Teaching for Promotion Applications: Peer Observation of Classroom Teaching. Information, Protocols and Observation Form for Internal Peer Review Team.

Babbie, E. & Mouton, J. (2006). The Practice of Social Research. Oxford University Press. ISBN: 0195718542.

Brent, R. & Felder, R.M. (2004). A Protocol for Peer Review of Teaching. Proceedings of the 2004 American Society for Engineering Education Annual Conference & Exposition.

Butcher, C. Davies, C. and Highton, M. (2006). Designing Learning. From Module Outline to Effective Teaching. Routledge. ISBN: 9780415380300.

Graduate Attributes (2006). Curtin University of Technology.

Peer Observation Guidelines and Recommendations. University of Minnesota, Center for Teaching and Learning.

Additional reading and resources
Classroom Observation Instruments. University of Minnesota, Center for Teaching and Learning. (a list of instruments that you can use in your own teaching practice)

McKenzie, J. & Parker, N. (2011). Peer review in online and blended learning environments. Report from the Australian Learning and Teaching Council.

Harris, K., Farrell, K., Bell, M., Devlin, M. & James, R. (2008). Peer Review of Teaching in Australian Higher Education. A handbook to support institutions in developing and embedding effective policies and practices. Centre for the Study of Higher Education. ISBN 9780734040459.

Resources on Peer Observation and Review. University of Minnesota, Center for Teaching and Learning.

Twitter Weekly Updates for 2012-02-20

Twitter Weekly Updates for 2012-02-06

Twitter Weekly Updates for 2011-10-10

  • Imagine what else he could have done http://t.co/YepH4bNv #iSad #
  • Professionally judgemental http://t.co/AjvJrsmT. Guidelines on reviewing a paper, as well as responding to reviews of your own work #
  • Daily Papert http://t.co/xpwrFKN0. What do we teach in schools that doesn’t serve any useful purpose in the real world? #
  • Daily Papert http://t.co/LpS9wDaH. “When they went 2 school, the first thing they had 2 learn was 2 stop learning and 2 begin being taught” #
  • Thanks again to the support team at #realmdigital for sorting out my technical issues promptly and effectively #

Twitter Weekly Updates for 2011-09-26

  • The Daily Papert http://t.co/hpkUkaCV. Technology doesn’t do anything. People do… #
  • Academic journals remain unnecessary and unhealthy whilst open access archives such as arXiv continue to grow. http://t.co/dP2TyF1T #
  • Procrastination: On Writing Tomorrow What You Should Have Written Last Year http://t.co/z1mHd2qb #
  • UCLA Researchers Use iPhone to Track Parkinson’s Disease http://t.co/qHguaadG #
  • Disgruntled College Student Starts ‘UnCollege’ to Challenge System http://t.co/SeckkdVt #
  • Reforming Peer Review. What are the practical steps? http://t.co/T6qKI1fh #
  • @scaprogramme @mpaskevi We’re discussing moving our faculty journal from print-based to online and open access. You want to chat? #
  • RT @ambrouk: Annotem: an open-source journal authoring and publishing platform based on WordPress bit.ly/oG8LXF…was missing the “F” #
  • @scaprogramme Also, interested in moving away from the concept of the “paper” as the manifestation of results. Study can embody so much more #
  • @scaprogramme Like the idea of author profiles, research notes, linking to prev/followup studies i.e. expanding the “footprint” of the study #
  • @scaprogramme #Annotum also looks like it has potential. The thing about #OJS is that it replicates the traditional model http://ht.ly/6y00k #
  • Annotem: an open-source journal authoring and publishing platform based on WordPress – YouTube http://t.co/nFDS9gqc #
  • @scaprogramme This is great, thanks for pointing it out. Will follow up and let you know how we’re getting on 🙂 #
  • @mpaskevi Thanks very much. Seems like OJS is quite popular, noticed IRRODL runs on it too. Intrigued with WP possibility #

Twitter Weekly Updates for 2011-08-08

  • Designing for Social Norms (or How Not to Create Angry Mobs) http://ow.ly/1vmJzH #
  • @alisa_williams Do u think it’s because no1 has shown them the value of collaboration? The system expects and rewards individual performance #
  • Anatomy of an incident: Helicopter crash at UCT http://ow.ly/1vmlK3. Interesting analysis of how the info spread #
  • Tell me again what you did? http://ow.ly/1viz3M. Useful framework for writing and brief insight into a no online learning community #
  • It’s easy criticise….and fun too (apparently) http://ow.ly/1viyT1. Writing papers is hard enough without nasty reviewer comments #
  • British man survives artificial heart transplant http://ow.ly/1viyNA #
  • elearnspace › 5 ways tech startups can disrupt education http://bit.ly/pxMWHf #
  • @RonaldArendse Been thinking about how much disruption can really happen in the institutional context. Can we disrupt at all? #
  • If you could throw everything out and start again, what would your classroom look like? Would you have a classroom? #
  • Visions of Students Today http://bit.ly/nIbERY. Another video by Michael Wesch #
  • Create a Research Space | Learning Journey http://bit.ly/nVyPhF. Great tips on using a framework for writing #

Twitter Weekly Updates for 2011-07-25