Research project exploring clinicians’ perspectives of the introduction of ML into clinical practice

I recently received ethics clearance to begin an explorative study looking at how physiotherapists think about the introduction of machine learning into clinical practice. The study will use an international survey and a series of interviews to gather data on clinicians’ perspectives on questions like the following:

  • What aspects of clinical practice are vulnerable to automation?
  • How do we think about trust when it comes to AI-based clinical decision support?
  • What is the role of the clinician in guiding the development of AI in clinical practice?

I’m busy finalising the questionnaire and hope to have the survey up and running in a couple of weeks, with more focused interviews following. If these kinds of questions interest you and you’d like to have a say in answering them, keep an eye out for a call to respond.

Here is the study abstract (contact me if you’d like more detailed information):

Background: Artificial intelligence (AI) is a branch of computer science that aims to embed intelligent behaviour into software in order to achieve certain objectives. Increasingly, AI is being integrated into a variety of healthcare and clinical applications and there is significant research and funding being directed at improving the performance of these systems in clinical practice. Clinicians in the near future will find themselves working with information networks on a scale well beyond the capacity of human beings to grasp, thereby necessitating the use of intelligent machines to analyse and interpret the complex interactions of data, patients and clinical decision-making.

Aim: In order to ensure that we successfully integrate machine intelligence with the essential human characteristics of empathic, caring and creative clinical practice, we need to first understand how clinicians perceive the introduction of AI into professional practice.

Methods: This study will make use of an explorative design to gather qualitative data via an online survey and a series of interviews with physiotherapy clinicians from around the world. The survey questionnaire will be self-administered and piloted for validity and ambiguity, and the interview guide informed by the study aim. The population for both survey and interviews will consist of physiotherapy clinicians from around the world. This is an explorative study with a convenient sample, therefore no a priori sample size will be calculated.

First compute no harm

Is it acceptable for algorithms today, or an AGI in a decade’s time, to suggest withdrawal of aggressive care and so hasten death? Or alternatively, should it recommend persistence with futile care? The notion of “doing no harm” is stretched further when an AI must choose between patient and societal benefit. We thus need to develop broad principles to govern the design, creation, and use of AI in healthcare. These principles should encompass the three domains of technology, its users, and the way in which both interact in the (socio-technical) health system.

Source: Enrico Coiera et al. (2017). First compute no harm. BMJ Opinion.

The article goes on to list some of the guiding principles for the development of AI in healthcare, including the following:

  • AI must be designed and built to meet safety standards that ensure it is fit for purpose and operates as intended.
  • AI must be designed for the needs of those who will work with it, and fit their workflows.
  • Humans must have the right to challenge an AI’s decision if they believe it to be in error.
  • Humans should not direct AIs to perform beyond the bounds of their design or delegated authority.
  • Humans should recognize that their own performance is altered when working with AI.
  • If humans are responsible for an outcome, they should be obliged to remain vigilant, even after they have delegated tasks to an AI.

The principles listed above are only a very short summary. If you’re interested in the topic of ethical decision making in clinical practice, you should read the whole thing.

The fate of medicine in the time of AI

Source: Coiera, E. (2018). The fate of medicine in the time of AI.

The challenges of real-world implementation alone mean that we probably will see little change to clinical practice from AI in the next 5 years. We should certainly see changes in 10 years, and there is a real prospect of massive change in 20 years. [1]

This means that students entering health professions education today are likely to begin seeing the impact of AI in clinical practice when they graduate, and very likely to see significant changes 3-5 into their practice after graduating. Regardless of what progress is made between now and then, the students we’re teaching today will certainly be practising in a clinical environment that is very different from the one we prepared them for.

Coiera offers the following suggestions for how clinical education should probably be adapted:

  • Include a solid foundation in the statistical and psychological science of clinical reasoning.
  • Develop models of shared decision-making that include patients’ intelligent agents as partners in the process.
  • Clinicians will have a greater role to play in patient safety as new risks emerge e.g. automation bias.
  • Clinicians must be active participants in the development of new models of care that will become possible with AI.

We should also recognise that there is still a lot that is unknown with respect to where, when and how these disruptions will occur. Coiera suggests that the best guesses we can make about predicting the changes that are likely to happen should probably focus on those aspects of practice that are routine because this is where AI research will focus. As educators, we should work with clinicians to identify those areas of clinical practice that are most likely to be disrupted by AI-based technologies and then determine how education needs to change in response.

The prospect of AI is a Rorschach blot upon which many transfer their technological dreams or anxieties.

Finally, it’s also useful to consider that we will see in AI our own hopes and fears and that these biases are likely to inform the way we think about the potential benefits and dangers of AI. For this reason, we should include as diverse a group as possible in the discussion of how this technology should be integrated into practice.


[1] The quote from the article is based on Amara’s Law: “We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run.”

An introduction to artificial intelligence in clinical practice and education

Two weeks ago I presented some of my thoughts on the implications of AI and machine learning in clinical practice and health professions education at the 2018 SAAHE conference. Here are the slides I used (20 slides for 20 seconds each) with a very brief description of each slide. This presentation is based on a paper I submitted to OpenPhysio, called: “Artificial intelligence in clinical practice: Implications for physiotherapy education“.


The graph shows how traffic to a variety of news websites changed after Facebook made a change to their Newsfeed algorithm, highlighting the influence that algorithms have on the information presented to us, and how we no longer really make real choices about what to read. When algorithms are responsible for filtering what we see, they have power over what we learn about the world.


The graph shows the near flat line of social development and population growth until the invention of the steam engine. Before that all of the Big Ideas we came up with had relatively little impact on our physical well-being. If your grandfather spent his life pushing a plough there was an excellent chance that you’d spend your life pushing one too. But once we figured out how to augment our physical abilities with machines we saw significant advances in society and industry and an associated improvement in everyones quality of life.


The emergence of artificial intelligence in the form of narrowly constrained machine learning algorithms has demonstrated the potential for important advances in cognitive augmentation. Basically, we are starting to really figure out how to use computers to enhance our intelligence. However, we must remember that we’ve been augmenting our cognitive ability for a long time, from exporting our memories onto external devices, to performing advanced computation beyond the capacity of our brains.


The enthusiasm with which modern AI is being embraced is not new. The research and engineering aspects of artificial intelligence have been around since the 1950s, while fictional AI has an even longer history. The field has been through a series of highs and lows (called AI Winters). The developments during these cycles were fueled by what has become known as Good Old Fashioned AI; early attempts to explicitly design decision-making into algorithms by hard coding all possible variations of the interactions in a closed-environment. Understandably, these systems were brittle and unable to adapt to even small changes in context. This is one of the reasons that previous iterations of AI had little impact in the real world.


There are 3 main reasons why it’s different this time. The first is the emergence of cheap but powerful hardware (mainly central and graphics processing units), which has seen computational power growing by a factor of 10 every 4 years. The second characteristic is the exponential growth of data, and massive data sets are an important reason that modern AI approaches have been so successful. The graph in the middle column is showing data growth in Zettabytes (10 to the power of 21). At this rate of data growth we’ll run out metric system in a few years (Yotta is the only allocation after Zetta). The third characteristic of modern AI research is the emergence of vastly improved machine learning algorithms that are able to learn without being explicitly told what to learn. In the example here, the algorithm has coloured in the line drawings to create a pretty good photorealistic image, but without being taught any of the concepts i.e. human, face, colour, drawing, photo, etc.


We’re increasingly seeing evidence that in some very narrow domains of practice (e.g. reasoning and information recall), machine learning algorithms can outdiagnose experienced clinicians. It turns out that computers are really good at classifying patterns of variables that are present in very large datasets. And diagnosis is just a classification problem. For example, algorithms are very easily able to find sets of related signs and symptoms and put them into a box that we call “TB”. And increasingly, they are able to do this classification better than the best of us.


It is estimated that up to 60% of a doctors time is spent capturing information in the medical record. Natural language processing algorithms are able to “listen” to the ambient conversation between a doctor and patient, record the audio and transcribe it (translating it in the process if necessary). It then performs semantic analysis of the text (not just keyword analysis) to extract meaningful information which it can use to populate an electronic health record. While the technology is in a very early phase and not yet safe for real world application it’s important to remember that this is the worst it’s ever going to be. Even if we reach some kind of technological dead end with respect to machine learning and from now on we only increase efficiency, we are still looking at a transformational technology.


An algorithm recently passed the Chinese national medical exam, qualifying (in theory) as a physician. While we can argue that practising as a physician is more than writing a text-based exam, it’s hard not to acknowledge the fact that – at the very least – machines are becoming more capable in the domains of knowledge and reasoning that characterise much of clinical practice. Again, this is the worst that this technology is ever going to be.


This graph shows the number of AI applications under development in a variety of disciplines, including medicine (on the far right). The green segment shows the number of applications where AI is outperforming human beings. Orange segments show the number of applications that are performing relatively well, with blue highlighting areas that need work. There are two other points worth noting: medical AI is the area of research that is clearly showing the most significant advances (maybe because it’s the area where companies can make the most money); and all the way at the far left of the graph is education, showing that there may be some time before algorithms are showing the same progress in teaching.


Contrary to what we see in the mainstream media, AI is not a monolithic field of research; it consists it consists of a wide variety of different technologies and philosophies that are each sometimes referred to under the more general heading of “AI”. While much of the current progress is driven by machine learning algorithms (which is itself driven by the 3 characteristics of modern society highlighted earlier), there are many areas of development, each of which can potentially contribute to different areas of clinical practice. For the purposes of this presentation, we can define AI as any process that is able to independently achieve an objective within a narrowly constrained domain of interest (although the constraints are becoming looser by the day).


Machine learning is a sub-domain of AI research that works by exposing an algorithm to a massive data set and asking it to look for patterns. By comparing what it finds to human-tagged patterns in the data, developers can fine-tune the algorithm (i.e. “teach it) before exposing it to untagged data and seeing how well it performs relative to the training set. This generally describes the “learning” process of machine learning. Deep learning is a sub-domain of machine learning that works by passing data through many layers, allocating different weights to the data at each layer, thereby coming up with a statistical “answer” that expresses an outcome in terms of probability. Deep learning neural networks underlie many of the advances in modern AI research.


Because machine and deep learning algorithms are trained on (biased) human-generated datasets, it’s easy to see how the algorithms themselves will have an inherent bias embedded in the outputs. The Twitter screenshot shows one of the least offensive tweets from Tay, an AI-enabled chatbot created by Microsoft, which learned from human interactions on Twitter. In the space of a few hours, Tay became a racist, sexist, homophobic monster – because this is what it learned from how we behave on Twitter. This is more of an indictment of human beings than it is of the algorithm. The other concern with neural networks is that, because of the complexity of the algorithms and the number of variables being processed, human beings are unable to comprehend how the output was computed. This has important implications when algorithms are helping with clinical decision-making and is the reason that resources are being allocated to the development of what is known as “explainable AI”.


As a result of the changes emerging from AI-based technologies in clinical practice we will soon need to stop thinking of our roles in terms of “professions” and rather in terms of “tasks”. This matters because increasingly, many of the tasks we associate with our professional roles will be automated. This is not all bad news though, because it seems probable that increased automation of the repetitive tasks in our repertoire will free us up to take on more meaningful tasks, for example, having more time to interact with patients. We need to start asking what are the things that computers are better at and start allocating those tasks to them. Of course, we will need to define what we mean by “better”; more efficient, more cost-effective, faster, etc.


Another important change that will require the use of AI-based technologies in clinical practice will be the inability of clinicians to manage – let alone understand – the vast amount of information being generated by, and from, patients. Not only are all institutional tests and scans digital but increasingly, patients are creating their own data via wearables – and soon, ingestibles – all of which will require that clinicians are able to collect, filter, analyse and interpret these vast streams of information. There is evidence that, without the help of AI-based systems, clinicians simply will not have the cognitive capacity to understand their patients’ data.


The impact of more patient-generated health data is that we will see patients being in control of their data, which will exist on a variety of platforms (cloud storage, personal devices, etc.), none of which will be available to the clinician by default. This means that power will move to the patient as they make choices about who to allow access to their data in order to help them understand it. Clinicians will need to come to terms with the fact that they will no longer wield the power in the relationship and in fact, may need to work within newly constituted care teams that include data scientists, software engineers, UI designers and smart machines. And all of these interactions will be managed by the patient who will likely be making choices with inputs from algorithms.


The incentives for enthusiastic claims around developments in AI-based clinical systems are significant; this is an acdemic land grab the likes of which we have only rarely experienced. The scale of the funding involved puts pressure on researchers to exaggerate claims in order to be the first to every important milestone. This means that clinicians will need to become conversant with the research methods and philosophies of the data scientists who are publishing the most cutting edge research in the medical field. The time will soon come when it will be difficult to understand the language of healthcare without first understanding the language of computer science.


The implications for health professions educators are profound, as we will need to start asking ourselves what we are preparing our graduates for. When clinical practice is enacted in an intelligent environment and clinicians are only one of many nodes in vast information networks, what knowledge and skills do they need to thrive? When machines outperform human beings in knowledge and reasoning tasks, what is the value of teaching students about disease progression, for example? We may find ourselves graduating clinicians who are well-trained, competent and irrelevant. It is not unreasonable to think that the profession called “doctor” will not exist in 25 years time, having been superseded by a collective of algorithms and 3rd party service providers who provide more fine-grained services at a lower cost.


There are three new literacies that health professions educators will need to begin integrating into our undergraduate curricula. Data literacy, so that healthcare graduates will understand how to manage, filter, analyse and interpret massive sets of information in real-time; Technological literacy, as more and more of healthcare is enacted in digital spaces and mediated by digital devices and systems; and Human literacy, so that we can become better at developing the skillsets necessary to interact more meaningfully with patients.


There is evidence to suggest that, while AI-based systems outperform human beings on many of the knowledge and reasoning tasks that make up clinical practice, the combination of AI and human originality results in the most improved outcomes of all. In other words, we may find that patient outcomes are best when we figure out how to combine human creativity and emotional response with machine-based computation.


And just when we’re thinking that “creativity” and “originality” are the sole province of human beings, we’re reminded that AI-based systems are making progress in those areas as well. It may be that the only way to remain relevant in a constantly changing world is to develop our ability to keep learning.

OpenPhysio abstract: Artificial intelligence in clinical practice – Implications for physiotherapy education

Here is the abstract of a paper I recently submitted to OpenPhysio, a new open-access journal with an emphasis on physiotherapy education.

About 200 years ago the invention of the steam engine ushered in an era of unprecedented development and growth in human social and economic systems, whereby human labour was supplanted by machines. The recent emergence of artificially intelligent machines has seen human cognitive capacity augmented by computational agents that are able to recognise previously hidden patterns within massive data sets. The characteristics of this second machine age are already influencing all aspects of society, creating the conditions for disruption to our social, economic, education, health, legal and moral systems, and which will likely to have a far greater impact on human progress than did the steam engine. As AI-based technology becomes increasingly embedded within devices, people and systems, the fundamental nature of clinical practice will evolve, resulting in a healthcare system requiring profound changes to physiotherapy education. Clinicians in the near future will find themselves working with information networks on a scale well beyond the capacity of human beings to grasp, thereby necessitating the use of intelligent machines to analyse and interpret the complex interactions of data, patients and the newly-constituted care teams that will emerge. This paper describes some of the possible influences of AI-based technologies on physiotherapy practice, and the subsequent ways in which physiotherapy education will need to change in order to graduate professionals who are fit for practice in a 21st century health system.

Read the full paper at OpenPhysio (note that this article is still under review).

How my students do case studies in clinical practice

Our students do small case studies as part of their clinical practice rotations. The basic idea is that they need to identify a problem with their own practice; something that they want to improve. They describe the problem in the context of a case study which gives them a framework to approach the problem like a research project. In this post I’ll talk about the process we use for designing, implementing, drafting and grading these case studies.

There are a few things that I consider to be novel in the following approach:

  1. The case studies are about improving future clinical practice, and as such are linked to students’ practices i.e. what they do and how they think
  2. Students are the case study participants i.e. they are conducting research on themselves
  3. We shift the emphasis away from a narrow definition of “The Evidence” (i.e. journal articles) and encourage students to get creative ideas from other areas of practice
  4. The grading process has features that develop students’ knowledge and skills beyond “Conducting case study research in a clinical practice module”

Design

Early on in their clinical practice rotations, the students identify an aspect of that block that they want to learn more about. We discuss the kinds of questions they want to answer, both in class and by email. Once the topic and question are agreed, they do mini “literature” reviews (3-5 sources that may include academic journals, blogs, YouTube videos, Pinterest boards…whatever) to explore the problem as described by others. They also use the literature to identify possible solutions to their problems, which then get incorporated into the Method. They must also identify what “data” they will use to determine an improvement in their performance. They can use anything from personal reflections to grades to perceived level of comfort…anything that allows them to somehow say that their practice is getting better.

Implementation and drafting of early case studies

Then they try an intervention – on themselves, because this is about improving their own practice – and gather data to analyse as part of describing a change in practice or thinking.  They must also try to develop a general principle from the case study that they can apply to other clinical contexts. I give feedback on the initial questions and comment on early drafts to guide the projects and also give them the rubric that will be used to grade their work.

Examples of case studies from last semester include:

  • Exploring the impact of meditation and breathing techniques to lower stress before and during clinical exams, using heart rate as a proxy for stress – and learning that taking a moment to breathe can help with feeling more relaxed during an exam.
  • The challenges of communicating with a patient who has expressive aphasia – and learning that the commonly suggested alternatives are often 1) very slow, 2) frustrating, and 3) not very effective.
  • Testing their own visual estimation of ROM against a smartphone app – and learning that visual estimation is (surprise) pretty poor.
  • Exploring the impact of speaking to a patient in their own language on developing rapport – and learning that spending 30 minutes every day learning a few new Xhosa words made a huge difference to how likely the patient was to agree to physio.

Submission and peer grading

Students submit hard copies to me so that I can make sure all submissions are in. Then I take the hard copies to class and randomly assign 1 case study to each student. They pair up (Reviewer 1 and 2) and we go through the case studies together, using the rubric as a guide. I think out loud about each section of the rubric, explaining what I’m looking for in each section and why it’s important for clinical practice. For example, if we’re looking at the “Language” section I explain why clarity of expression is important for describing clinical presentations, and why conciseness allows them to practice conveying complex ideas quickly (useful for ward rounds and meetings). Spelling and grammar are important, as is legibility, to ensure that your work is clearly understandable to others in the team. I go through these rationales while the students are marking and giving feedback on the case studies in front of them.

Then they swap case studies and fill out another rubric for the case study that their team member has just completed. We go through the process again, and I encourage them to look for additional places to comment on the case study. Once that’s done they compare their rubrics for the two case studies in their team, explaining why certain marks and comments were given for certain sections. They don’t have to agree on the exact mark but they do have to come to consensus over whether each section of the work is “Poor”, “Satisfactory” or “Good”. Then they average their marks and submit it to me again.

I take all the case studies with their 2 sets of comments (on the rubric) and feedback (on the case study itself) and I go through them all myself. This means I can focus on more abstract feedback (e.g. appropriateness of the question, analysis, ethics, etc.) because the students have already commented on much of the structural, grammatical and content-related issues.

Outcomes of the process

For me, the following outcomes of the process are important to note:

  1. Students learn how to identify an area of their own clinical practice that they want to improve. It’s not us telling them what they’re doing wrong. If we want lifelong learning to happen, our students must know how to identify areas for improvement.
  2. They take definite steps towards achieving those improvements because the case study requires them to implement an intervention. “Learning” becomes synonymous with “doing” i.e. they must take concrete steps towards addressing the problem they identified.
  3. Students develop the skills they need to find answers to questions they have about their own practice. Students learn how to regulate their own learning.
  4. Each student gets 3 sets of feedback on their case study. It’s not just me – the external “expert” – telling them how to improve, it’s their peers as well.
  5. Students get exposed to a variety of other case studies across a spectrum of quality. The peer reviewers need to know what a “good” case study looks like in order to grade one. They learn what their next case study should look like.
  6. The marking time for 54 case studies goes down from about 10 hours (I give a lot of feedback) to about 3 hours. I don’t have to give feedback on everything because almost all of the common errors are already identified and highlighted.
  7. Students learn how I think when I’m marking their work, which helps them to make different choices for the next case study. This process allows them access to how I think about case study research in clinical practice, which means they are more likely to improve their next submission, knowing what I’m looking for.

In terms of the reliability of the peer marking and feedback, I noted the following when I reviewed the peer feedback and grades from earlier in the year:

  • 15 (28%) students’ marks went up when I compared my mark with the peer average, 7 (13%) students’ marks went up by 5% or more, and 4 (7%) students went from “Fail” to “Pass”.
  • 7 (13%) students’ marks went down, 3 (6%) by 5% or more, and 0 students went from “Pass” to “Fail”.
  • 28 (52%) students’ marks stayed the same.

The points I take from the above is that it’s really important for me to review the marks and that I have a tendency to be more lenient with marking; more students had mark increases and only 3 students’ marks went down by what I would consider a significant amount. And finally, more than half the students didn’t get a mark change at all, which is pretty good when you think about it.

 

 

First draft of article on Informed Consent for the Clinical Teacher

Featured image “Diverging paths” by SDJ on Deviant Art.

I’ve just finished the first draft of my 13th article in the Clinical Teacher mobile app. It’s been a LONG time since I’ve managed to put some effort into this project, mainly for personal reasons (my second daughter was born in March last year and we moved house in September) but also because I found 2015 to be a very challenging year in terms of prioritising my writing projects (I have another post in progress where I talk about some new strategies for prioritisation in my workflow).

Anyway, here is the draft of my article on Informed Consent, which I’ll be publishing in the app within the next couple of weeks.

Introduction

A successful relationship between patient and health care practitioner is based on trust, which is developed partly by respecting the autonomy of the patient i.e. their right to make their own decisions about their bodies. Informed consent is the exercise of informed choice by a patient who has the capacity to give consent and is therefore a component of developing a trusting relationship between patient and therapist (Wilders, 2013). Patients need complete and honest information about various aspects of their health care, including diagnosis, prognosis, treatment options, likely treatment outcomes, common or serious side effects, and the timescale of the treatment in order for them to give their informed consent (Tippet, 2005).

Informed consent

The patient is the one who determines what is in their best interests, not the practitioner. Practitioners may recommend a course of action but should not pressurise patients to accept their advice. The practitioner must ensure that their opinion is a balanced point of view, providing the patient with enough detail from the various perspectives that they are able to give their own, voluntary consent. It is possible for patients to nominate a third party (provided they are still competent to make such a decision) to give consent on their behalf. This must be provided by the patient in writing.

Health care professionals must be confident that they have received consent from a patient – or other valid authority – before they perform an examination or investigation, provide treatment, or involve patients in teaching and research (General Medical Council, 2008). It should be based on the sharing of correct information, appropriate communication, and understanding and agreement on the part of the patient. In order for the process of informed consent to be correct and legal, the patient must have (Wilder, 2013):

  1. Knowledge of the nature or extent of the harm or risk
  2. Appreciated and understood the risk
  3. Consented to the risk
  4. The consent must have been comprehensive

All aspects of health care involves decision-making by patients and healthcare providers and the principles of informed consent apply to those decisions, from the treatment of minor conditions, to major interventions with significant risks or side effects. Whatever the context in which medical decisions are made, the health care professional must work in partnership with patients to ensure appropriate care. In order to achieve this, you should (General Medical Council, 2008):

  • Listen to patients and respect their views about their health
  • Discuss patients’ diagnosis, prognosis, treatment and care with them
  • Share the information that patients want or need in order to make decisions about their health
  • Maximise patients’ opportunities, and their ability, to make decisions for themselves
  • Respect the decisions that patients make

Written consent

Patients can express consent either orally or in written form. However, it is strongly recommended that consent be obtained in writing. This can be done either in the case notes, or on a specific consent form. The consent form should contain the information provided to the patient, specific requests by the patient and the scope of the consent given.

Once written consent has been given once, the practitioner can simply remind patients of the initial instance, if they present at a later date with a similar condition. However, this consent should be revisited regularly to ensure that the patient is still comfortable with the procedure. Verbal reminders and responses should still be noted in writing in the patient’s file. When a patient presents with a new condition, fresh consent must be obtained and appropriately documented.

Implied consent

Practitioners should use caution if relying on a patient’s compliance with a procedure as an indicator of consent because submission in itself may not indicate consent (Wilder, 2013). Just because a patient goes along with what is happening, does not mean that they understand the nature, extent and risks that are associated with it. Consent must be expressed explicitly at all times.

Sharing information

Informed consent is premised on the concept of sharing information with the patient. There is no single approach that will suit every patient, or apply in all circumstances. Some patients may want more or less information or involvement in the decision making process, depending on their needs. In addition, some patients may need additional support to understand information and express their views and preferences (General Medical Council, 2008).

The way you choose to discuss the various aspects of a patient’s condition, including diagnosis, prognosis and the treatment options that are available, can be just as important as the information itself (General Medical Council, 2013). When sharing information with a patient (or their family), you should:

  • Share information in a way that the patient can understand and when possible, in a place and at a time when they are best able to understand and retain it.

  • Be considerate when sharing information that the patient may find distressing.

  • Involve other members of the healthcare team in discussions with the patient, if this is appropriate.

  • Give the patient time to reflect, before and after they make a decision, especially if the information is complex or what you are proposing involves any risk.

  • Make sure the patient knows if there is a time limit on making their decision, and who they can contact if they have any questions or concerns.

  • Give information to patients in a balanced way. For example, if you recommend a course of action, you should explain your reasons for doing so but also be aware of putting unnecessary pressure on a patient to accept your advice.

  • Support your suggestions with written material or other visual aids (make sure the information is accurate and up to date).

  • Check to determine if the patient needs any additional support in order to understand the information you shared, to communicate their wishes, or to make a decision.

Bear in mind that some barriers to understanding and communication may not be obvious. For example, a patient may have certain anxieties that they have not mentioned before, or may be affected by pain or other underlying problems. Support for patients might include using an advocate or interpreter, asking those close to the patient about the patient’s communication needs, or giving the patient a written or audio record of the discussion and any decisions that were made.

The information provided to the patient should include the following:

  • The nature and extent of the procedure, using language that the patient understands
  • Whether students will be involved in the procedure, and the extent to which they will be involved
  • Reminders that the patient can withdraw consent at any time i.e. change their minds
  • Reminders that the patient can seek a second opinion
  • Details of the costs applicable (if relevant)

Possible reasons for not sharing information

Remember that patients are vulnerable and that overzealous truth-telling may be harmful to them. As a result they have an equal right to refuse to receive information about themselves and their condition (Tippet, 2005). From the General Medical Council (2008): “You should not withhold information necessary for decision making unless you judge that disclosure of some relevant information would cause the patient serious harm”.

If a patient has the capacity to make their own decisions but appears to not want to have relevant information shared with them, the health care practitioner should consider the following steps (General Medical Council, 2008):

  1. No-one else can make a decision on behalf of an adult who has the capacity to give their consent. If a patient asks you to make decisions on their behalf or wants to leave decisions to a relative, partner, friend, carer or another person close to them, you should explain that it is still important that they understand the options open to them, and what the treatment will involve. If they do not want this information, you should try to find out why.

  1. If, after discussion, a patient still does not want to know in detail about their condition or the treatment, you should respect their wishes, as far as possible. But you must still give them the basic information they need in order to give their consent to a proposed investigation or treatment.

  1. If a patient insists that they do not want even this information you must explain the potential consequences of them not having it, particularly if it means that their consent is not valid. You must record the fact that the patient has declined this information making it clear to them that they can change their mind at any time.

  1. You should not withhold the information that is necessary for making decisions for any other reason, including when a relative, partner, friend or carer asks you to, unless you believe that giving it would cause the patient serious harm. In this context serious harm means more than the patient simply becoming upset or deciding to refuse treatment.

  1. If you withhold information from the patient you must record your reason for doing so in the patient’s medical records and be prepared to explain and justify your decision. You should regularly review that decision and consider whether you could give information to the patient later, without causing them serious harm.

Obstacles to sharing information

Because of time or resource limitations, it might be difficult to give patients as much information or support in the decision making process as you, or they, would like. To help with this, consider the role that other members of the healthcare team might play and what other sources of information and support are available. For example, do you have access to patient information leaflets, advocacy services, expert patient programmes, or support groups for people with specific conditions?

Do your best to make sure that patients with additional needs, such as those with disabilities, have the time and support they need to make a decision. In all cases, treat patients fairly and make sure that you do not discriminate against them by providing them with less detailed information than you might do for others. If you believe that limits on your time or the information that is available for patients is compromising their ability to make an informed choice, you should raise your concerns with your line manager or other authority (General Medical Council, 2008).

Conditions for consent

In order for a patient to give valid informed consent, there are three components that must be present; disclosure, capacity and voluntariness (Faden & Beauchamp, 1986). Disclosure requires the therapist to provide the patient with the necessary information to make an independent decision. However, it is not enough that the therapist merely provides the information; they must also ensure that the patients have adequate comprehension of the information they provide. This latter aspect of the process implies that the consent form be written in language that is understandable by the general population, in addition to determining the level of the patient’s understanding during the initial assessment. Capacity refers to the ability of the patient to both understand the information provided and to subsequently form a reasonable judgement of the potential consequences of their decision. Finally, voluntariness refers to the patient’s right to freely exercise their decision-making without being subjected to external pressure such as coercion, manipulation, or undue influence (Beauchamp & Childress, 1994).

Capacity to give informed consent

In order to give consent patients must be able to understand the procedure that is being recommended, apply reasoning to consider the consequences of the procedure and its alternatives, appreciate the way in which this information applies to them, and be able to make a subsequent logical choice. Although psychiatric illness, in and of itself, does not change the presumption that an individual is competent, a patient’s ability to appreciate the consequences of a particular decision maybe shaped by specific mental symptoms. Thus, cognitive deficits (e.g., due to dementia or associated with depression) can impair the ability to recall and understand information about the procedure, suicidal ideas can affect perceptions of mortality risks, and the ambivalence and indecision that often occurs with depression can limit patients’ ability to make a choice about treatment. If a patient lacks decisional capacity, substituted consent (possibly by a family member) or a judicial hearing may be necessary (Lapid, Rummans, Pankratz & Appelbaum, 2004).

Therapists must always work from the premise that every adult patient has the capacity to make decisions about their health. This includes the ability to make decisions about whether to agree to, or refuse, an assessment, investigation or treatment. You should only regard a patient as lacking capacity once it is clear that, having been given all appropriate help and support, they cannot understand, retain, use or weigh up the information needed to make that decision, or communicate their wishes. It is essential that the therapist does not make the assumption that a patient lacks the capacity to make a decision because of their age, a disability, their appearance, behaviour, medical condition (including mental illness), their beliefs, their apparent inability to communicate, or the fact that they make a decision that you disagree with (General Medical Council, 2008).

If patients are not able to make decisions for themselves, the health care professional must work with those who are close to the patient and with other members of the team. They must take into account those views or preferences that were expressed by the patient and must also be aware of the legal context when a patient lacks the capacity to make their own choices (General Medical Council, 2008).

Children and informed consent

As children often lack the ability or legal power to provide true informed consent for medical decisions, it falls on parents (or legal guardians) to provide permission for medical assessments or procedures to be conducted on children. This “consent by proxy” usually works reasonably well but can lead to ethical dilemmas. This is particularly true when the judgement of the guardians differ with the therapist with regard to what constitutes an appropriate decision.

Children who are legally emancipated and unemancipated minors who are deemed to have medical decision making capacity, may be able to provide consent without the need for parental permission, although this depends on the laws of the country the child lives in (Committee on Bioethics, 1995).

Children are usually presumed to be incompetent to consent, but depending on their age and the influence of other factors, they can sometimes be asked to provide Informed assent.

Informed assent means a child’s agreement to medical procedures in circumstances where he or she is not legally authorised or lacks sufficient understanding for giving consent competently.

De Lourdes Levy, Larcher & Kurz (2003)

Medical professionals are advised to seek the assent of older children and adolescents by providing age appropriate information to these children to help empower them in the decision-making process (Committee on Bioethics, 1995). It should be noted that only legal guardians are able to provide informed consent for a child, and not adult siblings. In addition, parents may not order the termination of a treatment that is required to keep a child alive even if they feel it is in the best interests of the child.

Responsibility for seeking consent

It is the responsibility of the practitioner who will be providing the treatment, to obtain consent, since they will have a comprehensive understanding of the process, how it is to be carried out and the risks involved. In cases where the responsibility to obtain consent must be delegated to another party, they must ensure that the person they are delegating to:

  • Is suitably trained and qualified to take on the responsibility
  • Has sufficient knowledge of the investigation or treatment, and understands the risks involved
  • Understands, and agrees to act in accordance with established guidelines and legal frameworks that are related to issues of informed consent

If you delegate the responsibility of seeking consent to another party, you are still responsible for ensuring that the patient has been given enough time and information to make an informed decision, and has given their consent, before you start any investigation or treatment (General Medical Council, 2008).

Informed consent in research

Medical research is overseen by an institutional ethics committee that usually also oversees the informed consent process as part of any research project involving human beings or animals. Ethics clearance is applied for by researchers via a detailed proposal document, that lays out in detail – among other things – the ways in which the ethics inherent in the project are being considered.

Differences in local contexts (e.g. language and social norms) make the issue of informed consent in research a complex topic. There are five benchmarks for evaluating informed consent procedures in local contexts, with particular reference to developing countries (Emanuel, Wendler, Killen & Grady, 2004):

  1. The community should be involved in drawing up recruitment and incentive procedures that are consistent with local cultural, political and social practices. For example, in some cultures an incentive might be expected, while in others it could be considered offensive.

  2. Disclosure of information should be sensitive to the local context, using local languages, culturally appropriate idioms, and analogies that the local population will understand.

  3. Researchers may need to obtain consent from a range of different “spheres”, including community leaders, elders or other respected community members, and the heads of family. Note that this is not to imply that these individuals grant consent on behalf of other adults, only that they give permission for the researchers to enter the community.

  4. Researchers should use informed consent procedures that are contextually acceptable within a community but which also enable independent observers to verify that voluntary consent was obtained.

  5. Individuals must be made aware that their right to refuse to participate, or to withdraw from the study is actually enforced. Community or familial coercion or retribution needs to be guarded against, especially if compensation for participation is offered.

The World Health Organisation (WHO) has published a series of templates for Informed Consent in a variety of contexts, including research that involves children requiring parental consent, qualitative and clinical studies. These templates may be used to guide principal researchers as they develop an approach to informed consent in their research practices.

Seeking consent in the research process brings with it another set of challenges, especially in the social sciences where there is often little or no risk to participants. In addition, the fact that participants are aware that they are involved in a study may cause them to change their default behaviour (the Hawthorne Effect). In cases where the researcher is concerned that seeking consent will modify the outcomes of the study, the requirement for consent may be waived. However, this is only done after an Ethics Committee has weighed the possible risk to the study participants versus the benefit to society, as well as whether participants are present in the study of their own will and that they will be treated fairly.

Reviewing consent

Informed consent – especially in the clinical context – should be thought of as an ongoing dialogue between patient and healthcare practitioner (Emanuel, et al., 2004). The decision made by the patient should be reviewed at different points before treatment begins, to ensure that the patient’s point of view is consistent and can therefore be relied on. Before beginning treatment you or a member of the healthcare team should check that the patient still wants to go ahead and you should respond to any new or repeated concerns or questions they raise. This is particularly important if:

  1. A significant amount of time has passed since the initial decision was made.
  2. There have been material changes in the patient’s condition, or in any aspect of the proposed investigation or treatment.
  3. New information has become available, for example about the risks of treatment or other treatment options.

You need to make sure that patients are kept informed about the progress of their treatment and that they are able to make decisions at all stages, not just in the initial stage. If the treatment is ongoing you should make sure that there are clear arrangements in place to review decisions and if necessary, to make new ones (General Medical Council, 2008).

Conclusion

Informed consent is an essential aspect of clinical practice and medical research. The process of sharing relevant information with a patient to ensure that they can make an informed choice about their bodies and their health, is a central principle of ethics. Patient autonomy is premised on the idea that they – not the healthcare practitioner – are best positioned to make decisions about the relative risks and benefits of choosing a particular course of action.

Sharing information is an important aspect of consent, although what information to share, how much and when, are sometimes difficult to determine, especially in cases where patients are not competent. It is the responsibility of the healthcare practitioner to ensure that consent is obtained before proceeding down a particular path of management, and in cases where the actual process is delegated, it is the practitioners responsibility to ensure that the person the task is delegated to is trained to carry it out.

Seeking informed consent is complex, time consuming, possibly frustrating, and may even require health professionals to reconsider the role of power in their patient relationships. However, not only is it a legal requirement to involve the patient in decision-making around management, but it is a foundational ethical principle to adhere to in clinical practice.

References

Beauchamp, T.L. & Childress, J.F. (1994). Principles of Biomedical Ethics (4th edition). New York: Oxford University Press.

Committee on Bioethics (1995). Informed consent, parental permission, and assent in pediatric practice. Pediatrics, 95(2):314-7.

De Lourdes Levy, M., Larcher, V., & Kurz, R. (2003). Informed consent / assent in children. Statement of the Ethics Working Group of the Confederation of European Specialists in Paediatrics. European Journal of Pediatrics. 2003 Sep;162(9):629-33.

Emanuel, E. J., Wendler, D., Killen, J., & Grady, C. (2004). What makes clinical research in developing countries ethical? The benchmarks of ethical research. The Journal of infectious diseases, 189(5), 930–7.

Faden, R.R. & Beauchamp, T.L. (1986). A History and Theory of Informed Consent. New York: Oxford University Press.

General Medical Council (2008). Consent guidance: patients and doctors making decisions together.

Health Professions Council of South Africa (2008) Ethical rules, regulations and policy guidelines: Informed consent.

Lapid, M.I., Rummans, T.A., Pankratz, V.S. & Appelbaum, P.S. (2004). Decisional capacity of depressed elderly to consent to electroconvulsive therapy. Journal of Geriatric Psychiatry and Neurology, 17(1):42–46.

Spandorfer, J., Pohl, C. A., Rattner, S. L., & Nasca, T. J. (2010). Professionalism in medicine: A case-based guide for medical students. Cambridge University Press.

Tippett, V. (2005). “Trust me…I’m a medical student”: Truth and trust for student doctors. The Clinical Teacher, 2(1), 21–24.

Wilder, C. Seeking patients’ informed consent: The ethical considerations. Health Professions Council of South Africa news bulletin, March, 2013

World Health Organisation (n.d.). Informed Consent form templates.

Proposal abstract: Training in the ICU for physiotherapy students with a visual impairment (a case study)

Abstract for a project proposal that I submitted for ethics review earlier this week. If it gets approved we’ll begin data collection on our first visually impaired undergraduate student placement in the intensive care unit.

The Department of Physiotherapy at the University of the Western Cape (UWC) began accepting students with visual impairments (VI) into the undergraduate programme in 1996. To date, eight students with visual impairments have graduated with degrees in physiotherapy, all of whom have gone on to successful employment in the health system. In this area, the department has played an important role in leading transformative change, not only in the broader context of higher education but specifically in the area of providing equal opportunities for professional training for all South Africans.

While the department has done well to provide equal opportunities to students with VI in the general undergraduate course, we have yet to place a student with VI into the ICU setting as part of their clinical rotations. Early in 2015 however, the department engaged in a series of discussions with one of our final year students with VI, as well as clinicians and lecturers and decided to explore the possibility of placing the student into the ICU. This would enable us to align ourselves with national policies and priorities. As part of this process of placing a student in the ICU setting we want to describe the facilitators and barriers that exist, as seen from the perspective of those involved in the process. The aim of the study is therefore to explore the experiences of the student, clinicians, academics and peers, with the placement of a student with a VI in the ICU.

How do you negotiate this environment when you can’t see very well?

The project will make use of a case study design that aims to describe the process of placing an undergraduate physiotherapy student with VI in an ICU setting as part of a clinical practice rotation. The case study will include data gathered from the student’s reflective clinical diary as well as in-depth interviews with the student, clinical supervisor, VI and clinical coordinators in the physiotherapy department at UWC, and the clinician who is responsible for overseeing the student in the ICU. Peers who have engaged with the student during the specific clinical placement will also be included, and will be identified during the process.

The interviews will be audio recorded and then sent away for transcription. The transcribed interviews will be anonymised and thematically analysed in order to determine themes related to barriers and facilitators that are relevant to the student’s learning. The transcripts – along with the analyses – will be shared with participants in order to ensure that the themes that emerged are consistent with the meaning that they had intended during the interviews.

Proposal abstract: The use of medical and health-related smartphone apps by South African physiotherapists (a survey)

Abstract for a proposal I submitted earlier this week. This proposal is part of a larger project where we are developing an evaluation tool for decision-making around app use for physiotherapists in clinical practice, determining reliability of a range of pedometer apps on different hardware platforms, and evaluating the information provided by the top exercise-prescription apps. This proposal describes a national survey of South African physiotherapists around their app use in clinical practice. After the project concludes later this year we will prepare a report for physiotherapists interested in using medical and health-related apps as part of their professional practice.

The medical literature now refers to the practice of “prescribing apps” to patients, who monitor their activity and use the resulting data to change their behaviours and help reduce the risks associated with their condition. Mobile apps are expected to play an increasingly important role in health care, where patient data can be shared with health providers and funders to support decision-making at higher levels in the health system. Among physiotherapists there is evidence demonstrating that clinicians are using apps more frequently at the bedside, with the aim of increasing efficiency by enabling more rapid decision-making at the point of care.

Medical and health-related apps are increasing in number and scope on a daily basis. With the proliferation of apps providing medical and health-related information for both professionals and consumers, it is important to determine if these apps can be safely recommended for patients and healthcare professionals. However, there is a very limited evidence-base to inform decision-making when it comes to choosing and using medical and health-related mobile apps as part of clinical practice. In order to begin making informed decisions about how to make effective use of mobile apps in healthcare, there is a need for data describing current uses of apps by physiotherapists in the South African context. This study therefore aims to determine South African physiotherapists’ use of medical and health-related apps as part of their professional practice.

Fitness-Apps

This project makes use of a cross-sectional, descriptive design that aims to provide a snapshot of the profile of medical and health-related app use among South African physiotherapists. A survey will be conducted within this population in order to develop a better understanding of this emerging field of research in the South African physiotherapy context.

The population for the survey will include all South African physiotherapists who are registered with the SASP in 2015. There are currently about 3500 registered physiotherapists with the SASP and all will be invited to participate in the online survey. A self-administered questionnaire was developed using the available literature and will be piloted among 3rd year physiotherapy students in the University of the Western Cape Department of Physiotherapy. The questionnaire includes mostly closed-ended questions that aim to identify how physiotherapists make use of medical and health-related apps, as well as their experiences around the use of those apps as part of their professional practice.

The questionnaire will be administered using Google Forms, and the link to the survey emailed to all qualified physiotherapists who are registered with the South African Society of Physiotherapy. Since the survey will be conducted online, data will automatically be captured in a spreadsheet with no opportunity for errors in the data capturing process. The data will then be downloaded as an Excel spreadsheet.

The data gathered will be analysed using descriptive and inferential statistics. Descriptive data will be presented using frequencies and tables, and inferential statistics will be used to determine relationships between variables. In particular, we will look at the relationships between participant demographic information and trends related to app use.

The Delphi method in clinical research

Thank you to Conran Joseph for his contribution to this post. We began developing this content as part of another project that we’re working on (more to come on that later) and then extended it as I made notes for a paper than I’m writing for my PhD.

Introduction
The Delphi method was developed in the 1950’s with the purpose of soliciting expert opinion in order to reach consensus  (Dalkey & Helmer, 1963, p. 458). It was so named because it was originally developed as a systematic, interactive means of forecasting or prediction, much like ancient Grecians came to the Oracle at Delphi to hear of their fortunes. The approach relies on a collection of opinions from a panel of experts in a domain of real-world knowledge, and aggregates those decisions to reach consensus around a topic. It is different from traditional surveys in that it is an attempt to identify what could, or should be, as opposed to what is (Miller, 2006).

Delphi studies are generally used to (Delbecq, Van de Ven & Gustafson, 1975, pg. 11):

  • Determine or develop a range of possible program alternatives
  • Explore or expose underlying assumptions or information leading to different judgments
  • Seek out information which may generate a consensus on the part of the respondent group
  • Correlate informed judgments on a topic spanning a wide range of disciplines
  • Educate the respondent group as to the diverse and interrelated aspects of the topic

Some of the other key features in Delphi survey research is that the participants are unknown to each other and that the process is iterative, with each subsequent round being derived from the results of the previous one. In other words, each participant receives a summary of the range of opinions from the previous round, and is given an opportunity to reassess their own opinions based on the feedback of other panelists. This controlled feedback helps to reduce the effect of noise, defined as communication which distorts the data as it relates to individual interests and bias, rather than problem solving. The feedback occurs in the form of a summary of the prior iteration, distributed to the panel as an opportunity to generate additional insights and clarify what was captured in the previous iteration (Dalkey, 1972). In addition, participants need not be geographically collocated (i.e. can be physically dispersed). This provides some level of anonymity, which also serves to reduce the effect of dominant individuals and group pressure to conform.

Within the context of clinical education, Delphi studies have been used to develop assessment practices that are not always easy to define. The modifiable behaviours and clinical competence that clinical educators are interested in are not particularly the concepts and skills covered in the classroom, but rather their application in practice. Assessment of the knowledge and skills required for competent practice usually takes the form of a sampling of a small subset of the total possible range of items, since it isn’t feasible to assess all possible combinations. In addition, not all clinicians agree on what the most important components of practice and assessment are. The Delphi method is therefore an appropriate methodological approach that can be used to gain consensus around the critical issue of what to assess, how it should be assessed and what strategies can be used to improve practice. Delphi studies have been used in healthcare for the planning of services, analysis of professional characteristics and competencies, assessment tool design and curriculum (Cross, 1999; Powell, 2003; Joseph, Hendricks & Frantz, 2011).

Designing a Delphi study
The most important aspect of your Delphi study will be participant selection, as this will directly influence the quality of the results you obtain (Judd, 1972; Taylor & Judd, 1989; Jacobs, 1996). Participants who are selected to participate in a Delphi survey are usually experts in the field, and should provide valuable input to improve the understanding of problems, opportunities and solutions. Having said that, there is no standard description of who should be included in the panel, nor what an “expert” is (Kaplan, 1971). Although there are no set criteria that one can use to select the panel, eligible participants should come from related backgrounds and experiences within the domain, are capable of making helpful contributions, and be open to adapting their opinion for the purpose of achieving consensus. It is not enough for participants to simply be knowledgeable in the domain being explored (Pill, 1971; Oh, 1974). While it is recommended that general Delphi studies use a heterogeneous panel (Delbecq, et al., 1975), Jones and Hunter (1995) suggest that domain specialists be used in clinical studies. Owing to the factors highlighted above, it is essential to establish the credibility of the panel, in order to support the claim that they are indeed experts in the field.

The next aspect to consider is the panel size. This is often dependent on the scope of the problem and the number of knowledgeable informants / experts who are available to you, and there is no agreement in the literature on what size is optimal (Hsu & Sandford, 2007). Depending on the context, it may be that the more participants there are, the higher the degree of reliability of the aspects mentioned. However, it has been suggested that 10 to 15 participants could be sufficient if their background is homogeneous (Delbecq, Van de Ven & Gustafson, 1975).

The first round of questionnaires usually consists of open-ended questions that are used to gather specific information about an area of domain of knowledge, and serves as a cornerstone for subsequent rounds (Custer, Scarcella, & Stewart, 1999). It is acceptable for this questionnaire to be derived from the literature (Hsu & Sandford, 2007) and need not be tested for validity or reliability. The structuring of the questionnaires, types of questions and number of participants will determine the data analysis techniques that are used to reach consensus. While the process could theoretically continue indefinitely, there is some agreement that three rounds of surveys are usually sufficient to reach a conclusion.

Procedure
Typically, the results of the first round are often used to identify major themes emerging from the open-ended questions. Thereafter the responses are collated into questionnaires that will form the basis of the subsequent rounds. From the second round onwards the data is usually analysed quantitatively, using either a rank order or rating technique (this is usually dependent on larger sample sizes). The results are analysed in order to determine levels of agreement in the ranking order. Researchers caution that this level of agreement should be decided on before the commencement of the data collection and devise a plan of how the data will be analysed in order to have a clear cut-off point for inclusion and exclusion. The level of agreement is usually set at 75%, although this can be modified if agreement is not reached. In some cases, participants may also be asked to provide a rationale for their ranking decisions, especially when panelists provide opinions that lie outside the groups’ consensus for a domain or topic.

Procedure of running a Delphi study

  1. Determine your objectives. What is it that you want your panelists to achieve consensus on?
  2. Design your first set of questions using an extensive review of the available literature. Be sure to base this first round of questions on the objectives you wish to achieve.
  3. Test your questions for ambiguity, time, and appropriateness of responses. Send it out to a small sample of experts or at least colleagues and review their responses to ensure that your questions are useful in terms of achieving your objective.
  4. Send out the first round of the survey.
  5. Send a reminder for panelists to complete the first round, about 1-2 weeks after the initial survey was sent, although the actual time frames will depend on your study.
  6. Analyse the responses from round one, and use these results to design the survey for the second round.
  7. Test round two on a small sample of panelists, in order to make sure that the responses will provide the data you need.
  8. Send out the survey for the second round.
  9. Send a reminder for round two. Again the exact time will depend on your particular needs, and the context of your study.
  10. Analyse the responses from round two and use these results to design the survey for round three.
  11. Test the survey for the third round, and send it out when you are satisfied. Remind panelists to complete if necessary.
  12. Analyse the responses from the third round.
  13. Determine if your objectives have been achieved. Include additional rounds if you decide that you need more information.

Analysis of results
Quantitative analysis
The aspects to consider for the use of quantitative analysis are related to panel size and questionnaire design. Consequently this is often dependent on the scope of the problem and the number of knowledgeable informants/experts available to you. Some researchers believe that the more participants there are, the higher the degree of reliability of the aspects mentioned. The most widely used technique for gaining consensus in this paradigm is through obtaining an agreement level. Although controversy exists on the level or cut off point for agreement, numerous authors indicated a 75% of agreement as an appropriate level. Apart from obtaining a level of agreement other rating techniques are also commonly used to reach consensus. Some of these rating techniques include the of ranking elements in order of importance and calculating the mean to identify the most important to the least important elements. Also, likert -type scales are used to determine whether element should be included or not. Thus, the the nature of the analysis will depend strongly on the structuring of the questionnaires, types of questions and number of participants.

Qualitative analysis
A qualitative Delphi study does not rely on statistical measures to establish consensus among participants. Rather, it is the analysis of emergent themes (provided no structure was initially provided) that gives rise to the conclusion. The results from open-ended questions will usually be in the form of short narratives, which should be analysed using qualitative techniques. The researcher will review the responses and categorise them into emerging themes. This process will continue until saturation is reached i.e. until no new information or themes arise. These themes can then either be used to form the basis of the next round of questions (as in an exploratory or development Delphi study), or they can be used to derive a list of items that panelists can rank.

Advantages and disadvantages of using the Delphi method
Whereas in committees and face-to-face meetings, dominant individuals may monopolise the direction of the conversation, the Delphi method prevents this by placing all responses on an “equal” footing. Anonymity also means that participants should only take into account the information before them, rather than the reputation of any particular speaker. Anonymity also allows for the expression of personal opinions, open critique, and admission of errors by giving opportunities to revise earlier judgments. In addition, the researcher is able to filter, summarise and discard irrelevant information, which may be distracting for participants in face-to-face meetings. Thus, potentially distracting group dynamics are removed from the equation (Hsu & Sandford, 2007).

One of the major disadvantages is that there is a high risk of both low response rate and attrition. In addition, a Delphi study typically takes up a lot of time, and adds significantly to the workload of the researcher. However, it is felt that the advantages of using a Delphi study in the right context adds value that is difficult to achieve with other methods.

Conclusion
The Delphi method is a useful means of establishing consensus around topics that have no set outcomes and which are open to debate. The credibility of the panel you select for your study is vital if you want to ensure the results are taken seriously.

References

  • Butterworth T. & Bishop V. (1995) Identifying the characteristics of optimum practice: findings from a survey of practice experts in nursing, midwifery and health visiting. Journal of Advanced Nursing 22, 24–32
  • Cross, V. (1999). The Same But Different: A Delphi study of clinicians’ and academics’ perceptions of physiotherapy undergraduates. Physiotherapy, 85(1), 28-39
  • Custer, R. L., Scarcella, J. A., & Stewart, B. R. (1999). The modified Delphi technique: A rotational modification. Journal of Vocational and Technical Education, 15 (2), 1-10
  • Dalkey, N. C. & Helmer, O. (1963). An experimental application of the Delphi Method to the use of experts. Management Science, 9(3), 458 – 468
  • Delbecq, A.L., Van de Ven, A.H. & Gustafson, D.H. (1975). Group Techniques for Program Planning: a guide to nominal group and Delphi processes
  • Hsu, C.-chien, & Sandford, B. (2007). The Delphi Technique: Making sense of consensus. Practical Assessment, Research and Evaluation, 12(10)
  • Jacobs, J. M. (1996). Essential assessment criteria for physical education teacher education programs: A Delphi study. Unpublished doctoral dissertation, West Virginia University, Morgantown
  • Jones J. & Hunter, D. (1995). Qualitative research: Consensus methods for medical and health services research. British Medical Journal, 311, 376–380
  • Joseph, C., Hendricks, C., & Frantz, J. (2011). Exploring the Key Performance Areas and Assessment Criteria for the Evaluation of Students’ Clinical Performance: A Delphi study. South African Journal of Physiotherapy, 67(2), 1-7
  • Judd, R. C. (1972). Use of Delphi methods in higher education. Technological Forecasting and Social Change, 4 (2), 173-186
  • Kaplan, L. M. (1971). The use of the Delphi method in organizational communication: A case study. Unpublished master’s thesis, The Ohio State University, Columbus
  • Miller, L. E. (2006, October). Determining what could/should be: The Delphi technique and its application. Paper presented at the meeting of the 2006 annual meeting of the Mid-Western Educational Research Association, Columbus, Ohio
  • Murphy M.K., Black N., Lamping D.L., McKee C.M., Sanderson C.F.B., Askham J. et al. (1998) Consensus development methods and their use in clinical guideline development. Health Technology Assessment 2(3)
  • Oh, K. H. (1974). Forecasting through hierarchical Delphi. Unpublished doctoral dissertation, The Ohio State University, Columbus
  • Pill, J. (1971). The Delphi method: Substance, context, a critique and an annotated bibliography. Socio-Economic Planning Science, 5, 57-71
  • Powell, C. (2003). The Delphi technique: myths and realities. Journal of advanced nursing, 41(4), 376-82
  • Skulmoski, G. J., & Hartman, F. T. (2007). The Delphi Method for Graduate Research. Journal of Information Technology Education, 6