I recently received ethics clearance to begin an explorative study looking at how physiotherapists think about the introduction of machine learning into clinical practice. The study will use an international survey and a series of interviews to gather data on clinicians’ perspectives on questions like the following:
- What aspects of clinical practice are vulnerable to automation?
- How do we think about trust when it comes to AI-based clinical decision support?
- What is the role of the clinician in guiding the development of AI in clinical practice?
I’m busy finalising the questionnaire and hope to have the survey up and running in a couple of weeks, with more focused interviews following. If these kinds of questions interest you and you’d like to have a say in answering them, keep an eye out for a call to respond.
Here is the study abstract (contact me if you’d like more detailed information):
Background: Artificial intelligence (AI) is a branch of computer science that aims to embed intelligent behaviour into software in order to achieve certain objectives. Increasingly, AI is being integrated into a variety of healthcare and clinical applications and there is significant research and funding being directed at improving the performance of these systems in clinical practice. Clinicians in the near future will find themselves working with information networks on a scale well beyond the capacity of human beings to grasp, thereby necessitating the use of intelligent machines to analyse and interpret the complex interactions of data, patients and clinical decision-making.
Aim: In order to ensure that we successfully integrate machine intelligence with the essential human characteristics of empathic, caring and creative clinical practice, we need to first understand how clinicians perceive the introduction of AI into professional practice.
Methods: This study will make use of an explorative design to gather qualitative data via an online survey and a series of interviews with physiotherapy clinicians from around the world. The survey questionnaire will be self-administered and piloted for validity and ambiguity, and the interview guide informed by the study aim. The population for both survey and interviews will consist of physiotherapy clinicians from around the world. This is an explorative study with a convenient sample, therefore no a priori sample size will be calculated.
Earlier this year we started the International Ethics Project, a collaboration between physiotherapy departments from several countries who intend offering an online course in professional ethics to their undergraduate students. You can read more about the project here.
In June we started the process of developing a questionnaire that we can use to establish some baseline data on students’ levels of digital literacy. It’s taken a bit longer than expected but we’ve finally managed to complete the reliability testing of the questionnaire as part of a pilot study. Before we can begin planning the module and how it will be implemented we need to get a better understanding of how our population – drawn as they are from several countries from around the world – uses digital tools in the context of their learning practices. The results of the reliability study showed that most of the survey items had Kappa values between 0.5 – 0.6 (indicating moderate agreement); 0.7 – 0.8 (indicating strong agreement); or >0.8 (indicating almost perfect agreement). See this post on the project blog for more details on how the reliability testing was conducted.
Now that we have conducted quite a rigorous piloting of the questionnaire, we hope that this questionnaire might be useful for other health professional educators who are considering the use of digital tools in their classrooms. To this end we would like to report the results of this pilot, along with some preliminary results, at the ER-WCPT conference on 11-12 November, 2016 in Liverpool. We will therefore be submitting an abstract for the conference in the coming months.
A few thoughts on conducting test-retest reliability analysis on questionnaires, based on my own recent experiences;
– DO pay attention to your coding sheet before doing the test, it will influence your questionnaire design
– DO make sure you pilot your questionnaire for ambiguity and understanding before doing the test, it may not be essential but it is logical
– DO capture the data yourself, it will give you insight and a deeper appreciation of the process
– DO make sure you have a way to uniquely identify each questionnaire, and simple codes are better than complex ones
– DO make sure you ask participants to uniquely identify each form they complete, but make sure to preserve anonymity
– DO NOT rely on handwriting recognition to achieve the last point if you forgot to do it, it will waste your time and take you into a valley of despair
– DO NOT rush the process, you will make mistakes if you do