Research project exploring clinicians’ perspectives of the introduction of ML into clinical practice

I recently received ethics clearance to begin an explorative study looking at how physiotherapists think about the introduction of machine learning into clinical practice. The study will use an international survey and a series of interviews to gather data on clinicians’ perspectives on questions like the following:

  • What aspects of clinical practice are vulnerable to automation?
  • How do we think about trust when it comes to AI-based clinical decision support?
  • What is the role of the clinician in guiding the development of AI in clinical practice?

I’m busy finalising the questionnaire and hope to have the survey up and running in a couple of weeks, with more focused interviews following. If these kinds of questions interest you and you’d like to have a say in answering them, keep an eye out for a call to respond.

Here is the study abstract (contact me if you’d like more detailed information):

Background: Artificial intelligence (AI) is a branch of computer science that aims to embed intelligent behaviour into software in order to achieve certain objectives. Increasingly, AI is being integrated into a variety of healthcare and clinical applications and there is significant research and funding being directed at improving the performance of these systems in clinical practice. Clinicians in the near future will find themselves working with information networks on a scale well beyond the capacity of human beings to grasp, thereby necessitating the use of intelligent machines to analyse and interpret the complex interactions of data, patients and clinical decision-making.

Aim: In order to ensure that we successfully integrate machine intelligence with the essential human characteristics of empathic, caring and creative clinical practice, we need to first understand how clinicians perceive the introduction of AI into professional practice.

Methods: This study will make use of an explorative design to gather qualitative data via an online survey and a series of interviews with physiotherapy clinicians from around the world. The survey questionnaire will be self-administered and piloted for validity and ambiguity, and the interview guide informed by the study aim. The population for both survey and interviews will consist of physiotherapy clinicians from around the world. This is an explorative study with a convenient sample, therefore no a priori sample size will be calculated.

Summary: Ten simple rules for structuring papers

Good scientific writing is essential to career development and to the progress of science. A well-structured manuscript allows readers and reviewers to get excited about the subject matter, to understand and verify the paper’s contributions, and to integrate these contributions into a broader context. However, many scientists struggle with producing high-quality manuscripts and are typically untrained in paper writing. Focusing on how readers consume information, we present a set of ten simple rules to help you communicate the main idea of your paper. These rules are designed to make your paper more influential and the process of writing more efficient and pleasurable.

Mensh, B. & Kording, K. (2017). Ten simple rules for structuring papers. PLoS Computational Biology, 13(9): e1005619.

Thank you to Guillaume Christe for pointing to this paper on Twitter. While I’m not convinced that the title should refer to “rules” I thought it was a useful guide to thinking about article structure. I’m also aware that most people won’t have time to read the whole thing so I’m posting the summary notes I made while reading it. Having said that, I think whole paper (link here) is definitely worth reading. And, if you like this you may also like this table of suggestions from Josh Bernoff’s Writing without bullshit. OK, on with the summary.

First, there’s this helpful table from the authors as a very brief overview.

https://journals.plos.org/ploscompbiol/article/figure?id=10.1371/journal.pcbi.1005619.t001

Principles (Rules 1–4)

Rule 1: Focus your paper on a central contribution, which you communicate in the title. Adding more ideas may be necessary but they make it harder for the reader to remember what the paper is about. If the title doesn’t make a reader want to read the paper, all the work is for nothing. A focused title can also help the author to stay on track.

Rule 2: Write for flesh-and-blood human beings who do not know your work. You are the least qualified person to judge your writing from the perspective of the reader. Design the paper for someone who must first be made to care about your topic, and then who wants to understand your answer with minimal effort. This is not about showing how clever you are.

Rule 3: Stick to the context-content-conclusion (C-C-C) scheme. Aim to write “popular” (i.e. memorable and re-tellable) stories that have a clear beginning, middle and end. While there are many ways to tell stories, each of which engages different readers, this structure is likely to be appropriate for most. Also, the structure of the paper need not be chronological.

Rule 4: Optimize your logical flow by avoiding zig-zag and using parallelism. Only the central idea of a paper should be presented in multiple places. Group similar ideas together to avoid moving the reader’s attention around.

The components of a paper (Rules 5–8)

Rule 5: Tell a complete story in the abstract. Considering that the abstract may be (is probably) the only part of the paper that is read, it should tell the whole story. Ensure that the reader has enough context (i.e. background/introduction) to interpret the results). Avoid writing the abstract as an afterthought, as it often requires many iterations to do it’s job well.

Rule 6: Communicate why the paper matters in the introduction. The purpose of the introduction is to describe the gap that the study aims to fill. It should not include a broad literature review but rather narrow the focus of attention to the problem under consideration.

Rule 7: Deliver the results as a sequence of statements, supported by figures, that connect logically to support the central contribution. While there are different ways of presenting results, often discipline-specific, the main purpose is to convince the reader that the central claim is supported by data and argument. The raw data should be presented alongside the interpretation in order to allow the reader to reach their own conclusions (hopefully, these are aligned with the intent of the paper).

Rule 8: Discuss how the gap was filled, the limitations of the interpretation, and the relevance to the field. The discussion explains how the findings have filled the gap/answered the question that was posed in the introduction. If often includes limitations and suggestions for future research.

Process (Rules 9 and 10)

Rule 9: Allocate time where it matters: Title, abstract, figures, and outlining. Spend time on areas that demonstrate the central theme and logic of the argument. The methods section is often ignored, so budget time accordingly. Outline the argument throughout the paper by writing one informal sentence for each planned paragraph.

Rule 10: Get feedback to reduce, reuse, and recycle the story. Try not to get too attached to the writing, as it may be more efficient to delete whole sections and start again, than to proceed by iterative editing. Try to describe the entire paper in a few sentences, which help to identify the weak areas. Aim to get critical feedback from multiple readers with different backgrounds.


And finally, here’s a great figure to show how each section can be structured using the guidelines in the article.

https://journals.plos.org/ploscompbiol/article/figure?id=10.1371/journal.pcbi.1005619.g001

Read: Academic research

Many people think that “ivory tower” intellectuals make little difference in the world. But some of the highest impact people in history have been researchers, and if you have good personal fit with academic research, we think it can be one of the highest-impact paths…In this article, we’ll cover why we think a career in academia has the potential to be very high impact in the right circumstances, how to figure out whether this option is for you, and how to maximise the impact you can have as an academic.

Whittlestone, J. (2018). Academic research. 80,000 hours.

This article is an incredibly deep dive into the relative benefits and disadvantages of an academic career. Whether you’re in the early stages of thinking about starting on an academic career or are already a full professor, you may find some very valuable context for planning your next move. If you think of your career as something that happens to you, then this article may be a good place to start thinking differently about the rest of your working life.

80 000 hours is an organisation that aims to help as many people as possible lead high impact careers. The idea is that you have about 80 000 working hours in your career, which means that your choice of career is one of the biggest decisions you’ll ever make. Therefore, it’s worth spending a bit of time figuring out how to use that time for good. Their About page has links to many more resources that I’ve found very useful for thinking about my career as an academic.

Translating AI into the clinical setting at UC Irvine – AI Med

Ultimately, many of these shortcomings exist because few if any physicians are actively engaged in developing the next generation of technology, AI or otherwise. It is interesting to note the vast majority of medical startup companies are founded with limited if any physician involvement or oversight.Without experts that deeply understand both the medical and technical aspects of the problem, there is currently a significant gap in translating cutting-edge AI technology to healthcare.

Source: Translating AI into the clinical setting at UC Irvine – AI Med

I’m preparing an article on machine learning for clinicians and one of the recommendations I make is that we must ensure that the 21st century healthcare agenda is not driven by venture capital and software engineers. Even though private corporations and government are probably not malevolent, when surveillance and profit are your core concerns it’s unlikely that you’re going to develop something that truly works in the patients’ best interest. We really do need clinicians to be more involved in guiding the progression of AI integration in the clinical context.

See also: AMA passes first policy guidelines on augmented intelligence.

Note: If you’re interested in this topic, I’ve shared the first draft of my introduction to machine learning for clinicians on ResearchGate and would appreciate any feedback you may have.

Academic expert says Google and Facebook’s AI researchers aren’t doing science

Google and Facebook, and other corporate research labs are focused on AI for profit, not on advancing science..such laboratories aren’t advancing the field of cognitive science anymore than Ford is advancing the field of physics at the edge.

After all, no matter how impressive neural networks are, they operate on principles that date back decades. Perhaps the greatest good for humanity isn’t in fine-tuning algorithms that make people pay attention to Facebook at the expense of their mental health.

Source: Academic expert says Google and Facebook’s AI researchers aren’t doing science

I tend to agree with the main point that the work being done at Google and Facebook doesn’t count as science, in the sense that it’s not advancing our understanding of the world. The engineers at software companies spend a lot of time optimising algorithms until they get the answer they need. I’m not 100% sure but it sounds a bit like p-hacking.

Having said that I also think that there’s a difference between the kind of academic research that enhances our understanding of the world, and the kind of applied research that has commercial value. I’d also love to see more work devoted to cancer diagnosis (and therapeutic intervention development) than social media optimisation, but that’s not really the point. This is about choosing to make an intellectual contribution to a field of research (in which case, do a PhD at a university) or applying well-understood theoretical principles in the service of  real world application (in which, case go work for a startup).

AMA Passes First Policy Recommendations on Augmented Intelligence

Combining AI methods and systems with an irreplaceable human clinician can advance the delivery of care in a way that outperforms what either can do alone. But we must forthrightly address challenges in the design, evaluation and implementation as this technology is increasingly integrated into physicians’ delivery of care to patients.

Source: AMA Passes First Policy Recommendations on Augmented Intelligence

The American Medical Association recently released their policy recommendations on the use of agumented intelligence systems in the clinical context. Briefly, the AMA states that it will:

  1. Help set priorities for health care AI.
  2. Identify opportunities to integrate the perspectives of clinicians into the development of health care AI.
  3. Promote the development of thoughtfully-designed, high quality, clinically validated health care AI.
  4. Encourage the education of all stakeholders into the promise and limitations of health care AI.
  5. Explore the legal implications for health care AI.

To me, this looks like a set of objectives or lines of inquiry for anyone interested in a research programme looking at the use of AI in the context of healthcare and health professions education.

You are your Metadata: Identification and Obfuscation of Social Media Users using Metadata Information

We spend a lot of time focusing on the content of messaging systems as a means of identifying people but it looks like the metadata encoded alongside the content may be just as important when it comes to de-anonymising the data. This wasn’t always a problem because it’s hard to analyse multivariate relationships in large sets of data, especially when we don’t really know what we’re looking for. It turns out that machine learning algorithms are very good at finding patterns that we don’t have to explicitly define, which means we need to think carefully about what is included in the data we share.

This may also have implications for the publication of data sets that researchers are under pressure to include in their final publications. How long before we need to ensure that metadata – as well as names – are scrubbed from the data sets?

We also found that data obfuscation is hard and ineffective for this type of data: even after perturbing 60% of the training data, it is still possible to classify users with an accuracy higher than 95%. These results have strong implications in terms of the design of metadata obfuscation strategies, for example for data set release, not only for Twitter, but, more generally, for most social media platforms.

Source: [1803.10133v1] You are your Metadata: Identification and Obfuscation of Social Media Users using Metadata Information

IPE course project update

This post is cross-posted from the International Ethics Project site.

My 4th year students have recently completed the first writing task in the IEP course pilot project. I thought I’d post a quick update on the process using screenshots to illustrate how the course is being run. We’re using a free version of WordPress which has certain limitations. For example it’s hard to manage different cohorts of students, but there are many more advantages, which I’ll write about in another post.

My students will keep writing for their portfolios using the course website, which I’ll keep updating and refining based on our experiences. The idea is that by the end of the year we’ll have figured out how to use the site most effectively for students to work through the course for the project.

Psychology’s top 20 principles for enhancing teaching and learning

Every once in a while an article is published that you know is Important and that you should take Note of, and in this post I’m going to summarise a paper that I think fits into that category. It’s a recent publication in Mind, Brain and Education that attempts to summarise and explain the Top 20 principles of teaching and learning, as determined by the last few decades of psychological research. The article is called Science Supports Education: The behavioural research base for Psychology’s top 20 principles for enhancing Teaching and Learning, and it’s by Lucariello, Nastasi, Anderman, Dwyer, Ormiston, and Skiba. See the bottom of this post for the abstract and citation information.

After a brief introduction and description of the Methods the article gets stuck into the principles, which I’ll list and describe below. For some reason, Principle 8 – on the development of student creativity – is not included in the paper and no explanation is given for the omission.


Principles 1-8: How do students learn?

1. Students’ beliefs or perceptions about intelligence and ability affect their cognitive functioning and learning: If students believe that intelligence has a fixed value, they are less likely to learn than if they believe that intelligence can be changed. Teachers should communicate to students that “…failure at a task is not due to lack of ability and that performance can be enhanced, particularly with added effort or through the use of different strategies.”

2. What students already know affects their learning: Students prior knowledge influences how they incorporate new ideas because what they already know interacts with the new material being learned. This is an especially important concept when considering students’ misconceptions and how those misconceptions impede new learning. Teachers could create tasks that give students an active role in confronting and then reducing their cognitive dissonance.

3. Students’ cognitive development and learning is not limited by general stages of development: Cognitive growth is uneven and not linked to stages. Therefore, teachers’ ideas around how, and what new material should be presented, are more effective when they can take into consideration the domain-relevant and contextual knowledge of their students.

4. Learning is based on context, so generalizing learning to new contexts is not spontaneous, but rather needs to be facilitated: In order for learning to be effective, it should generalise to new or different contexts and situations. However, student transfer of knowledge and skills is not spontaneous or automatic. Teachers could therefore teach concepts in multiple contexts so that students can recognise contextual similarities, and focus on the application of their knowledge to the real world.

5. Acquiring long-term knowledge and skill is largely dependent on practice: What people know is laid down in long-term memory and information must be processed before it can move from short-term to long-term memory. This processing is accomplished through different strategies, and practice is key. Teachers should consider a variety of frequent assessment tasks given at spaced intervals (distributive practice). In addition, interleaved practice (a schedule of repeated opportunities) to rehearse and transfer skills or content by practicing with tasks that are similar to the target task, or using several methods for the same task, is also recommended.

6. Clear, explanatory, and timely feedback to students is important for learning: Students should receive regular, specific, explanatory, and timely feedback on their work. Feedback is more effective when it includes specific information that is linked to current knowledge and performance to clear learning goals. Teachers should consider providing feedback on assessment tasks – particularly after incorrect responses – in order to improve classroom performance in the future.

7. Students’ self-regulation assists learning and self-regulatory skills can be taught: Self-regulatory skills include setting goals for learning; such as planning, and monitoring progress; and self-reflection, which consists of making judgements about performance and self-efficacy in reaching goals. Self-regulatory skills include the regulation of motivation, which consists of students’ knowledge, monitoring, and active management of their motivation or motivational processing. Teachers can teach these skills directly to learners, by modelling strategies or coaching on their effectiveness. Teachers can also provide opportunities for learners to set goals and manage their attainment and for self-appraisal. A reflective community also can be established by teachers.

8. Missing from this paper

Principles 9-12: What motivates students?

9. Students tend to enjoy learning and perform better when they are more intrinsically than extrinsically motivated: Learners who are intrinsically motivated engage in academic tasks for the pure enjoyment of such engagement, and are more likely to achieve at higher levels and to continue engaging with activities in the future. Intrinsic motivation is linked to effective learning because students persist longer at tasks, experience lower levels of anxiety and develop positive competence beliefs. Learners who are extrinsically motivated engage in tasks in order to receive a reward or avoid a punishment, and are at risk for a number of problematic long term outcomes. Teachers can facilitate intrinsic motivation by de-emphasising high-stakes assessment, by allowing students to engage in projects they are interested in, encouraging students to take academic risks and by ensuring that students have enough time to engage with tasks.

10: Students persist in the face of challenging tasks and process information more deeply when they adopt mastery goals rather than performance goals: When teachers emphasise test scores, ability differences, and competition, students are more likely to adopt performance goals. Moreover, when test scores and grades are presented publicly, students are encouraged to focus on performance goals. In contrast, when teachers emphasise effort, self-improvement, and taking on challenges, students are more likely to adopt mastery goals. At the same time, they are likely to use effective and more complex cognitive strategies, to persist at challenging tasks, to report being intrinsically motivated, and to report feeling efficacious. Mastery goals are therefore more likely to be adopted when grades and test scores are shared privately and not compared across individuals.

11. Teachers’ expectations about their students affect students’ opportunities to learn, their motivation, and their learning outcomes: In classroom settings, teachers’ expectations for students’ successes and failures influence student achievement and motivation. When educators hold high expectations for their students, they often rise to the occasion and achieve at high levels (provided that the necessary support structures are in place). In contrast, when teachers hold low expectations for student success, students may come to believe that they lack skills and abilities, and thus confirm the teachers’ expectations. It is important to understand that teachers may interact differently with students, and provide differential instruction, based on their expectations for each student’s success or failure, regardless of how accurate those expectations are.

12. Setting goals that are short term (proximal), specific, and moderately challenging enhances motivation more than establishing goals that are long term (distal), general, and overly challenging: Goal setting is the process by which an individual sets a standard of performance and is important for motivation because students with a goal and adequate self-efficacy are likely to engage in the activities that lead to achievement of that goal. Three properties of goal setting are important for motivation. First, short-term goals are more motivating than long-term goals because it is easier to assess progress toward short-term goals. Students tend to be less adept at thinking concretely with respect to the distant future. Second, specific goals are preferable to more general goals because it is easier to quantify and monitor specific goals. Third, moderately difficult goals are the most likely to motivate students because they will be perceived as challenging but also attainable.

Principles 13–15: Why are social context, interpersonal relationships, and emotional well-being important to student learning?

13. Learning is situated within multiple social contexts.

14. Interpersonal relationships and interpersonal communication are critical to both the teaching–learning process and the social–emotional development of students.

15. Emotional well-being influences educational performance, learning, and development.

These principles are interrelated and are represented in theory and research relevant to schools as systems that support psychological (social and emotional) well-being as well as cognitive development and academic learning. According to developmental–ecological theory, the child or learner is best viewed as embedded within multiple social contexts or ecosystems (e.g., school, family, neighbourhood, peer group), that influence learning:

  • Microsystem: student-teacher and student-student interactions influence learning
  • Ecosystem: microsystem interactions occur within a school where policies and norms (teaching and learning practices and organisational structure) influence learning
  • Macrosystem: ecosystems interact (e.g. school and families) within a society which reflects culture, values and norms

These interactions within and between systems influence students’ learning significantly, and are documented more extensively in the article (pg. 61-62).

Principles 16–17. How can the classroom best be managed?

16. Expectations for classroom conduct and social interaction are learned and can be taught using proven principles of behaviour and effective classroom instruction.

17. Effective classroom management is based on (1) setting and communicating high expectations, (2) consistently nurturing positive relationships, and (3) providing a high level of student support.

Classroom management is a fundamental, bedrock set of
procedures and skills that establish a climate for instruction and learning. Class and school rules must be positively stated, concrete, observable, posted, explicitly taught, frequently reviewed, and positively reinforced. This allows students to learn the social curriculum in each classroom and enables teachers to develop classroom climates that maximise student engagement and minimises conflict and disruption.

Classrooms that are structured to offer multiple opportunities for students to respond facilitate the development of quality teacher–student relationships, which in turn lead to fewer behavioural problems and increased academic performance. Students who are at risk for classroom disruption may need more attention to relationship-building in order to develop and maintain connections in the classroom.

Culturally responsive classroom management is an approach that aims to actively engage students by offering a curriculum that is relevant to their lives. Teachers demonstrate a willingness to learn about important aspects of their students’ lives and create a physical environment that is reflective of students’ cultural heritage. Culturally responsive teachers understand the ways in which schools reflect and perpetuate discriminatory practices of the larger society and are characterised as “warm demanders”; “strong yet compassionate, authoritative yet loving, firm yet respectful”.

Finally, a high ratio of positive statements / rewards to negative consequences, and nurturing an atmosphere of respect for all students and their heritage, builds trust in the classroom that can prevent behavioural conflict.

Principles 18–20: how to assess student progress?

18. Formative and summative assessments are both useful, but they require different approaches: Formative assessments are carried out during instruction and are aimed at improving learning in the classroom setting. Summative assessments measure learning at a given point in time, usually at the end of some period of instruction where they are used to provide a judgement about student learning. The goal of both formative and summative assessments is to produce valid, fair, useful, and reliable information for decision making. Teachers can also use their understanding of assessment information to decide whether they covered the material that they intended to cover, or to judge how effectively they met the objectives for student learning.

19. Students’ skill and knowledge should be assessed with processes that are grounded in psychological science and that have provided well-defined standards for quality and fairness: Valid and reliable assessments enable teachers to make inferences about what students are learning. To understand the validity of an assessment, there are four question that need to be considered:

  1. How much of what you intended to measure is actually being measured?
  2. How much of what you did not intend to measure actually ended up being measured?
  3. What consequences, either intended or unintended, occurred with the assessment?
  4. Do you have solid evidence to support your answers to the first three questions?

Validity is a judgement, over time and across a variety of situations, about what inferences can be drawn from the test data, and the consequences of using the test. Valid assessment entails specifying what an assessment is supposed to measure. Teachers can improve assessment quality by aligning teaching and testing. However, they should also:

  • Be mindful that valid tests in one context may not be valid for another
  • Ensure that high-stakes decisions be based on multiple measures, not on a single test
  • Examine outcomes for any discrepancies in performance among different cultural groups

20. Good use of assessment data depends on clear, appropriate, and fair interpretation: Effective teaching depends heavily on teachers being informed consumers of educational research, effective interpreters of data for classroom use, and good communicators to students and their families about assessment data and decisions that affect them. The interpretation of assessments involves addressing the following questions:

  1. What was the assessment intended to measure?
  2. On what are comparisons of the assessment data based? Are students being compared to one another? Or, are responses being directly compared to samples of acceptable and unacceptable responses?
  3. Are scores being classified using a standard or cut point, such as letter grades, or another indicator of satisfactory/unsatisfactory performance?How were these standards set?

Awareness of the strengths and limitations of any assessment is critical. Such awareness enables teachers to make others aware of important caveats, such as the imperfect reliability of scores and the importance of using multiple sources of evidence for high-stakes decisions.


And there you have it. Twenty principles (19 without the one on fostering student creativity) on how best to go about enhancing teaching and learning practices in the classroom. While I don’t think it’s feasible to try and incorporate all of these principles in every classroom session, it’s definitely worthwhile having these at the back of your mind when planning assessment tasks, assignments, lectures and activities in class. I also recommend reading the whole paper which provides additional insight and links to further reading that would be useful to dig into.

Abstract

Psychological science has much to contribute to preK-12 education because substantial psychological research exists on the processes of learning, teaching, motivation, classroom management, social interaction, communication, and assessment. This article details the psychological science that led to the identification, by the American Psychological Association’s Coalition for Psychology in Schools and Education, of the “Top 20 Principles from Psychology for PreK-12 Teaching and Learning.” Also noted are the major implications for educational practice that follow from the principles.

Citation: Lucariello, J. M., Nastasi, B. K., Anderman, E. M., Dwyer, C., Ormiston, H., & Skiba, R. (2016). Science Supports Education: The behavioural research base for Psychology’s top 20 principles for enhancing Teaching and Learning. Mind, Brain and Education, 10(1), 55–67.

The CONSORT guidelines for systematic reviews of RCTs

When I was at the WCPT conference last year I came across the CONSORT guidelines for the publication of systematic reviews of RCTS, which I’d never heard of before. I made a note to look it up and finally got around to doing it. I thought would be quite helpful in planning and carrying out these kinds of research projects, so I’m sharing a few notes here.

CONSORT stands for Consolidated Standards of Reporting Trials and is an “evidence-based, minimum set of recommendations for reporting randomized trials”. In addition to the CONSORT statement, there is a checklist that can be used for evaluating the quality of reports of clinical trials. It is, in essence, a description of how to conduct and report on systematic reviews. If you’re interested in conducting systematic reviews of any trials, then this is definitely something to pay attention to.

consort-flow-diagram

Additional resources for the CONSORT guidelines