Comment: Developing medical artificial intelligence leaders

This is an open-access Really Good Stuff article from the journal Medical Education. I was excited to read it as I thought it might provide some useful ideas for some of my teaching, but I was disappointed in the evaluation of the project.

Machine learning applications are increasingly used in medicine. The demand for ‘augmented doctors,’1 or ‘enhanced physicians’, that is, physicians capitalising rather than opposing incoming technology, has been recognised. However, teaching medical Artificial Intelligence is challenging, especially in already oversaturated medical curricula with most medical schools lacking Artificial Intelligence expertise.

Lee, Y., Cao, K., Leech, M., & De Ponti, F. (2021). Developing medical artificial intelligence leaders: International university consortium approach. Medical Education.

The article briefly describes a virtual summer school programme run from the University of Bologna in July 2020, with the theme of AI in medicine and medical education.

Students attended lectures and interactive sessions, and then worked in groups on a project to develop an AI tool and literature review that identified a real-world need.

I thought that the general thrust of the programme seems like a potentially useful format for generating new ideas and AI-based clinical and health professions education projects, and also for providing a baseline foundation in basic AI concepts for health professionals and students.

But how the authors chose to evaluate the programme was disappointing. The use of a pre- and post-intervention survey design baffles me. Who cares about student satisfaction re. self-assessed competency and quality of lectures, etc.? This tells us nothing about whether or not the programme was effective in achieving it’s outcomes which, I assume must have had something to do with developing leaders in medical AI.

If you really want to know how useful the programme was, you might get an external panel to evaluate the quality of the students’ projects. Some questions they could consider in the evaluation might include things like:

1. Does the project identify a real-world clinical or medical education problem that has a high impact on patient outcomes or physician efficacy, for example?
2. Is the problem currently neglected? Or, are current attempts to address the problem very expensive? Too time-consuming? Would this proposed solution really add value?
3. Is the project likely to make some significant progress in moving that problem forward?
4. Is the solution elegant i.e. what variables are being included in the model? How technically simple is the model? I’d like to see whether or not the use of an AI-tool is just adding complexity to the problem.
5. Are there any blind spots in the students’ perspectives, especially re. patient data, bias, etc.?
6. Proposed projects should also include implementation plans because, no matter how technically feasible they are, they’re useless without uptake. Since the programme is supposed to be about developing leaders in medical AI, I’d want to know whether or not this group is at least aware of some of the challenges of implementation in large organisations like hospitals and health trusts.

I understand that I may be coming across as the peer reviewer asking the authors to do the study that I thought they should have done. But I do think that surveys of students’ self-reported satisfaction with changes in competency are less than useless in most scenarios, and would like to see changes in how we evaluate the outcomes of projects like this. This ‘study’ – and countless – others, feels more like a write-up of the evaluation forms we ask students to complete at the end of a module.

If you’re building a programme that aims to develop leaders in medical AI, you have to at least try and assess variables associated with the characteristics you’d expect to find in those leaders.