Using Claude to rapidly build an evidence-based CPD activity

I wanted to prepare an evidence-based CPD activity for clinicians that aimed to support the development of clinical reasoning. I was passingly familiar with the key feature approach for assessing clinical reasoning and was interested to see if this might be a useful framework to guide the development of the CPD activity.

I went to Google Scholar and searched for key feature approach decision making skills, and downloaded 4 articles with relatively high citation counts.

• Bordage, G., & Page, G. (2018). The key-features approach to assess clinical decisions: Validity evidence to date. Advances in Health Sciences Education: Theory and Practice, 23(5), 1005–1036. https://doi.org/10.1007/s10459-018-9830-5
• Farmer, E. A., & Page, G. (2005). A practical guide to assessing clinical decision-making skills using the key features approach. Medical Education, 39(12), 1188–1194. https://doi.org/10.1111/j.1365-2929.2005.02339.x
• Hrynchak, P., Glover Takahashi, S., & Nayer, M. (2014). Key-feature questions for assessment of clinical reasoning: A literature review. Medical Education, 48(9), 870–883. https://doi.org/10.1111/medu.12509
• Page, G., Bordage, G., & Allen, T. (1995). Developing key-feature problems and examinations to assess clinical decision-making skills. Academic Medicine, 70, 194–201.

Then I uploaded the articles to Claude, and used the prompt below to get it to help me understand the approach.

Prompt: Use the attached articles to give me an overview of the key features approach to developing clinical reasoning. Then I want you to tell me why this approach has merit and why it might be useful for me in my role as a professor in a school of health and social care. Third, give me 5 key takeaways from the articles. And finally, give me a set of principles I can use to incorporate this concept into future assessment tasks.

Claude then gave me a great overview, which I include below for anyone interested in the key feature approach. I followed it up with another prompt (see further below) to build the CPD activity, using the articles and overview it had just provided.

You can use this prompt, lightly edited, for any concept. I think that this represents a simple way to very quickly build the outline of any evidence-based CPD activity. Of course, you can use the same approach to develop an activity for students in the classroom, on almost any topic. And it has the added benefit of creating an outline that could be used to create a handout for students or activity participants.

---

Overview of the key feature approach

The key feature approach to clinical decisions is an approach to clinical decision-making that rewards a focus on the key elements in the case, rather than emphasising ‘thoroughness’.

Oral and written exams tend to overly reward thoroughness i.e. the more ‘good’ things the student does did, the higher their scores. However, we also know that thoroughness is a poor predictor of performance. Therefore, scoring that rewards only the key decisions contributes to more reliable and valid test scores.

In other words, the resolution of a clinical problem is contingent on your ability to successfully manipulate only a few key elements that are case-specific, meaning that assessing general problem-solving skills may not be a good use of time spent in assessment tasks.

Assessing clinical decision-making should therefore focus on the specific challenges in each case i.e. the key features of the case.

We should shift assessment tasks (in oral and written assessments) from general problem-solving to focus only on the most challenging decisions and actions in each case, meaning tests that contain many short, focused cases.

A higher number of cases per unit testing time results in better content representation of the domain assessed i.e. more cases give better outcomes.

The key features approach focuses on assessing the essential steps in solving a clinical problem where examinees are most likely to make errors in the resolution of the problem. Key features problems (KFPs) present a clinical scenario followed by questions that require examinees to make decisions at critical steps in the diagnosis and management of the problem. Only the challenging, essential steps in resolving the problem are tested, not all the steps. The questions aim to assess clinical decision-making skills rather than just factual recall. Various question formats can be used including short-menu (select answer from list of options), write-in, and mixed formats.

5 key takeaways of the key feature approach

1. KFPs focus only on essential challenging decisions, enabling more content sampling.
2. Multiple short cases with a few questions each provides optimal reliability vs fewer long cases.
3. Write-in and short menu formats provide good validity and reliability.
4. Structured test development is required for high quality KFPs: defining key features, developing cases and questions, validating content, piloting.
5. KFPs scores discriminate based on level of training and predict future clinical performance.

In summary, the key features approach provides an efficient, valid and reliable method for assessing clinical decision making. With a structured test development process, KFPs can be a valuable component of an assessment program in medical and health professions education to drive learning of important clinical problem-solving skills. The principles provided can guide thoughtful incorporation of KFPs into an assessment strategy.

Merits of using the key feature approach

1. Efficiency – By focusing only on the critical decision-making steps, more cases can be tested in a fixed exam time compared to longer simulations, enabling broader sampling of content.
2. Validity – KFPs aim to assess clinical decision making in authentic case scenarios, not just factual knowledge. Studies show KFPs can discriminate between examinees based on experience level and can predict future clinical performance.
3. Reliability – With proper test construction, including an adequate number of cases (~40) and questions per case (2-3), reliability coefficients of 0.80 or higher are achievable.
4. Flexibility – KFPs can assess a range of decisions including diagnostic, investigative, management and ethical decisions. They can be administered in written or computerised format.
5. Educational Impact – Focusing assessment on critical decisions can guide examinees to study clinical problem solving, not just medical facts. The cases are authentic representations of practice.

Principles for incorporating key feature problems into assessment

1. Use a systematic test development process: select problems, define key features, write cases and questions, review for content validity, pilot test.
2. Involve content experts in problem development to ensure content validity.
3. Determine number of cases and questions per case based on target reliability.
4. Balance authenticity with a degree of structure in question and response formats.
5. Use KFPs as part of an assessment program that includes other methods, such as knowledge tests, performance assessments, etc.
6. Align KFP content with curricular objectives and intended outcomes.
7. Provide clear instructions and training to examinees on the KFP format.
8. Score cases based on partial credit for key features rather than all-or-none.
9. Leverage KFP scores for formative feedback to guide learning, not just summative decisions.
10. Evaluate quality of KFP exams through item analysis and reliability measures, making improvements as needed.

Designing a CPD activity

Prompt: I want to use the key feature approach to design a CPD activity that aims to support the development of clinical reasoning in Band 5 physiotherapy clinicians in the NHS. I have access to a local network of clinician experts who have experience in the knowledge domain I want the activity to focus on. I want to use their expertise to help me develop a structured prompt that will enable a large language model to take on the personas of a range of patients who all fit within this knowledge domain. I then want to give the CPD participants these prompts, and have them engage in an interview with the AI personas. They must then submit the transcripts of their conversations with the AI. I want to then use the key feature approach as a framework to make a judgement about how well the CPD participants are likely to manage a similar patient in practice. You are a clinician educator with a lot of experience in designing valid and reliable assessment tasks. You also have lots of experience designing continuing professional development activities. Use your experience to help me develop this activity. Think carefully through all the steps that need to take place and explain to me the best way to go about preparing this activity. Let me know the risks and how to mitigate them.

Designing a CPD activity using the key features approach with AI-based patient personas is an innovative idea. Here’s a step-by-step guide to help you develop this activity, along with potential risks and mitigation strategies:

Define the knowledge domain and learning objectives

• Clearly outline the specific knowledge domain and the clinical reasoning skills you want the Band 5 physiotherapists to demonstrate.
• Develop specific, measurable learning objectives that align with the knowledge domain.

Engage with clinician experts

• Collaborate with the local network of clinician experts to identify common and challenging patient scenarios within the knowledge domain.
• Work with the experts to define the key features for each scenario, focusing on critical decision-making points where clinicians are most likely to make errors.

Develop structured prompts for AI personas

• Based on the identified scenarios and key features, create detailed prompts that will guide the large language model to take on the personas of patients.
• Ensure the prompts include relevant patient background, presenting complaints, and responses to potential questions from the clinicians.
• Validate the prompts with the clinician experts to ensure they accurately represent realistic patient scenarios.

Set up the AI-based patient personas

• Input the structured prompts into the large language model to create the AI-based patient personas.
• Test the AI personas to ensure they provide consistent and realistic responses aligned with the intended scenarios.

Develop instructions for CPD participants

• Create clear instructions for the CPD participants, outlining the purpose of the activity, the knowledge domain, and the expectations for their interaction with the AI personas.
• Provide guidance on the format and content of the conversation transcripts they must submit.

Implement the CPD activity

• Distribute the instructions and access to the AI personas to the CPD participants.
• Set a deadline for participants to complete their interactions and submit their conversation transcripts.

Assess the conversation transcripts

• Develop a scoring rubric based on the key features identified earlier, focusing on the critical decision-making points and expected responses from the clinicians.
• Train assessors to use the rubric consistently and reliably.
• Have multiple assessors evaluate each transcript to ensure reliability.
• Provide feedback to the participants based on their performance, highlighting areas of strength and areas for improvement.

Risks and mitigation strategies

AI persona inconsistency

• Risk: The AI personas may provide inconsistent or unrealistic responses, affecting the validity of the activity.
• Mitigation: Thoroughly test the AI personas and refine the prompts as needed to ensure consistency and realism.

Participant engagement

• Risk: Some participants may not fully engage with the AI personas, leading to superficial or incomplete conversation transcripts.
• Mitigation: Provide clear instructions emphasizing the importance of thorough and authentic engagement, and consider incentives for meaningful participation.

Scoring reliability

• Risk: Inconsistent scoring by assessors may affect the reliability of the assessment.
• Mitigation: Develop a clear, detailed scoring rubric and train assessors to use it consistently. Employ multiple assessors for each transcript.

Generalisability

• Risk: Performance in the AI-based scenarios may not fully translate to real-world clinical practice.
• Mitigation: Ensure the scenarios and key features are validated by clinician experts and closely align with real-world challenges. Use this activity as part of a comprehensive CPD program that includes other learning and assessment methods.

By carefully designing the prompts, engaging with clinician experts, and using a structured approach to assessment, you can create a valuable CPD activity that supports the development of clinical reasoning skills among Band 5 physiotherapists. Continuously monitor and evaluate the activity’s effectiveness and make iterative improvements based on participant feedback and assessor insights.