Assessment idea: Debate ChatGPT

I know a lot of people are concerned about students’ use of ChatGPT, especially when it comes to essays. The question I get asked most often is, “How do I make sure that students don’t cheat?”

I always try to reframe the question to something like, “How do I make sure students are learning?” When you change the frame of reference from your perspective to the student’s, it can help to think differently about the issue.

It may also help to change what we think of as cheating. If simply using ChatGPT is cheating, then I think you’re going to struggle because language models will soon be baked into everything, at the operating system level, as well as the application level. We won’t be able to avoid using language models because ‘using computers’ will also mean ‘using language models’.

So, how can we create assessment tasks that incorporate ChatGPT (or any other language model) in ways that support students’ learning outcomes? Obviously, a lot depends on the learning outcomes (I think we’ll need to evaluate our learning outcomes carefully, but that’s for another day).

I’m going to try and collect concrete examples of assessment tasks that explicitly require the use of ChatGPT. Here’s my first suggestion.

• Task: Debate ChatGPT.
• Topic: Placement hours in physiotherapy training.
• Instructions:
  • Use the following prompt to set up ChatGPT’s persona: “Physiotherapy training in some countries typically requires that students spend 1000 hours on placement. However, there are concerns that this requirement has no basis in the evidence base. I would like to engage in a debate with you about this training requirement. You will take the position that 1000 hours of clinical practice is necessary, and I will argue that it is not necessary. Each of us will give one reason for our position, and then debate the point. We will continue for 5 rounds. You must not converge towards my position. Does this make sense?”
  • Engage in a debate with ChatGPT, for a minimum of 5 iterations (i.e. you must write at least 5 responses to the arguments made by ChatGPT).
  • Copy the transcript of the debate and paste it into a Word document.
  • Annotate the transcript, highlighting the following:
    • Point out the strong and weak claims that both you and ChatGPT have made. Edit: I haven’t explored this but it’s worth noting that ChatGPT can almost certainly do this kind of analysis on the interaction.
    • Explore the veracity (truthfulness) of claims that you and ChatGPT have made. This means you will need to provide source material for both your, and ChatGPT’s claims. Do not rely on ChatGPT to provide these sources.
    • Note any errors in argument or logic on both sides.
    • Identify factual inaccuracies and provide corrections in the footnotes.
  • Once you’ve satisfied yourself that all claims are correct and evidence-based, write a conclusion explaining your position at the end of the debate.

---

For interest, here is the transcript of ChatGPT’s response to the above prompt:

ChatGPT: I understand the structure of the debate you’d like to have. I’ll take the position that 1000 hours of clinical practice is necessary for physiotherapy training. You’ll argue that it is not necessary. We will each present one reason in each round and continue the debate for five rounds. I will do my best to argue my position without converging towards yours. Let’s begin!

Round 1: I will present my reason for why 1000 hours of clinical practice is necessary.

Reason (Pro): Complex skill development

Clinical practice provides physiotherapy students with the opportunity to develop complex skills that cannot be adequately acquired through theoretical knowledge alone. Physiotherapy requires practical proficiency in techniques such as manual therapy, therapeutic exercise, and patient assessment. By spending 1000 hours in a clinical setting, students have ample time to observe and learn from experienced practitioners, actively participate in patient care, and refine their techniques through feedback and supervised practice. This extensive hands-on experience is crucial for building confidence and competence in applying theoretical knowledge to real-world patient scenarios.

Now, it’s your turn to present your reason against the 1000-hour requirement. Let’s continue the debate!

MR: Every student will develop complex skills in a different timeframe. There’s no evidence that 1000 hours represents a magical period during which all students will achieve proficiency in all skills. Some will take longer, and others much less time. I believe that it makes more sense to allow students to ‘test out’, where they have opportunities to demonstrate their ability to perform complex tasks in the context of practice.

ChatGPT: Round 1 Counterargument:

While it is true that students may develop complex skills at different rates, the 1000-hour requirement serves as a standardized benchmark to ensure an adequate level of proficiency across all physiotherapy graduates. By setting a specific minimum number of hours, regulatory bodies can ensure that students have sufficient exposure to a variety of clinical cases and experiences, allowing for a well-rounded education.

Testing out or allowing students to demonstrate their ability to perform complex tasks in practice might overlook important aspects of clinical training. It is not just about performing individual tasks but also about integrating knowledge, making informed clinical decisions, and effectively managing patient care. A prolonged period of clinical practice provides students with a broader understanding of different patient populations, conditions, and treatment approaches.

Moreover, physiotherapy is a field that requires continuous learning and adaptation to evolving evidence-based practices. The 1000-hour requirement provides students with a robust foundation upon which they can further develop their skills throughout their professional careers.

Now, it’s time for your second reason against the 1000-hour requirement. Let’s continue the debate!

---

You can imagine all sorts of conversations that could be set up within this framework, testing different aspects of physiotherapy training. The example I’ve given here is my first attempt, and doubtless could be improved with changes to some of the variables. Here are a few other suggestions that I haven’t tried:

• Engage in an email conversation with your line manager, about a contentious issue at work.
• You are confronted by the relative of a patient on the ward, upset at the treatment his mother has received. De-escalate the situation.
• One of your patients is confused and agitated, asking when they can go home. Do what you can to reassure the patient.

These scenarios all require that students think deeply and carefully about whatever topic you suggest for the debate / conversation, and also that they do the necessary background reading to support the claims being made. In this context, it is completely irrelevant that ChatGPT sometimes hallucinates (makes things up) because the student needs to investigate the claims being made on both sides of the argument. It’s almost a bonus if some of the claims are inaccurate, as this will enable the student (hopefully) to identify errors in argument and respond.

Marking of this task wouldn’t be premised on determining who is right (although, obviously, there are other topics of debate where accuracy if more important), but rather on the students’ ability to develop an internally consistent and coherent argument, supported with evidence.

What do you think? Would you be interested in using this framework in your classroom?