Paper Review: the Babylon Chatbot

…it is fantastic that Babylon has undertaken this evaluation, and has sought to present it in public via this conference paper. They are to be applauded for that. One of the benefits of going public is that we can now provide feedback on the study’s strength and weaknesses.

Source: Coiera, E. (2018). Paper Review: the Babylon Chatbot.

There’s been a lot of coverage of Babylon Health recently, with the associated controversy around what this might mean for GPs and patients. However, what might be even more interesting than the claim that a chatbot could replace a GP, is the fact that Babylon is one of the few companies that have published some of their work openly. This is quite unusual in an industry where startups are reluctant to share their methods for fear of exposing their “secret sauce”. But, as the open review by Enrico Coiera demonstrates, publication of methods for peer review and scientific scrutiny is an essential aspect of moving the field of clinical AI forward.

Mixed methods research: John Cresswell seminar


For me, mixed methods research (MMR) is about using qualitative and quantitative data to strengthen an argument that is difficult to support with only one type of data. It’s about bringing together the numbers (quantitative) and stories (qualitative) to gain a more complete understanding of the world (research problems and questions). We often think of those two approaches as being separate and distinct, but when combined they produce something greater than the sum of the two parts. Earlier this year we had the opportunity to attend a seminar by John Cresswell & Tim Guetterman. Here are my notes.

Introduction to Mixed Methods Research

Practical uses of mixed methods research:

  • Explaining survey results
  • Exploring the use of new instruments in new situations
  • Confirming quantitative results with qualitative findings (why is it often the quantitative component that comes first; any situations where the quantitative could be used to explain the qualitative results?)
  • Adding qualitative data into experiments
  • Understanding community health research
  • Evaluating programme implementation

What are the major elements of MMR?

  • A methodology (popular way of conducting research)
  • Collecting and analysing quantitative and qualitative data
  • Integrating different sets of data
  • Framing the study within a set of procedures (called mixed methods designs)
  • Being conscious of a philosophical stance and theoretical orientation

Quantitative data collection (closed ended) makes use of instruments, checklists and records. Quantitative data analysis uses numeric data for description, comparison, relating variables

Qualitative data collection (open ended) uses interviews, observations, documents and audio-visual materials. Data analysis revolves around using text and image data for coding, theme development, and then relating themes.

What does “integration” mean? We can do this by merging (using one set of data with another), connecting (using one set of data to explain or build on another), or embedding (quan within qual, or qual within quan) the data.

What is MMR not?

  • Reporting quan and qual data separately (they should be combined)
  • Using informal methods (it is systematic)
  • Simply using the name (it must be rigorous)
  • Collecting either multiple sets of quan or qual data (i.e. not multimethod research, must collect both quan or qual sets of data)
  • Collecting qual data and then quantitatively analysing it (instead of content analysis, collecting both forms of data)
  • Simply considering it an evaluation approach (it is a complete methodology)

Specific benefits of MMR:

  • Quan to qual: make quan results more understandable
  • Qual to quan: understand broader applicability of small-sample qual findings
  • Concurrent: robust description and interpretation of multiple sets of different data

Popular mixed method designs:

  • Basic: convergent (bringing qual and quan data together), explanatory sequential (use one set of data to explain more clearly another set of data), exploratory (use qual data to develop something quantitatively that leads to an intervention design)
  • Advanced: intervention, social justice, multistage evaluation

Research questions related to MMR

  • Convergent design: To what extent to the quan and qual results converge?
  • Explanatory design: In what ways to the qual data help to explain the quan results?
  • Exploratory design: In what ways do the quan results generalise the qual findings?

How do we display quan and qual results together (joint display)? Lots of variation in how both sets of data can be presented. MaxQDA is an application that can be used to analyse and display different sets of data.

How do we publish MMR? Consider publishing the different sets of data in different papers.

How do we link writing structure to design? Writing about and publishing mixed methods research may require different approaches to article structure and style of writing.

The importance of qualitative research in mixed methods

Key features of qual research:

  • Follows the scientific method
  • Listening to participant views
  • Asking open ended questions
  • Build understanding based on participant views
  • Developing a complex understanding of the problem
  • Go to the setting to gather data
  • Be ethical
  • Analyse the data inductively – let the findings emerge
  • Write in a user-friendly way
  • Include rich quotes
  • Researcher presence in the study (reflexivity)

Types of problems that qual research is suited to:

  • A need to explore a context
  • When it is important to listen
  • Unusual / different culture
  • Don’t know the questions to ask
  • Understanding a process
  • Need to tell a story

How do our backgrounds inform the way we interpret the world? There is an element of reflexivity and an understanding that data interpretation is dependent on our individual personal and professional contexts.

Writing a good qual purpose statement:

  • Single sentence, often in the form of “The purpose of this study…”
  • A focus on one central phenomenon
  • “Qualitative words” e.g. explore, describe, understand, develop
  • Includes participants and setting

Understanding a central phenomenon:

  • Quan: explaining or predicting variables
  • Qual: understanding a central phenomenon

Data collection:

  • Sampling (purposeful)
  • Site selection (gatekeepers, permissions)
  • Recruitment (incentives)
  • Types of data (observation, interview, public/private documents, audio-visual)

Interview procedures:

  • Create a protocol
  • 5-7 open ended questions (first question is easy to answer e.g. participant role or experience; last question could be “Who else should I speak to in order to get more information about this?”)
  • Allows the participant to create options for responding
  • Participants can voice their experiences and perspectives
  • Record and transcribe for analysis


  • Observation protocol
  • Descriptive notes (portrait of informant, setting, event) and reflective notes (personal reflections, insight, ideas, confusion, hunches, initial interpretation)
  • Decide on observational stance (e.g. outsider, participant, changing roles)
  • Enter site slowly
  • Conduct multiple observations
  • Summarise at the end of each observation

Types of audio-visual material:

  • Physical trace evidence
  • Videotape or film a social situation, individual or group
  • Examine website pages
  • Collect sounds
  • Collect email or social network messages
  • Examine favorite possessions or ritual objects

How to code data:

  • Read through the data (many pages of text)
  • Divide text into segments (many segments of text)
  • Label segments of information with codes (30-40 codes)
  • Reduce overlap and redundancy (reduce codes to 20)
  • Collapse codes into themes (reduce codes to 5-7 themes)

A good qual researcher can identify fine detail but also step back and see the larger themes

How to write a theme passage:

  • Use themes as headings
  • Use codes to build evidence for themes
  • Use quotes and sources of information to demonstrate themes

Writing up the qual study:

  • Description
  • 5-7 themes
  • Use codes and quotes to support themes
  • Tell a good story

Five approaches to qual research:

  • Narrative (comes out of literature)
  • Phenomenology (psychology)
  • Grounded theory (sociology)
  • Ethnography (anthropology)
  • Case study
  • Can also include discourse analysis, participatory approaches

Ethical issues:

  • Respect the site, develop trust, anticipate the extent of the disruption
  • Avoid deceiving participants, discuss purpose
  • Respect potential power imbalances
  • Consider incentive for participants

MAXApp is a mobile app for collecting data on Android and iOS devices:

  • Take photos
  • Write memos
  • Audio recording
  • Location data (geotagging)
  • How is this different to something like Evernote? If you’re already using MAXQDA, it offers integration with the desktop client. If you use another data analysis package, then MAXApp may not be as useful.

Relevant readings

Twitter Weekly Updates for 2010-03-29

Research development workshop: research methods

This section included a general discussion on methods, then provided a brief overview of the 2 main types.

Make sure to choose a design that’s appropriate for your project

Research tends to fall broadly into one approach or the other, and is often not entirely quantitative or qualitative

Make sure to avoid using the language of one approach when you are using the other. Example: talking about “proving” something when using a qualitative approach isn’t appropriate

Continuum from QuantitativeQualitative

  • Predetermined ↔ Unfolding
  • Tight design ↔ Emerging
  • Architectural / blueprint ↔ Open ended

Planning your research design

  • Often begins with an identified gap in the literature → initial research question
  • What do you want to find out?
  • What is the purpose of your study?
  • What are the questions you want to address?
  • What methods can you use to answer these questions (data collection / analysis)?

Identify the problem and then choose a method, rather than deciding from the outset what type of research you want to do

Overview of quantitative research: what types of questions can quantitative research answer?

  • Give an overview of information with regard to a population
  • Measuring the extent of something using numerical values
  • Identify trends over time
  • Measure attitudes / opinions of large groups e.g. political surveys
  • There is a tightly designed structure that comes before implementing the research to ensure that one is measuring what one intends to measure
  • There are clear variables
  • You would define concepts
  • Formulate measures or indicators for assessing outcomes

Often makes a claim of a causal relationship between 2 variables that requires:

  • Control for interfering variables
  • Sample and control groups
  • Period of time in which to run the intervention
  • Pre- and post-test
  • What statistical tests can you run to analyse data
  • What results would be significant
  • What can one claim based on the sample size
  • Can your results be generalised to a larger population?

Overview of qualitative research: what types of questions can qualitative research answer?

  • Aim to gain more understanding of people, processes, organisations and relationships
  • Naturalistic approach, research something within a natural context in it’s full complexity i.e. not trying to control for interfering variables
  • Aims for depth, rather than breadth → limits how many cases one can study i.e. not a large population
  • Sceptical about the concept of objectivity i.e. acknowledges that the researcher comes to the project with a background and their own values and doesn’t try to completely eliminate bias
  • Don’t claim that findings are generalisable
  • Tends to have a more flexible design, open-ended and iterative process
  • Data analysis → codes for analysis and themes are derived from the data
  • May apply initial theory to data analysis (deductive research)
  • May use grounded theory → themes emerge from the data and influences the conceptual framework (inductive research)
  • Often there is a combination of deductive and inductive research