Mixed methods research: John Cresswell seminar


For me, mixed methods research (MMR) is about using qualitative and quantitative data to strengthen an argument that is difficult to support with only one type of data. It’s about bringing together the numbers (quantitative) and stories (qualitative) to gain a more complete understanding of the world (research problems and questions). We often think of those two approaches as being separate and distinct, but when combined they produce something greater than the sum of the two parts. Earlier this year we had the opportunity to attend a seminar by John Cresswell & Tim Guetterman. Here are my notes.

Introduction to Mixed Methods Research

Practical uses of mixed methods research:

  • Explaining survey results
  • Exploring the use of new instruments in new situations
  • Confirming quantitative results with qualitative findings (why is it often the quantitative component that comes first; any situations where the quantitative could be used to explain the qualitative results?)
  • Adding qualitative data into experiments
  • Understanding community health research
  • Evaluating programme implementation

What are the major elements of MMR?

  • A methodology (popular way of conducting research)
  • Collecting and analysing quantitative and qualitative data
  • Integrating different sets of data
  • Framing the study within a set of procedures (called mixed methods designs)
  • Being conscious of a philosophical stance and theoretical orientation

Quantitative data collection (closed ended) makes use of instruments, checklists and records. Quantitative data analysis uses numeric data for description, comparison, relating variables

Qualitative data collection (open ended) uses interviews, observations, documents and audio-visual materials. Data analysis revolves around using text and image data for coding, theme development, and then relating themes.

What does “integration” mean? We can do this by merging (using one set of data with another), connecting (using one set of data to explain or build on another), or embedding (quan within qual, or qual within quan) the data.

What is MMR not?

  • Reporting quan and qual data separately (they should be combined)
  • Using informal methods (it is systematic)
  • Simply using the name (it must be rigorous)
  • Collecting either multiple sets of quan or qual data (i.e. not multimethod research, must collect both quan or qual sets of data)
  • Collecting qual data and then quantitatively analysing it (instead of content analysis, collecting both forms of data)
  • Simply considering it an evaluation approach (it is a complete methodology)

Specific benefits of MMR:

  • Quan to qual: make quan results more understandable
  • Qual to quan: understand broader applicability of small-sample qual findings
  • Concurrent: robust description and interpretation of multiple sets of different data

Popular mixed method designs:

  • Basic: convergent (bringing qual and quan data together), explanatory sequential (use one set of data to explain more clearly another set of data), exploratory (use qual data to develop something quantitatively that leads to an intervention design)
  • Advanced: intervention, social justice, multistage evaluation

Research questions related to MMR

  • Convergent design: To what extent to the quan and qual results converge?
  • Explanatory design: In what ways to the qual data help to explain the quan results?
  • Exploratory design: In what ways do the quan results generalise the qual findings?

How do we display quan and qual results together (joint display)? Lots of variation in how both sets of data can be presented. MaxQDA is an application that can be used to analyse and display different sets of data.

How do we publish MMR? Consider publishing the different sets of data in different papers.

How do we link writing structure to design? Writing about and publishing mixed methods research may require different approaches to article structure and style of writing.

The importance of qualitative research in mixed methods

Key features of qual research:

  • Follows the scientific method
  • Listening to participant views
  • Asking open ended questions
  • Build understanding based on participant views
  • Developing a complex understanding of the problem
  • Go to the setting to gather data
  • Be ethical
  • Analyse the data inductively – let the findings emerge
  • Write in a user-friendly way
  • Include rich quotes
  • Researcher presence in the study (reflexivity)

Types of problems that qual research is suited to:

  • A need to explore a context
  • When it is important to listen
  • Unusual / different culture
  • Don’t know the questions to ask
  • Understanding a process
  • Need to tell a story

How do our backgrounds inform the way we interpret the world? There is an element of reflexivity and an understanding that data interpretation is dependent on our individual personal and professional contexts.

Writing a good qual purpose statement:

  • Single sentence, often in the form of “The purpose of this study…”
  • A focus on one central phenomenon
  • “Qualitative words” e.g. explore, describe, understand, develop
  • Includes participants and setting

Understanding a central phenomenon:

  • Quan: explaining or predicting variables
  • Qual: understanding a central phenomenon

Data collection:

  • Sampling (purposeful)
  • Site selection (gatekeepers, permissions)
  • Recruitment (incentives)
  • Types of data (observation, interview, public/private documents, audio-visual)

Interview procedures:

  • Create a protocol
  • 5-7 open ended questions (first question is easy to answer e.g. participant role or experience; last question could be “Who else should I speak to in order to get more information about this?”)
  • Allows the participant to create options for responding
  • Participants can voice their experiences and perspectives
  • Record and transcribe for analysis


  • Observation protocol
  • Descriptive notes (portrait of informant, setting, event) and reflective notes (personal reflections, insight, ideas, confusion, hunches, initial interpretation)
  • Decide on observational stance (e.g. outsider, participant, changing roles)
  • Enter site slowly
  • Conduct multiple observations
  • Summarise at the end of each observation

Types of audio-visual material:

  • Physical trace evidence
  • Videotape or film a social situation, individual or group
  • Examine website pages
  • Collect sounds
  • Collect email or social network messages
  • Examine favorite possessions or ritual objects

How to code data:

  • Read through the data (many pages of text)
  • Divide text into segments (many segments of text)
  • Label segments of information with codes (30-40 codes)
  • Reduce overlap and redundancy (reduce codes to 20)
  • Collapse codes into themes (reduce codes to 5-7 themes)

A good qual researcher can identify fine detail but also step back and see the larger themes

How to write a theme passage:

  • Use themes as headings
  • Use codes to build evidence for themes
  • Use quotes and sources of information to demonstrate themes

Writing up the qual study:

  • Description
  • 5-7 themes
  • Use codes and quotes to support themes
  • Tell a good story

Five approaches to qual research:

  • Narrative (comes out of literature)
  • Phenomenology (psychology)
  • Grounded theory (sociology)
  • Ethnography (anthropology)
  • Case study
  • Can also include discourse analysis, participatory approaches

Ethical issues:

  • Respect the site, develop trust, anticipate the extent of the disruption
  • Avoid deceiving participants, discuss purpose
  • Respect potential power imbalances
  • Consider incentive for participants

MAXApp is a mobile app for collecting data on Android and iOS devices:

  • Take photos
  • Write memos
  • Audio recording
  • Location data (geotagging)
  • How is this different to something like Evernote? If you’re already using MAXQDA, it offers integration with the desktop client. If you use another data analysis package, then MAXApp may not be as useful.

Relevant readings

SAFRI: managing change and research methods

The third day of SAFRI 2010 has come and gone and I’m exhausted. The sessions are intense and for the first time in years, I found myself counting the minute ’til the coffee break. But even that doesn’t offer any respite because it’s regarded as an opportunity for informal discussion about your project.

We switched back to a presentation / workgroup format today, with the focus being on managing change in the morning, and on research methods in the afternoon. I found the session on change management really interesting. The presentation was interesting but didn’t have much that you couldn’t find easily online. The really interesting bit were the activities we had to work through:

  • Stakeholder analysis – identify the stakeholders in your project, taking into account their level of enthusiasm and influence. Plot the stakeholders on a matrix, creating links between them to highlight how you could create opportunities to have influential and enthusiastic stakeholders encourage those who are resistant to your project
  • “Elevator speech” – prepare a 2 minute speech that you will give to a stakeholder in your project, highlighting key objectives, significance of the project and requirements. Make sure to get their attention and on your side
  • Force analysis – identify forces (factors) that will work for and against the change you propose. And although we didn’t address it today, there’s space for the action one plans to take to either address the forces against, or to take advantage of the forces for your project
  • SOCKS analysis – Strengths, Opportunities, Challenges, Knowledge, Stakeholders. Identify different components within each of these factors, as they relate to your project

What I liked about this session was that we were guided through the various components by using our own projects as the framework. This not only allowed us to make progress on our respective projects, but made the content much clearer than if it had only been presented to us.

As far as this afternoon’s research methods session went, we covered focus groups and survey questionnaires, but in a slightly different format. Eight volunteer, participated in a “real” focus group where the discussion was recorded and documented, then transcribed and printed out. Each group was then guided through an analysis of the transcript to identify key themes. Those themes then formed the basis of the questionnaire. This is the first time I’ve seen a qualitative method leading to a more refined quantitative method, and I think it’d be really useful for my own project.

Tomorrow we’ll be using the survey we created today, to generate data that we can analyse. While I really saw the value in today’s session, I’d have loved to have been able to use my own project for the afternoon session, much like we did in the morning session.

Research development workshop: research methods

This section included a general discussion on methods, then provided a brief overview of the 2 main types.

Make sure to choose a design that’s appropriate for your project

Research tends to fall broadly into one approach or the other, and is often not entirely quantitative or qualitative

Make sure to avoid using the language of one approach when you are using the other. Example: talking about “proving” something when using a qualitative approach isn’t appropriate

Continuum from QuantitativeQualitative

  • Predetermined ↔ Unfolding
  • Tight design ↔ Emerging
  • Architectural / blueprint ↔ Open ended

Planning your research design

  • Often begins with an identified gap in the literature → initial research question
  • What do you want to find out?
  • What is the purpose of your study?
  • What are the questions you want to address?
  • What methods can you use to answer these questions (data collection / analysis)?

Identify the problem and then choose a method, rather than deciding from the outset what type of research you want to do

Overview of quantitative research: what types of questions can quantitative research answer?

  • Give an overview of information with regard to a population
  • Measuring the extent of something using numerical values
  • Identify trends over time
  • Measure attitudes / opinions of large groups e.g. political surveys
  • There is a tightly designed structure that comes before implementing the research to ensure that one is measuring what one intends to measure
  • There are clear variables
  • You would define concepts
  • Formulate measures or indicators for assessing outcomes

Often makes a claim of a causal relationship between 2 variables that requires:

  • Control for interfering variables
  • Sample and control groups
  • Period of time in which to run the intervention
  • Pre- and post-test
  • What statistical tests can you run to analyse data
  • What results would be significant
  • What can one claim based on the sample size
  • Can your results be generalised to a larger population?

Overview of qualitative research: what types of questions can qualitative research answer?

  • Aim to gain more understanding of people, processes, organisations and relationships
  • Naturalistic approach, research something within a natural context in it’s full complexity i.e. not trying to control for interfering variables
  • Aims for depth, rather than breadth → limits how many cases one can study i.e. not a large population
  • Sceptical about the concept of objectivity i.e. acknowledges that the researcher comes to the project with a background and their own values and doesn’t try to completely eliminate bias
  • Don’t claim that findings are generalisable
  • Tends to have a more flexible design, open-ended and iterative process
  • Data analysis → codes for analysis and themes are derived from the data
  • May apply initial theory to data analysis (deductive research)
  • May use grounded theory → themes emerge from the data and influences the conceptual framework (inductive research)
  • Often there is a combination of deductive and inductive research

Research development workshop: writing a proposal

These are some of the notes I made during the presentation on preparing a proposal, but include some points that relate to the general process of conducting research

There are 3 things a proposal should try to address:

  • What? – What contribution will this research make? What is it about? What do you want to study?
  • Why? – Why should be bother? Why is it significant? Why do you want to study this?
  • How? – How are you going to study this? What tools and techniques will you use? Who else will be involved? How long is it going to take?

There is no “one way” to write a proposal, rather conform to the norms of your department. How do you find your “voice”, and what is the voice of your department? Bear in mind that the proposal voice is more tentative and uncertain than the thesis voice, which has defiinite ideas to convey

Research = Inquiry

PhD research = Making a contribution to a field, linking it to what already exists

Your research question should be real i.e. the answer is not readily available

Hypothesis = “a bold guess” → I think the answer could be…

Research journey

  1. Ask a question
  2. Begin reading (the “river of words” – who else has asked this question, where do they live, what did they find i.e. what is already out there, what already exists
  3. Could lead to the question changing → you find your question has been answered but there are other, related questions that need answering
  4. Interact with others who are looking at similar questions
  5. Research design is central, may include a redesign following a pilot
  6. Be aware of gatekeepers and how to get around them
  7. Know when to stop gathering data, and when to start analysing it (if this is addressed in part in the proposal, it can give guidance to this process)
  8. Following the analysis, you may have to adapt or even to discard some ideas if the data doesn’t support it
  9. Be careful of despair when your ideas aren’t supported
  10. Following analysis comes synthesis (this is the hardest part where many candidates drop out) → putting the thesis together / linking all the ideas
  11. During the writing / rewriting process, you may have to return to the “river of words” to review significant contributions
  12. What is the contribution of the editor and supervisor of the final document?
  13. What do you have to say when all is said and done i.e. what is your contribution to the current understanding and knowledge base of your field?
  14. Be wary of those who presume to “know it all”. How will you defend the unique nature of your research?

Synthesis involves structuring data (following analysis) and linking it to literature. Analysis follows a formula, whereas synthesis is the creative component that leads to your unique contribution

How do you persuade your reader that your data is valid? Without valid data, you can’t build an argument on it

Is your methodology sound enough to convince your reader that your results are trustworthy?

Differing interpretations of the same data can be a result of using different theoretical frameworks that underly the analysis

The proposal should lay out what the researcher wants to do, but should also include limitations i.e. significance (what it will include and why that’s important) vs. limitations (what will be excluded and why)

Negotiate with the supervisors as to what wasn’t done and write it up i.e. explain to examiners and readers what was left out of the study

The literature review is being written and rewritten throughout the process, because you’re reading throughout the journey

Make sure the research question is clear and concise. What is the background to the question? Why is it relevant now and didn’t arise 10 years ago? What makes “now” a good time to try and answer the question?

Can established research methodologies cope with social media?

Yesterday I was talking to my supervisor about how I’m having difficulty designing a protocol for my systematic review.  The guidelines I’m looking at are very good for designing a structured process for searching through the literature, but they’re not very good at helping me to define a search that includes social media.  The JBI Manual doesn’t mention Twitter or Facebook at all, and Cochrane is equally useless to me in this regard.

As if in response to that conversation, I had the following experience earlier today.  I got an email from Twitter informing me that I had a new follower.  I clicked the link and was taken to the profile of someone interested in similar things to me.  I followed him, went through a few of his tweets and ended up following a few of his followers.  One of those followers had tweeted about a page on danah boyd‘s site that was a collection of Research on Twitter and Microblogging.  I found 18 useful papers on that page that I probably would never have found if I’d had to stick to a review protocol that was designed to search commonly recognised sources (e.g. PubMed, CINAHL, library databases, etc).

How can I define the process that I went through today in generic terms (because the same thing can happen when I’m going through news feeds, Delicious, Slideshare, etc.) when it’s so serendipitous?  There doesn’t seem to be an easy way to describe that process in terms that my Dean of Research would understand (I’m uncertain, but I suspect that he’s not on Twitter).

There are other issues.  For example, I can use the blog of an expert in the field to extract an opinion about an intervention, which is great (let’s exclude the problem of defining an expert).  So I can make a list of the blogs of all the experts that I’ll consult, which will never be even close to comprehensive anyway.  How do I then get around the problem of the blog that I add tomorrow, which I might find because of a Google Group that I subscribe to?  Or the “non-expert” blogger I come across who links to a recently published report that I must include?  How about using Mendeley as an article database?  Will my examiners accept it as an appropriate source of literature?  And I can’t even imagine the chaos that’s going to erupt when Wave really gets going in education.

It seems that I can define my protocol loosely, which means that no-one else will be able to reproduce the study and will therefore negate the whole point of a systematic review.  Or, I can define my protocol strictly and potentially miss a hundred important articles, which will make my review equally poor.  Do we need to re-evaluate established research methodologies to take into account the disruptive nature of social media, or am I missing something?

Open research

I’ve been thinking about the concept of open research since listening to Jon Udell’s interview with Jean-Claude Bradley on his open notebook science project.  The idea is similar to the open approach to writing software in that the process is transparent and open to scrutiny by anyone.  This could have important implications for the soundness of the methodology behind the research, the distribution of results and the potential for massive collaboration on research projects.

Open research makes use of social tools like wikis (wikiresearch), blogs, Google Docs and social networks of like-minded individuals, that allow for collaboration, rapid publication and increased access to information for anyone with an internet connection.  There is also the suggestion that openness in research could lead to more innovation by stimulating ideas that allow others to make contributions to the body of knowledge that may not have been the original intent of the researcher.

However, not everyone is comfortable with the idea of conducting research in an open environment, that is subject to scrutiny by everyone and largely against the culture of secrecy in scientific research.  There are definitely issues with the process and one example of how conflict could arise is by publishing primary data openly.  This has the obvious benefit in that anyone could take that information and use it in ways not intended by the researcher, taking data that may have never seen the light of day and creating new knowledge.  The downside is that someone else could beat you to the finish line by publishing your results and negating your work.

There are other approaches that aren’t as “open” as publishing everything concerned with the project.  For example, you could choose to publish only your methodology or ideas around where the project is headed and request input around that, or raw data could be summarised before publishing online.  Other, similar fields are also becoming more mainstream, like open peer review, in which the peer review process of publication is made public, and open notebook science.

What will the world be like when all knowledge is freely available?