Yesterday I was talking to my supervisor about how I’m having difficulty designing a protocol for my systematic review. The guidelines I’m looking at are very good for designing a structured process for searching through the literature, but they’re not very good at helping me to define a search that includes social media. The JBI Manual doesn’t mention Twitter or Facebook at all, and Cochrane is equally useless to me in this regard.
As if in response to that conversation, I had the following experience earlier today. I got an email from Twitter informing me that I had a new follower. I clicked the link and was taken to the profile of someone interested in similar things to me. I followed him, went through a few of his tweets and ended up following a few of his followers. One of those followers had tweeted about a page on danah boyd‘s site that was a collection of Research on Twitter and Microblogging. I found 18 useful papers on that page that I probably would never have found if I’d had to stick to a review protocol that was designed to search commonly recognised sources (e.g. PubMed, CINAHL, library databases, etc).
How can I define the process that I went through today in generic terms (because the same thing can happen when I’m going through news feeds, Delicious, Slideshare, etc.) when it’s so serendipitous? There doesn’t seem to be an easy way to describe that process in terms that my Dean of Research would understand (I’m uncertain, but I suspect that he’s not on Twitter).
There are other issues. For example, I can use the blog of an expert in the field to extract an opinion about an intervention, which is great (let’s exclude the problem of defining an expert). So I can make a list of the blogs of all the experts that I’ll consult, which will never be even close to comprehensive anyway. How do I then get around the problem of the blog that I add tomorrow, which I might find because of a Google Group that I subscribe to? Or the “non-expert” blogger I come across who links to a recently published report that I must include? How about using Mendeley as an article database? Will my examiners accept it as an appropriate source of literature? And I can’t even imagine the chaos that’s going to erupt when Wave really gets going in education.
It seems that I can define my protocol loosely, which means that no-one else will be able to reproduce the study and will therefore negate the whole point of a systematic review. Or, I can define my protocol strictly and potentially miss a hundred important articles, which will make my review equally poor. Do we need to re-evaluate established research methodologies to take into account the disruptive nature of social media, or am I missing something?