By way of shameless promotion, I am currently encouraging people to help me evaluate an experimental IR system that searches microblog (Twitter) data. To participate, please see:
Please consider giving it a moment…even a brief moment.
Now, onto a more substantive matter: I’ve been wrestling with the validity of testing an IR system (particularly a microblog IR system) using a so-called snowball sampling technique. For the uninitiated, snowball sampling involves recruiting a small number of people to participate in a study with the explicit aim that they will, in turn, encourage others to participate. The hope is that participation in the study will extend beyond the narrow initial sample as subjects recruit new participants.
Snowball sampling has clear drawbacks. Most obviously, it is ripe for introducing bias into one’s analysis. The initial “seed” participants will drive the demographics of subsequent recruits. This effect could amplify any initial bias. The non-random (assuming it is non-random) selection of initial participants, and their non-random selection of recruits calls into question the application of standard inferential statistics at the end of the study. What status does a confidence interval on, say, user satisfaction derived from a snowball sample have with respect to the level of user satisfaction in the population?
However, snowball sampling has its merits, too. Among these is the possibility of obtaining a reasonable number of participants in the absence of a tractable method for random sampling.
In my case, I have decided that a snowball sample for this study is worth the risks it entails. In order to avoid poisoning my results, I’ll keep description of the project to a minimum.
But I feel comfortable saying that my method of recruiting includes dissemination of a call for participation in several venues:
- Via a twitter post with a call for readers to retweet it.
- Via this blog post!
- By email to two mailing lists (one a student list, and the other a list of Twitter researchers).
In this case, the value of a snowball sample extends beyond simply acquiring a large N. The fact that Twitter users are connected by Twitter’s native subscription model suggests to me that the fact that my sample will draw many users who are “close” to my social network is not a liability. Instead it will, I hope, lend a level of realism to how a particular sub-community functions.
One problem with these rose-colored lenses is that I have no way to characterize this sub-community formally. Inferences drawn from this sample may generalize to some group. But what group is that?
Obviously some of the validity of this sample will have to do with the nature of the data collected and the research questions to be posed against it, neither of which I’m comfortable discussing yet. But I would be interested to hear what readers think: does snowball sampling have merits or liabilities for research on the use of systems that inherently rely on social connections that do not pertain to situations lacking explicit social linkage?