IR, crowdsourcing and protein foldingPosted: August 10, 2010
This morning the New York Times ran an article describing a newly minted approach by researchers at U. of Washington for analyzing the process of amino acid folding. A more thorough discussion appears in Nature.
To avoid the burdensome statistical modeling that is the norm in the field (and about which I confess to near no knowledge) the researchers developed Foldit a game inviting amateurs to help in this work. Quoting the Times on Foldit:
The game, which was competitive and offered the puzzle-solving qualities of a game like Rubik’s Cube, quickly attracted a dedicated following of thousands of players.
In other words, the researchers crowdsourced the problem by posing the work as a game. Aside from sidestepping heavy computation, the researchers found that Foldit led to a level of accuracy on a par with established methods.
With a lot of current interest in crowdsourcing for IR (see Omar Alonso’s slides from an ECIR tutorial and proceedings from the SIGIR2010 crowdsourcing workshop), the article begs the question (for me): what retrieval work could be approached in this way? Of course this question isn’t new; cf. Google’s image labeler. But it’s still resonant.
Gathering relevance judgment via Mechanical Turk is an obvious place for crowdsourcing to enter IR research. But this type of crowdsourcing is qualitatively different than what we see in Foldit: participants are paid to do work that presumably they otherwise wouldn’t do. Not only is this model limiting–it’s also ripe for people to fudge the process in order to get paid without completing the task appropriately.
Opening research problems to crowds in the form of games strikes me as a way to mitigate the problem of people gaming (sorry for the pun) the system. The approach might also help us expand the scope of problems that can be aided by crowdsourcing. In Foldit, user interaction is abstracted in a way that makes it difficult for people to cheat. Andwithout a work/payment model, there’s little incentive to do the job poorly. Most striking, though: Foldit has broken a tremendously complex problem into sub-problems whose solutions make plausible entertainment.
What IR problems lend themselves to this kind of crowdsourcing? The image labeler is certainly one example, though I personally found it about as fun as waiting in an airport terminal for a delayed flight.
What else could we do in this space?
To start what I hope might become a discussion, I’ll offer a few criteria that I think a compelling crowdsourcing game should meet:
- Instant feedback: the game should give information to the player at all times. A real-time display of a performance-based score might do the trick.
- Abandonment & restarting: players should be able to quit or start a game at any time while still making their participation useful.
- Level of difficulty: obviously the game should be neither too hard nor too easy to be enjoyable. Better yet, let the player should choose his or her preferred level of challenge (e.g. work on a larger or smaller part of the problem).
- Manageable chunks of work: Foldit operates by presenting the player with ‘puzzles.’ These are scenarios that involve solving a well-defined problem such as freeing atoms of moving a chain from an unsuitable location to a better spot. Each of these problems is solvable and discrete.
Of course this list of only the sketchiest effort. I’m curious if others have more and better ideas. And of course the real question is how all this can be made to work in IR settings. What problems in IR lend themselves to this kind of solution? If we identify such problems, how do we transform the work into a viable ‘game’ that people would undertake voluntarily and to good effect?