The goal of this project is to create a system that will collect URLs for a variety of topics, submitted and verified through Amazon Mechanical Turk.

A. URL collection part

The system should create HITs on Amazon Mechanical Turk that will ask Amazon Mechanical Turk workers to submit URLs about a particular topic.
In order to avoid collecting duplicate URLs, we will have the following verification procedure:

The user submits the URL and presses a "check URL" button. Then, using some lightweight querying mechanism (e.g., through AJAX) we will verify that the URL is not in the list of already collected URLs. If the URL is indeed not in the database, then we will allow the MTurk worker to submit the HIT.

The system should save though *all* the URLs checked/submitted by the Turkers, so that we can see if there are URLs that are entered too often, and we can explicitly ask workers not to enter these URLs.

We may need some extra safeguards to avoid submission of "almost duplicate" HITs (e.g., strip down the URL to only its domain). But we can work this out as we work on the system.

B. URL verification part

The submitted URLs will be then submitted to MTurk again, when 5-10 other workers will vote on whether the URL is indeed about the specified topic. If the majority of users agree that the URL is about the topic, the original Turkers that submitted the URL gets paid.

C. FrontEnd

The system should be possible to manage through a web-based front-end. The administrator of the system should be able to add requests for new topics, specify how many URLs are needed, and specify payments for HITs, and be able to specify required qualifications for the MTurk workers (e.g., 95% approval rate, 500 approved HITs, and so on).

Some guidelines:

The TurkIt toolkit maybe of value, but you are definitely not required to use it.


The HITs on Mechanical Turk should be created as "External HITs" and will be hosted in an external server. I can give you access to a Windows 2003 server or we can set up an Amazon EC2 instance to handle that.