Assignment adapted from James Allan's CMPSCI 646 course (Fall, 2004) at U. Mass. and Doug Oard's LBSC 796/INFM 718R (Spring 2011) at UMD.
This Excel spreadsheet (based on your hw2) contains three separate sets of judgments for the hits examined in Homework 2. For one of the topics, you will:
1. Agreement on Relevance Judgments
The above spreadsheet should contain three sets of judgments for every document (Web page). The first question you'll answer is: How often do judges agree on relevance? There are four possibilities:
(1.1) For your chosen topic, figure out how often each case happens (both in terms of counts and in terms of percentage). Turn this information in. [10 points]
(1.2) Pick three cases where judgments about a particular hit are not uniform, and briefly speculate why this may be so. Try to employ the concepts of relevance discussed in lecture. Turn this in. [10 points]
2. Adjudication
Adjudication is simply the process of reconciling inconsistent judgments. Do this by simple majority voting; for example, the adjudicated result of NNR is N (based on majority voting). You do not need to turn anything in for this, but you will need the results for the third question.
3. Evaluation of Bing and Google
(3.1) Now, evaluate Bing and Google using the adjudicated relevance judgments you just created (for the topic you chose). Make sure you are pooling judgments from both systems! [10 points for pooling]
Turn in the following information for both search engines:
4. In addition, answer the following questions:
Please post your assignment on your Website.