Precise guidelines for GikiCLEF assessors

(Difference between revisions)

Current revision

Upon entering the system, and choosing the option Assess answers, each assessor receives a list of answers to assess, depending on the languages he registered in, and in fact also depending on the administrators knowledge of his or her linguistic capabilities and the pool size for each language.

Some of these have been chosen to on purpose overlap with other assessors, but the assessor does not know which are which.

From the point of view of system design, note that:

• Assessors are only requested to assess answers which are not yet stored in the system as correct and self-justified.
• Also, and in order to minimize assessors' work, when the system already knows that a particulart answer is correct but that the page in itself does not provide a justification for the answer, the assessor is not required to do any work.
• If, on the other hand, the answer is known by the system to be correct and in need of external justification, the assessor is only required to check the validity of the justification provided.

To make the assessment process easier, the assessor is able to see the pre-selected answers to the topic, although warned that they are not complete -- and in fact in a few cases they have turned out to even be wrong (and were accordingly removed). The only exception is the one case where the GikiCLEF question was closed instead of open (name the five Italian regions), where only five can be accepted as correct.

Finally, if a particular answer was considered incorrect by the assessor, in principle no more equal answers (with different justifications) will be presented to that assessor for assessing. This was however not implemented yet.

An assessor is however free to comment and check all answers if s/he wishes to, as well as redo the assessment as many times as necessary, as well as fill in comments about things that are not or may not be totally clear.

In the bottom of the page for each answer to be assessed there was a link that identified the answer for discussion or problem reporting that assessors were also able to use to report problems or discuss complex issues.

Specific check lists

After repeated discussion with individual assessors, the following instructions were sent to the assessors' list -- unfortunately not on time for avoiding a lot of reassessing work -- which may be interesting also to participants and the public in general:

• GikiCLEF assessment should be easy to perform, because in the first step you don't need to read the whole document to find the answer. It the answer (=title of the wikipedia article) is not of the type that was asked for, the answer is INCORRECT. Only if it is correct do you need to read or find whether the justification is enough. So, "flag of Peru" is never a right answer for a question like "Which countries", although it might be perfect for justification.

• Note about topic 2: although images are not shown in the collection, we are assuming that a user would be able to see at once the flag, and therefore and based on the visual clue, decide. So please try to visualize the flag when assessing that topic. The same for images of mountains and of rivers if they appear on the main page of the answer.

Detailed example

Take the following answer to GC-2009-01 "List the Italian places where Ernest Hemingway visited during his life." and imagine you are assessing the answer <Fossalta di Piave> (which is correct).

In the text of this page it stands:

<While serving as an ambulence driver, Ernest Hemingway was injured there on July 8, 1918.>

which is an obvious justification for claiming that Hemingway WAS there. So you click on Correct as far as the answer is concerned (or the system has already done that for you).

Then you proceed to assess whether the asnwer is justified, and read: <Justification: Fossalta_di_Piave_e6b6 (XML) No further justification presented.>

This means: the justification presented by the system is the document itself, no further proofs are supposed to be necessary.

And this is true in that case. So please click on Justified and marvel on such a miracle.

If the answer was incorrect, mark the answer as Incorrect, and you don't need to do anything about justification

If you are uncertain but are sure that the document itself and or further "justifications" presented do not let you decide, please mark the answer as Uncertain, and Unjustified

Now imagine that you had got this very same answer to assess in another language, let us say Portuguese, where nothing was said about Hemingway, but you already knew that it was correct. Then you could classify it as Correct, but Not Justified.

On the other hand, you could have been asked to assess it BEFORE you had read the English or Italian one, and you are not an expert on Hemingway. Then Uncertain was the right answer. You are not required to go and reassess this answer if you learned it afterwards. But you could, of course. Both results would result in the same overall result for GikiCLEF systems.

Back to the main page