![[Main Page]](/GikiCLEF/images/logoGikiCLEF.png)
For assessment, the first thing to be created after the submissions have been made is a pool of answers, where different justifications for the same answer are considered different answers. In this pooling mechanism differences between HTML and XML are dealt with (that is, only one will remain in the pool).
Then the system can automatically add the information that some answers are correct and self-justified given the info already in the topic management system, as well as correct (but not justified).
Then assessors are presented with a list of answers which they have to - check the correctness through inspecting the pages and the justifications - just check the justifications if one knows the answer is correct but not self justified - add comments for interesting issues (incompatible information in different languages, Wikipedia link translations incorrect, etc.)
In order to make the system as flexible as possible, we may give the same answers to different assessors, and then have a conflict-solving procedure if they do not agree.
After all answers in the pool have been classified, the evaluation program computes the evaluation measures also taking into consideration the more complicated cases:
- correct answers but no justification in any language -- deemed incorrect
- answers which depend on the language (for cases in which different language Wikipedias have conflicting answers)