[Main Page]

Assessment in the GikiCLEF assessment system

(Difference between revisions)
Current revision (12:05, 18 June 2009) (view source)
(→The GikiCLEF assessment system)
 
(6 intermediate revisions not shown.)
Line 1: Line 1:
-
For assessment, the first thing to be created after all submissions have been received is an answer pool, in which different justifications for the same answer are considered as different answers.
+
=== The GikiCLEF assessment system ===
-
This pooling mechanism handles differences between HTML and XML, that is, only one will remain in the pool, as well as automatically filters out those pages which are easy to discard as invalid answers (such as disambiguation pages).
+
After all submissions have been received, the assessment process starts by producing an answer pool, in which different justifications for the same answer are considered as different answers.  
-
Then, and in order to minimize the assessors' workload, the system automatically adds the information that some answers are correct and self-justified given the info already in the topic management system, as well as correct (but not justified).  
+
This pooling mechanism handles differences between HTML and XML, that is, only one will remain in the pool, as well as automatically filters out those pages which are easy to discard as invalid answers (such as disambiguation pages, or redirects).
-
From an assessor point of view, they are presented with a list of answers which they have to  
+
Then, and in order to minimize the assessors' workload, the system automatically adds the information it already has about the topics, namely that some answers are correct and self-justified, or correct (but not self justified), provided this information is already present in the topic management system.
-
- either check the correctness through inspecting the pages and the justifications if existing
+
 
 +
An assessor is then presented with a list of answers for which s/he has to  
 +
- either check the correctness through inspecting the pages and the justifications
- or just check the justifications because it is already known by the system that the answer is correct but not self justified
- or just check the justifications because it is already known by the system that the answer is correct but not self justified
Also, the assessor can add comments about interesting issues (incompatible information in different languages, Wikipedia link translations incorrect, etc.) which may have a bearing on the evaluation score
Also, the assessor can add comments about interesting issues (incompatible information in different languages, Wikipedia link translations incorrect, etc.) which may have a bearing on the evaluation score
-
In order to make the system as flexible as possible, the same answers can be evaluated by different assessors, and then have a conflict-solving procedure if they do not agree.
+
The system allows the same answers to be evaluated by different assessors, and then dutifully stores all assessments, indexed by assessor, so that a subsequent a conflict-solving procedure can be stared if they do not agree.
-
After all answers in the pool have been classified, the evaluation program computes the evaluation measures, also taking into consideration the more complicated cases:
+
Assessors do not have access to the other assessors judgements while assessing, nor to the comments already entered about this particular answer.
-
 
+
-
* correct answers but no justification in any language -- deemed incorrect
+
-
* answers which depend on the language (for cases in which different language Wikipedias have conflicting answers)
+
 +
After all answers in the pool have been classified, it is time for the evaluation system to take control.
[http://www.linguateca.pt/GikiCLEF/index.php/Main_Page Back to the main page]
[http://www.linguateca.pt/GikiCLEF/index.php/Main_Page Back to the main page]
 +
 +
[[Documentation]] of the topic and assessment system (in progress, so far only in Portuguese)

Current revision

The GikiCLEF assessment system

After all submissions have been received, the assessment process starts by producing an answer pool, in which different justifications for the same answer are considered as different answers.

This pooling mechanism handles differences between HTML and XML, that is, only one will remain in the pool, as well as automatically filters out those pages which are easy to discard as invalid answers (such as disambiguation pages, or redirects).

Then, and in order to minimize the assessors' workload, the system automatically adds the information it already has about the topics, namely that some answers are correct and self-justified, or correct (but not self justified), provided this information is already present in the topic management system.

An assessor is then presented with a list of answers for which s/he has to - either check the correctness through inspecting the pages and the justifications - or just check the justifications because it is already known by the system that the answer is correct but not self justified Also, the assessor can add comments about interesting issues (incompatible information in different languages, Wikipedia link translations incorrect, etc.) which may have a bearing on the evaluation score

The system allows the same answers to be evaluated by different assessors, and then dutifully stores all assessments, indexed by assessor, so that a subsequent a conflict-solving procedure can be stared if they do not agree.

Assessors do not have access to the other assessors judgements while assessing, nor to the comments already entered about this particular answer.

After all answers in the pool have been classified, it is time for the evaluation system to take control.

Back to the main page

Documentation of the topic and assessment system (in progress, so far only in Portuguese)

Retrieved from "https://www.linguateca.pt/GikiCLEF/index.php/Assessment_in_the_GikiCLEF_assessment_system"