Assessment in the GikiCLEF assessment system - Revision history

Diana: /* The GikiCLEF assessment system */

2009-06-18T12:05:30Z

The GikiCLEF assessment system

←Older revision		Revision as of 12:05, 18 June 2009
Line 12:		Line 12:
	Also, the assessor can add comments about interesting issues (incompatible information in different languages, Wikipedia link translations incorrect, etc.) which may have a bearing on the evaluation score		Also, the assessor can add comments about interesting issues (incompatible information in different languages, Wikipedia link translations incorrect, etc.) which may have a bearing on the evaluation score

-	The system allows the same answers to be evaluated by different assessors, and then ~~start have~~ a conflict-solving procedure if they do not agree.	+	The system allows the same answers to be evaluated by different assessors, and then dutifully stores all assessments, indexed by assessor, so that a subsequent a conflict-solving procedure can be stared if they do not agree.
		+
		+	Assessors do not have access to the other assessors judgements while assessing, nor to the comments already entered about this particular answer.

	After all answers in the pool have been classified, it is time for the evaluation system to take control.		After all answers in the pool have been classified, it is time for the evaluation system to take control.
Line 19:		Line 21:

	[[Documentation]] of the topic and assessment system (in progress, so far only in Portuguese)		[[Documentation]] of the topic and assessment system (in progress, so far only in Portuguese)
-
-
-	Upon entering the system, and choosing the option ''Assess answers'', each assessor receives a list of answers to assess, depending on the languages he was registered with and chosen so that some of them will also be assessed by others.
-
-	Assessors are only requested to assess answers which are not yet stored in the system as correct and self-justified. Also, when the system already knows that a particulart answer is correct but that the page in itself does not provide a justificatin for the answer, the assessor does not need to say that again.
-
-	~~If on the other hand the answer is known by the system to be correct and in need of external justification, the assessor is only required to check the validity of the justification provided.~~
-
-	~~Finally, if a particular answer was considered incorrect by an assessor, no more equal answers (with different justifications) will be presented to that assessor for assessing.~~
-
-	~~An assessor is however free to comment and check all answers if s/he wishes to, as well as fill in comments about things that are not or may not be totally clear.~~

Diana at 11:39, 18 June 2009

2009-06-18T11:39:33Z

←Older revision		Revision as of 11:39, 18 June 2009
Line 1:		Line 1:
		+	=== The GikiCLEF assessment system ===
		+
	After all submissions have been received, the assessment process starts by producing an answer pool, in which different justifications for the same answer are considered as different answers.		After all submissions have been received, the assessment process starts by producing an answer pool, in which different justifications for the same answer are considered as different answers.

Line 17:		Line 19:

	[[Documentation]] of the topic and assessment system (in progress, so far only in Portuguese)		[[Documentation]] of the topic and assessment system (in progress, so far only in Portuguese)
		+
		+
		+	Upon entering the system, and choosing the option ''Assess answers'', each assessor receives a list of answers to assess, depending on the languages he was registered with and chosen so that some of them will also be assessed by others.
		+
		+	Assessors are only requested to assess answers which are not yet stored in the system as correct and self-justified. Also, when the system already knows that a particulart answer is correct but that the page in itself does not provide a justificatin for the answer, the assessor does not need to say that again.
		+
		+	If on the other hand the answer is known by the system to be correct and in need of external justification, the assessor is only required to check the validity of the justification provided.
		+
		+	Finally, if a particular answer was considered incorrect by an assessor, no more equal answers (with different justifications) will be presented to that assessor for assessing.
		+
		+	An assessor is however free to comment and check all answers if s/he wishes to, as well as fill in comments about things that are not or may not be totally clear.

Diana at 11:17, 29 April 2009

2009-04-29T11:17:29Z

←Older revision		Revision as of 11:17, 29 April 2009
Line 16:		Line 16:
	[http://www.linguateca.pt/GikiCLEF/index.php/Main_Page Back to the main page]		[http://www.linguateca.pt/GikiCLEF/index.php/Main_Page Back to the main page]

-	[[Documentation ~~Documentation~~ of the topic and assessment system]] (in progress, so far only in Portuguese)	+	[[Documentation]] of the topic and assessment system (in progress, so far only in Portuguese)

Diana at 11:16, 29 April 2009

2009-04-29T11:16:57Z

←Older revision		Revision as of 11:16, 29 April 2009
Line 16:		Line 16:
	[http://www.linguateca.pt/GikiCLEF/index.php/Main_Page Back to the main page]		[http://www.linguateca.pt/GikiCLEF/index.php/Main_Page Back to the main page]

-	[[Documentation of the topic and assessment system]] (in progress, so far only in Portuguese)	+	[[Documentation Documentation of the topic and assessment system]] (in progress, so far only in Portuguese)

Diana at 11:16, 29 April 2009

2009-04-29T11:16:02Z

←Older revision		Revision as of 11:16, 29 April 2009
Line 16:		Line 16:
	[http://www.linguateca.pt/GikiCLEF/index.php/Main_Page Back to the main page]		[http://www.linguateca.pt/GikiCLEF/index.php/Main_Page Back to the main page]

-	[[Documentation]] (so far only in Portuguese)	+	[[Documentation of the topic and assessment system]] (in progress, so far only in Portuguese)

Diana at 11:12, 29 April 2009

2009-04-29T11:12:50Z

←Older revision		Revision as of 11:12, 29 April 2009
Line 15:		Line 15:

	[http://www.linguateca.pt/GikiCLEF/index.php/Main_Page Back to the main page]		[http://www.linguateca.pt/GikiCLEF/index.php/Main_Page Back to the main page]
		+
		+	[[Documentation]] (so far only in Portuguese)

Diana at 12:41, 30 March 2009

2009-03-30T12:41:30Z

←Older revision		Revision as of 12:41, 30 March 2009
Line 1:		Line 1:
-	~~For assessment, the first thing to be created after~~ all submissions have been received is an answer pool, in which different justifications for the same answer are considered as different answers.	+	After all submissions have been received, the assessment process starts by producing an answer pool, in which different justifications for the same answer are considered as different answers.

-	This pooling mechanism handles differences between HTML and XML, that is, only one will remain in the pool, as well as automatically filters out those pages which are easy to discard as invalid answers (such as disambiguation pages).	+	This pooling mechanism handles differences between HTML and XML, that is, only one will remain in the pool, as well as automatically filters out those pages which are easy to discard as invalid answers (such as disambiguation pages, or redirects).

-	Then, and in order to minimize the assessors' workload, the system automatically adds the information that some answers are correct and self-justified ~~given the info already in the topic management system~~, ~~as well as~~ correct (but not justified).	+	Then, and in order to minimize the assessors' workload, the system automatically adds the information it already has about the topics, namely that some answers are correct and self-justified, or correct (but not self justified), provided this information is already present in the topic management system.

-	~~From an~~ assessor ~~point of view, they are~~ presented with a list of answers which ~~they have~~ to	+	An assessor is then presented with a list of answers for which s/he has to
-	- either check the correctness through inspecting the pages and the justifications ~~if existing~~	+	- either check the correctness through inspecting the pages and the justifications
	- or just check the justifications because it is already known by the system that the answer is correct but not self justified		- or just check the justifications because it is already known by the system that the answer is correct but not self justified
	Also, the assessor can add comments about interesting issues (incompatible information in different languages, Wikipedia link translations incorrect, etc.) which may have a bearing on the evaluation score		Also, the assessor can add comments about interesting issues (incompatible information in different languages, Wikipedia link translations incorrect, etc.) which may have a bearing on the evaluation score

-	~~In order to make the~~ system ~~as flexible as possible,~~ the same answers ~~can~~ be evaluated by different assessors, and then have a conflict-solving procedure if they do not agree.	+	The system allows the same answers to be evaluated by different assessors, and then start have a conflict-solving procedure if they do not agree.
-		+
-	~~After all answers in the pool have been classified, the evaluation program computes the evaluation measures, also taking into consideration the more complicated cases:~~	+
-		+
-	* correct answers but no justification in any language -- deemed incorrect	+
-	* answers which depend on the language (for cases in which different language Wikipedias have conflicting answers)	+

		+	After all answers in the pool have been classified, it is time for the evaluation system to take control.

	[http://www.linguateca.pt/GikiCLEF/index.php/Main_Page Back to the main page]		[http://www.linguateca.pt/GikiCLEF/index.php/Main_Page Back to the main page]

Diana at 14:15, 26 March 2009

2009-03-26T14:15:09Z

←Older revision		Revision as of 14:15, 26 March 2009
Line 1:		Line 1:
-	For assessment, the first thing to be created after ~~the~~ submissions have been ~~made~~ is a pool ~~of answers~~, ~~where~~ different justifications for the same answer are considered different answers~~. In this pooling mechanism differences between HTML and XML are dealt with (that is, only one will remain in the pool)~~.	+	For assessment, the first thing to be created after all submissions have been received is an answer pool, in which different justifications for the same answer are considered as different answers.

-	~~Then the system can automatically add the information that some answers are correct~~ and ~~self-justified given the info already~~ in the ~~topic management system~~, as well as ~~correct~~ (~~but not justified~~).	+	This pooling mechanism handles differences between HTML and XML, that is, only one will remain in the pool, as well as automatically filters out those pages which are easy to discard as invalid answers (such as disambiguation pages).

-	Then ~~assessors are presented with a list of answers which they have~~ to	+	Then, and in order to minimize the assessors' workload, the system automatically adds the information that some answers are correct and self-justified given the info already in the topic management system, as well as correct (but not justified).
-	~~- check~~ the ~~correctness through inspecting~~ the ~~pages and~~ the ~~justifications~~	+
-	- ~~just check~~ the ~~justifications if one knows~~ the ~~answer is~~ correct but not ~~self~~ justified	+
-	~~- add comments for interesting issues (incompatible information in different languages, Wikipedia link translations incorrect, etc~~.)	+

-	~~In order~~ to ~~make~~ the system ~~as flexible as possible~~, ~~we may give~~ the ~~same answers to~~ different ~~assessors~~, ~~and then~~ have a ~~conflict-solving procedure if they do not agree.~~	+	From an assessor point of view, they are presented with a list of answers which they have to
		+	- either check the correctness through inspecting the pages and the justifications if existing
		+	- or just check the justifications because it is already known by the system that the answer is correct but not self justified
		+	Also, the assessor can add comments about interesting issues (incompatible information in different languages, Wikipedia link translations incorrect, etc.) which may have a bearing on the evaluation score

-	After all answers in the pool have been classified, the evaluation program computes the evaluation measures also taking into consideration the more complicated cases:	+	In order to make the system as flexible as possible, the same answers can be evaluated by different assessors, and then have a conflict-solving procedure if they do not agree.
		+
		+	After all answers in the pool have been classified, the evaluation program computes the evaluation measures, also taking into consideration the more complicated cases:

	* correct answers but no justification in any language -- deemed incorrect		* correct answers but no justification in any language -- deemed incorrect

Diana at 13:17, 24 March 2009

2009-03-24T13:17:07Z

←Older revision		Revision as of 13:17, 24 March 2009
Line 1:		Line 1:
-	For assessment, the first thing to be created after the submissions have been made is a pool of answers, where different justifications for the same answer are considered different answers.	+	For assessment, the first thing to be created after the submissions have been made is a pool of answers, where different justifications for the same answer are considered different answers. In this pooling mechanism differences between HTML and XML are dealt with (that is, only one will remain in the pool).

	Then the system can automatically add the information that some answers are correct and self-justified given the info already in the topic management system, as well as correct (but not justified).		Then the system can automatically add the information that some answers are correct and self-justified given the info already in the topic management system, as well as correct (but not justified).
Line 6:		Line 6:
	- check the correctness through inspecting the pages and the justifications		- check the correctness through inspecting the pages and the justifications
	- just check the justifications if one knows the answer is correct but not self justified		- just check the justifications if one knows the answer is correct but not self justified
		+	- add comments for interesting issues (incompatible information in different languages, Wikipedia link translations incorrect, etc.)

	In order to make the system as flexible as possible, we may give the same answers to different assessors, and then have a conflict-solving procedure if they do not agree.		In order to make the system as flexible as possible, we may give the same answers to different assessors, and then have a conflict-solving procedure if they do not agree.
		+
		+	After all answers in the pool have been classified, the evaluation program computes the evaluation measures also taking into consideration the more complicated cases:
		+
		+	* correct answers but no justification in any language -- deemed incorrect
		+	* answers which depend on the language (for cases in which different language Wikipedias have conflicting answers)
		+

	[http://www.linguateca.pt/GikiCLEF/index.php/Main_Page Back to the main page]		[http://www.linguateca.pt/GikiCLEF/index.php/Main_Page Back to the main page]

Diana at 12:57, 24 March 2009

2009-03-24T12:57:19Z

←Older revision		Revision as of 12:57, 24 March 2009
Line 9:		Line 9:
	In order to make the system as flexible as possible, we may give the same answers to different assessors, and then have a conflict-solving procedure if they do not agree.		In order to make the system as flexible as possible, we may give the same answers to different assessors, and then have a conflict-solving procedure if they do not agree.

-	[[http://www.linguateca.pt/GikiCLEF/index.php/Main_Page Back to the main page]]	+	[http://www.linguateca.pt/GikiCLEF/index.php/Main_Page Back to the main page]