Evaluation contests for Portuguese

Esta página em português

At the beginning of 2002, Linguateca started a process for the evaluation of the various areas of the computational processing of Portuguese. The final goal of this process was to initiate one or more evaluation contests involving the relevant members of the scientific community as participants.

Currently, Linguateca's main activities are

• Participating on the organization of the future of CLEF in what concerns Portuguese

So far, this process has produced (most of the information below is only available in Portuguese)

• a motivation page with some examples: recuperação de informação, corpora anotados
• an overview of international evaluation contests (dated 2002)
• an initial list of application areas and resources to which an evaluation contest model could be applied, together with a form for registering interest per area. (People registered: list by area, sorted by number of people; list by area, weighted by number of areas each person chose.)
• a discussion list avaliofilia to discuss the steps to be taken in the preparation of the different contests, as well as provide a scientific forum for those interested in evaluation of Portuguese
• a preparatory encounter as a satelite of PorTAL , in Faro 27 June 2002 (call, final program including presentations)
• a reference list on NLP evaluation applied to Portuguese (still embryonic)
• a general bibliography (in English) about evaluation of NLP systems and in language engineering
• the organization of the first evaluation contest for Portuguese, Primeiras morfolimpíadas para o português
• the start of evaluating named entity recognition in Portuguese: Identificação e estudo preliminar dos requisitos (2003)
• the start of MT evaluation involving Portuguese
• the start of several activities involving IR evaluation
• the first workshop devoted to evaluation contest involving Portuguese, Avalon'2003, Encontro de Avaliação Conjunta de Sistemas de Processamento Computacional do Português, in Faro 28 June 2004, as a satellite of PROPOR 2003.
• the publication of the first book about this paradigm in Portuguese (24 chapters, 30 authors).
• First HAREM, the first named entity recognition evaluation contest in Portuguese
• the organization of the First HAREM workshop at 15 July 2006 in Porto, together with the First Linguateca Summer School
• the publication of a book about the First HAREM
• the organization of GikiP at CLEF, a multilingual evaluation contest on finding lists of answers in Wikipedia
• Second HAREM, the second named entity recognition evaluation contest in Portuguese
• the organization of the Second HAREM workshop at 7 September 2008 as satelite of PROPOR 2008.
• the publication of another book, now about the Second HAREM
• the organization of GikiCLEF at CLEF, a multilingual evaluation contest on finding culturally biased lists of answers in 10 different Wikipedia language versions