INEX 2009 Entity-Ranking Track

Overview

The necessity to return entities instead of only documents or elements is becoming increasingly important. This track aims to provide a forum for researchers to compare and evaluate techniques for search systems to return lists of entities.

The track uses the new INEX Wikipedia 2009 collection, where systems may exploit the category metadata associated with entities in entity retrieval as well as available annotations for understanding and aggregating information about entities. For example, consider a category "Italian politicians". The relevant entities are assumed to be labelled with this category or other closely related category in the categorization hierarchy, e.g. "Prime Ministers of Italy". Also, a system could understand the period in which an entity such as "Silvio Berlusconi" has been Prime Minister of Italy by exploiting the info-box annotations.

The track will run in a similar format of the one in 2008, including the main two tasks from 2008:

Entity Ranking (XER), which aims to return entities that satisfy a topic described in natural language text.
List Completion (LC) task, where a number of examples are given. The aim is to complete this partial list of answers. The List Completion runs will contribute to the pool of documents used for the relevance assessments.

For these tasks the set of 2007 and 2008 genuine XER and LC topics will be used as topics on the new collection. Relevance judgement will be performed by participants on a pool of entities built using submitted runs.

Depending on the number of participants, new XER and LC topics could be created as well.

The LC task could possibly be extended in order to consider as initial examples the set of entities assessed as relevant during previous years assessment, making it more similar to a relevance feedback task on Entity Ranking.

Files

2009 Topics
2008 Topics

Schedule

23/Jun/2009	Release of final set of topics
23/Jun/2009	Release of Result Submission Specification
15/Aug/2009	Submission deadline for ER search results
31/Aug/2009	Release of assessment pool
2/Oct/2009	Submission deadline for relevance assessments
2/Nov/2009	Release ER evaluation results
23/Nov/2009	Submission deadline for papers for pre-proceedings (all tracks)
30/Nov/2009	Release of workshop pre-proceedings
6-10/Dec/2009	INEX Workshop in Brisbane, Australia

Organizers

Gianluca Demartini
L3S
demartini@l3s.de

Tereza Iofciu
L3S
iofciu@l3s.de

Arjen P. de Vries
CWI
Arjen.de.Vries@cwi.nl