Task-1 submission checker and number of runs

[Javier Artiles]

Dear WePS-3 task-1 participants,

I've uploaded a script that should be used to check your task-1 submission before sending it to the organizers, DTD files of the task-1 and task-1 + AE XML files are also included (http://nlp.uned.es/weps/weps3/weps3-task1-submission_checker.zip). Please take a moment to run this script on your data before sending it to us. It will speed up the evaluation process and avoid confusions in case some bug slips into your system output. In case you have any question about the script please write directly to me (jav...@bec.uned.es),

Regarding the number of runs allowed: each team can submit up to five runs.

Best regards,

Javier Artiles.

----------------------------------------------------------

-- WePS-3 Task-1 submission checker (nlp.uned.es/weps/) --

----------------------------------------------------------

USAGE: java -jar weps3-task1-submission-checker.jar [testDataDir] [submissionsDir] -verbose

  [testDataDir]    --> Path to the test web pages directory (e.g. weps-3/data/task-1/test/web_pages/)

  [submissionsDir] --> Path to the runs directory (up to five runs). Each run must be in one directory named with the team ID and the number of run (e.g. submission_dir/uned_1/, submission_dir/uned_2/, ...)

  -verbose         --> With this optional parameter the script outputs each clustering and attributes (in case these are included for participation on the Attribute Extraction subtask).

WePS-3 task-1 submission format:

   - Each run must contain clustering xml files for all the person names in the test set

   - Clustering XML files are named using the person name (+ the '.xml' extension), lowercased and with spaces replaced by underscores (e.g. 'al_feldstein.xml')

   - Clustering solutions must include references to all and only the documents for that name in the test set.

   - Attributes must have as source document one that appears in the same cluster

   - Make sure not to leave empty clusters, empty attribute values, etc.

EXAMPLE: java -jar weps3-task1-submission-checker.jar weps-3/data/task-1/test/web_pages/ submission_dir/ -verbose

--
------------------------------------------------------------
Dr. Javier Artiles Picón
Departamento de Lenguajes y Sistemas Informáticos
ETSI Informática, UNED

Home page: http://nlp.uned.es/~javart

Phone:        +34 91 398 8106
Fax:            +34 91 398 65 35
------------------------------------------------------------