Evaluation Scripts

[Isabel Segura]

Dear participants, 

Evaluation scripts  are now available in the following address:

http://www.cs.york.ac.uk/semeval-2013/task9/index.php?id=evaluation-metrics

Best, 

Isabel

[Isabel Segura]

Some errors have been detetected in the evaluation script for the task 9.2 (these errors were made concerning the scoring of ddi classification)
Please again download the script evaluateDDI.jar from http://www.cs.york.ac.uk/semeval-2013/task9/index.php?id=evaluation-metrics

We would like to thank all the participants for their active participation and valuable feedback that allows us to improve the DDIExtraction 2013 task.

Best,

Isabel

[Faisal Chowdhury]

Dear Isabel,

Could you please tell us what is the format of the gold standard dataset?

I downloaded the latest evaluation script for the task 9.2 and ran it as per the instructions mentioned in the readme file. I tested on the annotated xml files which are provided as the training data. I got the following output:

..........
..............

 Gold standard saved in goldNER.txt

Gold loaded. Sentences=0, entities: 0
task9.2-FBK-irst-1 created...
task9.2-FBK-irst-1 does not follow the convention file name for the task!

..........
..............

Best,
Faisal

--
You received this message because you are subscribed to the Google Groups "ddiextraction_semeval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ddiextraction_se...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Faisal Chowdhury]

Dear Isabel,

Could you please tell us what is the format of the gold standard dataset?

I downloaded the latest evaluation script for the task 9.2 and ran it as per the instructions mentioned in the readme file. I tested on the annotated xml files which are provided as the training data. I got the following output:

..........
..............

 Gold standard saved in goldNER.txt

Gold loaded. Sentences=0, entities: 0
task9.2-FBK-irst-1 created...
task9.2-FBK-irst-1 does not follow the convention file name for the task!

..........
..............

Best,
Faisal

On Sun, Mar 10, 2013 at 11:33 AM, Isabel Segura <iseg...@gmail.com> wrote:

--

[Isabel Segura]

Dear Faisal, 

Are you sure that you are running "java -jar evaluateDDI.jar "?

Best, isabel

--
Isabel Segura Bedmar
Despacho 2.2.A.10
Telf: 91 624 99 88
Departamento de Informática
Universidad Carlos III de Madrid,

http://www.inf.uc3m.es/component/comprofiler/userprofile/isegura

[Faisal Chowdhury]

Dear Isabel,

My mistake. I ran the evaluateNER.jar instead of evaluateDDI.jar .

It seems the script is working fine.

Thanks,
Faisal

[Isabel Segura]

Great!!!

[Tamara Bobic]

Hi Isabel, 

I also have a question regarding the evaluation script. I get errors like this one:

MedLine.d135.s1 id sentence wrong[checkFormatTaskDDI] Warning! Some wrongs in the line: MedLine.d135.s1|MedLine.d135.s1.e0|MedLine.d135.s1.e1|0|null

The gold standard folder contains the annotated train data with such ids (e.g. MedLine.d135.s1).

Do you have an idea what could be a mistake here?

Thanks a lot!

Tamara

[Isabel Segura]

Hi Tamara,

Why the name of your dataset is "MedLine"? It should be "DDI-MedLine".

Thanks,

Isabel

[Tamara Bobic]

Hi Isabel,

thanks, it was a small oversight, now it works :)

All the best,

Tamara

[Isabel Segura]

Don't worry!!!. Please do not hesitate to ask me further doubts.

Best,

Isabel