Test script

Lucia Specia

unread,

Dec 10, 2011, 7:48:24 AM12/10/11

to SemEval2012_LexicalSimplification

Dear all,

We've uploaded the test script to the task's website:
http://www.cs.york.ac.uk/semeval-2012/task1/data/uploads/test-script.zip

There's a readme file with a brief explanation about our main
evaluation metric. Let us know if you have any question about it.

Best,

Lucia, Sujay and Rada

Himanshu

unread,

Jan 27, 2012, 1:25:54 AM1/27/12

to SemEval2012_LexicalSimplification

Hi ,

> We've uploaded the test script to the task's website:http://www.cs.york.ac.uk/semeval-2012/task1/data/uploads/test-script.zip
>
> There's a readme file with a brief explanation about our main
> evaluation metric. Let us know if you have any question about it.

On your main website this is specified :

> Evaluation:
> The evaluation methodology is based on ranking. Two variations are considered:
> the best simplification (simplest substitution) and the overall ranking of the substitutes.

Is the "best simplification" still another criteria for evaluating
the system performance.
Since it is not included in the test-script and the Evaluation Metric
Description.
Would it be included ? Please Clarify.

--- Himanshu

Lucia Specia

unread,

Jan 27, 2012, 5:02:07 AM1/27/12

to semeval2012_lexi...@googlegroups.com

Dear Himanshu,

Well spotted. After experimenting with a few evaluation strategies, we
found that the pairwise ranking is the best option, so there will be
no 'best substitution', I'll change the text on the website.