Questions about evaluation?

[Joseph Turian]

1. For the evaluation, will we be told what are the gold-standard
clusters for the word senses? (And then choose between them with our
system.)

2. For the relaxed evaluation, can our system outputs a guess more
than once, in order to increase the weight of this guess?

Thanks,
Joseph

[Els]

Hi Joseph,

1. for the evaluation, we will only provide English test sentences
where the target word
has been marked (format we also used for the example sentences for the
trial data),
no lists of gold-standard clusters.

2. I will come back to you for the second question. I know this is
allowed for the
lexical substitution task, but there the construction of the gold
standard is not
bound to a parallel corpus.

Best,
Els

[Els]

Hi,

we decided to NOT allow to have the same guess more than once.
For some words, you can get good scores by picking the most frequent
translation
and adding this translation a couple of times would influence the
scores too much.

Best,
Els.