Notion of 'clusters'

[Lucia Specia]

Hi,

I'm a bit confused with the notion of "clusters" for this task: the
description says we should provide as result for the test set:

1) best substitutes
2) 10 best substitutes

That is, just like the lexical substitution task. For the evaluation,
does it make any difference if the substitutions come from a given
cluster? Do we actually need to think in terms of clusters of
"translations"? That is, say my system provides *different* sets of
substitutions for 2 occurrences of "bank" with the same sense. Some of
the replacements in these two sets might be correct, but since the
sets are different, will my system be scored lower?

In other words, are we supposed to identify the clusters of
occurrences of a given ambiguous word and then assign them ALL the
same set of substitutions?

Also, in the .gold files of the trial data there are some numbers,
e.g.:

plant.n.fr 10 :: entreprise 1;exploitation 1;four 1;incinérateur
1;installation 2;station 1;usine 2;établissement 1;

Are the numbers "1, 2" indicative of the cluster?

Thanks,

Lucia

[Els]

Hi Lucia,

- the goal of the task is indeed similar to the English substitution
task:
provide best substitutes + 5 (not 10 !!) best substitutes.

- there will be two evaluations:

1. Evaluation versus the gold standard translations:
these translations were manually picked by the annotators and
can be chosen from different clusters if more than one cluster is
appropriate
=> this evaluation will be used for ranking the systems

2. evaluation on cluster level:
here we will take as a gold standard all translations that are in the
appropriate clusters (picked by the human annotators). If your
system provides good translation outputs, they should also
occur in the clusters that were picked by the annotators.

So you don't need to identify the clusters yourself
(you can of course do that in an intermediate step for
grouping possible translations to choose from),
you only have to provide good translations based on the context of the
target
word in the test sentences.

The numbers you find in the gold standard
are an indication of the frequency of the translations
(e.g. "2" means this translation has been picked twice by the
annotators),
and this is used in the scoring formula.

In case it is still not clear,
you can have a look at the documentation for the task,
that you can find at
http://lt3.hogent.be/semeval/Trial/Task3_doc.pdf

Hope this helps,
Els