Re: System names for contrastive systems in WMT evaluation task?

[Chris Callison-Burch]

Hi Gregor,

> I am currently running some systems to our CDER evaluation scripts,
> and I noticed that the sysid does not contain the "contrastive
> number" in the contrastive files.
> Should I use "rwth" + "rwth.contrastive1" + "rwth.contrastive2"
> + ... as pseudo-sysids in the result files?

Yes, this sounds like a sensible proposal.

Thanks,
Chris

On Jan 16, 2009, at 2:36 PM, Gregor Leusch wrote:

> Hi,
>
> I am currently running some systems to our CDER evaluation scripts,
> and I noticed that the sysid does not contain the "contrastive
> number" in the contrastive files.
>
> So, for example, in all of the files
> rwth.de-en.contrastive1.sgm
> rwth.de-en.contrastive2.sgm
> rwth.de-en.contrastive3.sgm
> rwth.de-en.contrastive4.sgm
> rwth.de-en.primary.sgm
>
> sysid is set to "rwth".
>
> According to http://www.statmt.org/wmt09/evaluation-task.html , the
> result file should identify the system by its sysid -- which would
> mean, then, that the scores for the different contrastive systems
> (and the primary) can not be decided from each other.
>
> So what should we do?
>
> Use "rwth" + "rwth.contrastive1" + "rwth.contrastive2" + ... as
> pseudo-sysids in the result files?
>
> Best,
>
> Gregor
>
> (who, unfortunately, can not post to the google list)
>
>
> --
> Dipl.-Inform. Gregor Leusch Tel
> +49-241-80-21618
> Chair of Computer Science 6 Fax
> +49-241-80-22219
> RWTH Aachen University leu...@informatik.rwth-aachen.de
> D-52056 Aachen, Germany www-i6.informatik.rwth-aachen.de/
> ~leusch/
>