General MT: Croatian only one directional; clarification for humeval

[Tom Kocmi]

Hi All,

 

the language pair English-Croatian is going to be evaluated only in the direction from English to Croatian (devsets are created in this direction as well).

 

As in past years, we are going to run human evaluation for all submitted systems. This year, we do not require participants to participate in the human evaluation, however, we require a system paper submission. As an initial check, each team has to submit at least (one page) abstract of the system description one week after the system submission deadline (TBA: second half of July). It may be a full system description paper or only a draft that can be later modified for the final system description paper.

 

In the unlikely event of an unprecedented number of system submissions that we couldn’t evaluate by human judgement, we may decide to preselect systems for human evaluation by automatic metrics (especially not evaluating lowest-performing unconstrained systems). However, we believe this won't be applied and all primary systems will be evaluated by humans.

 

 

Have a lovely day,

Tom

(in Germany, he/him)