submission question

[Ted Pedersen]

Greetings all,

Just a question on the submission process.

Must our systems produce results on the same scale as you have in the
training data (a 0 - 100 scale, where 0-25 is low, 38-62 is medium,
and 75-100 is high?).

If that is the case, should we assume that our fine grained scores
will automatically be considered as coarse grained scores, or do we
need to submit those separately?

If that's not the case and we can use our own scoring scale, should we
then submit separate fine and coarse grained files (since you won't
necessarily know what our binning strategy is...) And if we did that,
would each pair of fine/coarse grained scores count as a single
variant (so we'd end up with 3 pairs of scores, or six files...)

The 3 variant limit is based on the following from the test data...

> 3. Submission information
> Please send the output of your participating system to disco201...@gmail.com, no later than
> April 8, 2011. You can submit up to 3 variants for your system.
> Don't forget to submit a 4-page system description on the same date or earlier. Please check
> http://disco2011.fzi.de/ for details.

Thanks!
Ted

--
Ted Pedersen
http://www.d.umn.edu/~tpederse

[Organizer DISCo Workshop 2011]

Hi all, 

thanks for pointing out the ambiguities, Ted, I'll be happy to clarify:

The fine grained scores should be on the same scale as ours since we directly measure how much 'off' they are using our scoring scripts (besides other measures). 

For the coarse-grained scores, you are free to use your own binning. However, if you have optimized according to the training and validation data, we recommend using "low" for 0-33, "medium" for 34-66 and "high" for 67-100. Note that we will only test on clear-cut cases, having removed scores between 26 and 37, a well as between 63 and 74 from the test portion. Still, you should submit coarse scores for all words.  

A system submission per language consists of a pair of files for fine and coarse-grained values. 

In total, you are welcome to submit up to three variants, i.e. 6 files per language.

looking forward to the submissions, 

Chris