Multiple runs for ablations

[Milan Straka]

Dear MultiLexNorm organizers,

we were wondering if it will be possible to submit several runs for the
sake of ablation experiments in the paper? Or will the test data be
released before the deadline of the paper?

If we knew that we can evaluate multiple systems in the paper, we can
train/keep around the suitable models (otherwise it would not have much
sense to train/keep "worse" models).

Thanks & cheers,
Milan Straka

[Rob v]

Dear Milan Straka,

We will allow for 2 submissions per team on the test data. More details on how to submit should follow very soon in this google groups (I aim for later today Copenhagen-time.)

The test data will be released after we have smoothened out all the issues with the submissions. This will probably be on time for you to use the data, as we will use codalab I do not expect too many issues.

However, I would be against doing (too many) ablation experiments on the test data, this is what dev should be used for (I can't stop you though ;) ). 

Best, 

Rob

[Rob v]

An example of a baseline submission have been added to the repo: https://bitbucket.org/robvanderg/multilexnorm/src/master/dev-eval/ 

The tryout (dev) phase codalab will be made public soon as well