Information about system description papers

[Milan Straka]

Dear MultiLexNorm organizers,

if I may, I have several questions regarding the system description
paper:

- what will be the page limit (I would assume a long EMNLP paper style)?

- should we use the post-competition phase on CodaLab for ablation
measurements on the test set ? (Yes, we will use a reasonable amount ;-)

- will it be possible to also run ablation experiments using the
extrinsic evaluation (it may be interesting for some settings)?

- would it be possible to get MoNoise predictions on the test set? It
might be interesting to inspect the differences with our system
and see if we can eyeball something out. Alternatively, did the
baseline used https://bitbucket.org/robvanderg/monoise/src/master/
and the models at http://www.itu.dk/people/robv/data/monoise/
so that we can gen them ourselves?

- also, our system is _extremely_ slow (I cannot stress it enough ;-),
so it would be fair to include runtime performance comparison with
MoNoise -- do you have an idea about MoNoise speed, or is it fine
if we again use the above linked version of MoNoise to perform the
measurements ourselves?

Thanks & cheers,
Milan Straka

[robvanderg]

Dear Milan Straka, 

Thanks for your questions:

- page limit is 8 pages

- not sure about "should", but "could" is now possible, as the test data (with labels) has been pushed to the repo

- I plan to upload the code/models for the extrinsic evaluation later today. However, unfortunately I do not have time to thoroughly refactor it.For the Turkish treebank, you would need to sign an agreement: http://tools.nlp.itu.edu.tr/Datasets

- All submissions are shared on the repo now. The new MoNoise models and the code are now available publicly as well: https://bitbucket.org/robvanderg/monoise/src/master/ including the models and the commands used to train them on: http://www.itu.dk/people/robv/data/monoise/  (code-switched models are in the first language of the pair). I have also included all scripts I have used to get the models (not much tuning is done): https://bitbucket.org/robvanderg/monoise/src/master/scripts/sharedTask/

- MoNoise is not fast either, but it will do ~30 words per second. Section 6 of https://aclanthology.org/P19-3032.pdf was generated on a relatively standard machine (cpu), and the model has not changed since then. For a completely fair comparison I would suggest to run it on the same machine though.

Best, 

Rob

Op maandag 6 september 2021 om 12:57:23 UTC+2 schreef str...@ufal.mff.cuni.cz:

[Milan Straka]

Hi Rob,

thanks for all your quick answers and the extrinsic evaluation results :-)

Note that given the minor differences in LAS, we will not include
extrinsic evaluation in ablations, so (at least from our perspective)
there is no need to have the extrinsic evaluation models soon.

Thanks for all your work,
cheers,
Milan Straka

> -----Original message-----
> From: robvanderg <robva...@live.nl>
> Sent: 8 Sep 2021, 03:58
>
> Updated results are including extrinsic evaluation are attached. Ranking of
> the teams seems similar, but interestingly MFR ranks much higher in
> comparison to ERR

> --
> You received this message because you are subscribed to the Google Groups "MultiLexNorm" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to multilexnorm...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/multilexnorm/4ddfde6c-21df-4606-a77d-cc9135de6069n%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[David Samuel]

Hi Rob,

Thank you very much for your answers!

Could you please increase the submission limit in CodaLab, so that we can evaluate the ablation experiments? Not a big issue if that is not possible, it would just be easier and less error-prone than manual evaluation :)

Thanks,

David Samuel

[robvanderg]

Hi David, 

I set it to 5, hope that that's enough? don't want to encourage having too many runs on it

Best, 

Op woensdag 8 september 2021 om 14:25:17 UTC+2 schreef davda...@gmail.com: