Question about test results

[Zhengfei Ren]

Dear organizer, 

As I updated my test set on the post-evaluation phase and got some 0 scores, I just would like to check whether the results in our own test are effective for paper submission without the results from the leaderboard?  if we have seen some different results compared to the leaderboard, how do we quote those results instead of results on the leaderboard with an illustration? 

Thank you very much.

Best regards,

Zhengfei

Séanadh Ríomhphoist/
Email Disclaimer
Tá an ríomhphost seo agus aon chomhad a sheoltar leis faoi rún agus is lena úsáid ag an seolaí agus sin amháin é. Is féidir tuilleadh a léamh anseo. 
This e-mail and any files transmitted with it are confidential and are intended solely for use by the addressee. Read more here. 

[Mariia Fedorova]

Dear Zhengfei,

most likely all you need to fix zeros in the leaderboard is uploading the results for all three languages named track1_test_fi.tsv , track1_test_ru.tsv , track1_test_surprise.tsv (or track2... respectively) in a single *.zip file. Your current submissions contain Finnish only, and results on Finnish are non-zero.

Regarding the citation of the results in the paper, I believe, other organizers know better.

Best regards,

Maria

---

From: 'Zhengfei Ren' via AXOLOTL-24 <axolo...@googlegroups.com>
Sent: 13 May 2024 22:28:53
To: axolo...@googlegroups.com
Subject: [axolotl] Question about test results

 

--
You received this message because you are subscribed to the Google Groups "AXOLOTL-24" group.
To unsubscribe from this group and stop receiving emails from it, send an email to axolotl-24+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/axolotl-24/CABNWX4EcA1neS3fwXCMs0Ekg284cpDuW%3DoT5yBbXrTWH-s8F1Q%40mail.gmail.com.

[Zhengfei Ren]

Thanks a million! 

Mariia Fedorova <mar...@ifi.uio.no>于2024年5月13日 周一21:55写道：

To view this discussion on the web visit https://groups.google.com/d/msgid/axolotl-24/9eb2a00ace9e4c71a617a09f0a21b057%40ifi.uio.no.

[Andrey Kutuzov]

Dear Zhengfei,

If I understood you correctly, you want to mention your post-evaluation
phase results in your system description paper which you are going to
submit to the LChange'24 workshop.

This is perfectly fine, but of course, please mention in the text that
those are _post-evaluation_ results. They will not be mentioned in the
"official" AXOLOTL shared task description paper.

On 13.05.2024 22:56, 'Zhengfei Ren' via AXOLOTL-24 wrote:
> Thanks a million!
>

> Mariia Fedorova <mar...@ifi.uio.no <mailto:mar...@ifi.uio.no>>于2024年

> 5月13日 周一21:55写道：
>
> Dear Zhengfei,
>
> most likely all you need to fix zeros in the leaderboard is
> uploading the results for all three languages named
> track1_test_fi.tsv , track1_test_ru.tsv , track1_test_surprise.tsv
> (or track2... respectively) in a single *.zip file. Your current
> submissions contain Finnish only, and results on Finnish are non-zero.
>
> Regarding the citation of the results in the paper, I believe, other
> organizers know better.
>
> Best regards,
> Maria
>

> ------------------------------------------------------------------------
> *From:* 'Zhengfei Ren' via AXOLOTL-24 <axolo...@googlegroups.com
> <mailto:axolo...@googlegroups.com>>
> *Sent:* 13 May 2024 22:28:53
> *To:* axolo...@googlegroups.com <mailto:axolo...@googlegroups.com>
> *Subject:* [axolotl] Question about test results

> Dear organizer,
>
> As I updated my test set on the post-evaluation phase and got some 0
> scores, I just would like to check whether the results in our own
> test are effective for paper submission without the results from the
> leaderboard?  if we have seen some different results compared to the
> leaderboard, how do we quote those results instead of results on the
> leaderboard with an illustration?
>
> Thank you very much.
>
> Best regards,
> Zhengfei
>

> /
> /
>
> *Séanadh Ríomhphoist/*
>
> //
>
> *Email Disclaimer*
>
> /
>
> /
>
> /

>
> Tá an ríomhphost seo agus aon chomhad a sheoltar leis faoi rún agus
> is lena úsáid ag an seolaí agus sin amháin é. Is féidir tuilleadh a
> léamh anseo.

> <https://www.dcu.ie/iss/seanadh-riomhphoist.shtml><https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>
>
> /
>
> /
>
> /This e-mail and any files transmitted with it are confidential and

> are intended solely for use by the addressee. Read more here.

> <https://www.dcu.ie/iss/email-disclaimer.shtml> /
>
> ///

>
> --
> You received this message because you are subscribed to the Google
> Groups "AXOLOTL-24" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to axolotl-24+...@googlegroups.com

> <mailto:axolotl-24+...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/axolotl-24/CABNWX4EcA1neS3fwXCMs0Ekg284cpDuW%3DoT5yBbXrTWH-s8F1Q%40mail.gmail.com <https://groups.google.com/d/msgid/axolotl-24/CABNWX4EcA1neS3fwXCMs0Ekg284cpDuW%3DoT5yBbXrTWH-s8F1Q%40mail.gmail.com?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "AXOLOTL-24" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to axolotl-24+...@googlegroups.com

> <mailto:axolotl-24+...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/axolotl-24/9eb2a00ace9e4c71a617a09f0a21b057%40ifi.uio.no <https://groups.google.com/d/msgid/axolotl-24/9eb2a00ace9e4c71a617a09f0a21b057%40ifi.uio.no?utm_medium=email&utm_source=footer>.
>
>
> /
> /
>
> *Séanadh Ríomhphoist/*
>
> //
>
> *Email Disclaimer*
>
> /
>
> /
>
> /

>
> Tá an ríomhphost seo agus aon chomhad a sheoltar leis faoi rún agus is
> lena úsáid ag an seolaí agus sin amháin é. Is féidir tuilleadh a léamh
> anseo.

> <https://www.dcu.ie/iss/seanadh-riomhphoist.shtml><https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>
>
> /
>
> /
>
> /This e-mail and any files transmitted with it are confidential and are

> intended solely for use by the addressee. Read more here.

> <https://www.dcu.ie/iss/email-disclaimer.shtml> /
>
> ///

>
> --
> You received this message because you are subscribed to the Google
> Groups "AXOLOTL-24" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to axolotl-24+...@googlegroups.com

> <mailto:axolotl-24+...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/axolotl-24/CABNWX4E%3DqZxopNPJ4mW5h74KHs662SNjfxguO7%2BpYrTFCMvX3Q%40mail.gmail.com <https://groups.google.com/d/msgid/axolotl-24/CABNWX4E%3DqZxopNPJ4mW5h74KHs662SNjfxguO7%2BpYrTFCMvX3Q%40mail.gmail.com?utm_medium=email&utm_source=footer>.

--
Andrey
Language Technology Group (LTG)
University of Oslo

[Zhengfei Ren]

Dear  Andrey,

Thank you very much for your reply.

Best regards,

Zhengfei

To unsubscribe from this group and stop receiving emails from it, send an email to axolotl-24+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/axolotl-24/96d9fc96-ec1a-41ef-b1f4-1535eee3e4d1%40ifi.uio.no.

Séanadh Ríomhphoist/
Email Disclaimer

Tá an ríomhphost seo agus aon chomhad a sheoltar leis faoi rún agus is lena úsáid ag an seolaí agus sin amháin é. Is féidir tuilleadh a léamh anseo. 

This e-mail and any files transmitted with it are confidential and are intended solely for use by the addressee. Read more here. 

[Zhengfei Ren]

Dear organizer,

I just want to check the post-evaluations task 1's leaderboard can not be worked on for submission showing an unexpected error occurred. Can you solve this problem?

Thank you.

best regards,

Zhengfei

[Mariia Fedorova]

Dear Zhengfei, the post-evaluation task 1's leaderboard looks fine, regarding the task 2's leaderboard, your last submission contains a single file named track2_test_ge.tsv, while the allowed names are track1_test_fi.tsv , track1_test_ru.tsv , track1_test_surprise.tsv (or track2... respectively) . German was our surprise language, that's why it is so.

Best regards,

Maria

---

From: 'Zhengfei Ren' via AXOLOTL-24 <axolo...@googlegroups.com>
Sent: 16 May 2024 01:23:21
To: axolo...@googlegroups.com
Subject: Re: [axolotl] Question about test results

 

To view this discussion on the web visit https://groups.google.com/d/msgid/axolotl-24/CABNWX4GoHQO%2BZFg3RB-b0EVt%2BpxBN7wgum%3DabOnOW%3DZg%3DOSRvQ%40mail.gmail.com.

[Zhengfei Ren]

Dear organizer, 

Thank you for your email. I was trying to upload several times with another account, it didn't work again when I upload the 11st file , the error with the unexpected error occurred has shown. 

Can you please check that? The account was zhengf...@adaptcentre.ie

Best regards,

Zhengfei

To view this discussion on the web visit https://groups.google.com/d/msgid/axolotl-24/71b2aacdaa8b4a3db5946d9f2a3f8ef5%40ifi.uio.no.

[Mariia Fedorova]

Dear Zhengfei,

We don't have an account with this email, the username zhengfei666 has an email nmg...@163.com, but it shouldn't be the cause of the problem anyway.

We see only 10 submissions from zhengfei666 and the 10th is fine (it is currently on the leaderboard). So it seems the error happened earlier than our scoring program began to run and we don't have access to the logs, it seems to be a Codalab issue. Maybe you can send us your submission and we will have a look at it?

Best regards,

Maria

---

From: 'Zhengfei Ren' via AXOLOTL-24 <axolo...@googlegroups.com>

Sent: 16 May 2024 19:06:00

To view this discussion on the web visit https://groups.google.com/d/msgid/axolotl-24/CABNWX4GOHzFis3CKNP-YZFPY1xWbmOxRe4JxQBw3Xc4Ys6hS9A%40mail.gmail.com.

[Zhengfei Ren]

Dear Maria, 

Sure, thanks for the reply. I will send a quick test result for the submission, by the way, the 11th file can not be uploaded. 

Thank you very much. 

Best regards,

Zhengfei

To view this discussion on the web visit https://groups.google.com/d/msgid/axolotl-24/3af250044a5543f386c94b2b8d655c26%40ifi.uio.no.

[Mariia Fedorova]

Dear Zhengfei,

the problem with the zip you sent was that it contained a redundant folder inside, so the path to the tsv after decompressing was "Test set for submission/track1_test_fi.tsv" and our scoring program couldn't find it. I've tried to compress the tsv only (see attached) and it works.

Best regards,
Maria

---

From: 'Zhengfei Ren' via AXOLOTL-24 <axolo...@googlegroups.com>

Sent: 17 May 2024 01:42:49

To view this discussion on the web visit https://groups.google.com/d/msgid/axolotl-24/CABNWX4EGAdUuPGs_UOgm1%2B1EdGnUmS8GWS%2B3cBqyDrAwGj6XRA%40mail.gmail.com.

[Zhengfei Ren]

Dear Organizer,

I just tried with several submissions for the new folder with the requirements, but it seems still not working. Are there any other methods for submission ? Or maybe we could work with the evaluation scripts by ourselves?

Thank you very much.

Best regards,

Zhengfei

To view this discussion on the web visit https://groups.google.com/d/msgid/axolotl-24/4c9d2e07f4884a15921cecb191dd7dbe%40ifi.uio.no.

[Mariia Fedorova]

Dear Zhengfei,

I also noticed there was a submission named '...test...' in the competition instance that is not post-evaluation. There everything is zeros because that instance accepts predictions for the test subset only.

I believe that given that the paper submission deadline is today AOE, it is really better for you to run our evaluation script on your own  (without codalab) and report those results. And after that there will be time to figure out what's wrong with codalab and change the scores for the camera ready version, if they will be different.

Best regards,

Maria

---

From: 'Zhengfei Ren' via AXOLOTL-24 <axolo...@googlegroups.com>

Sent: 17 May 2024 16:14:31

To view this discussion on the web visit https://groups.google.com/d/msgid/axolotl-24/CABNWX4GQMqa%2Bh13g621LB0FJRRDjfNvBr4598aLHMUBQn4gdJQ%40mail.gmail.com.

[Mariia Fedorova]

There was a typo in the previous message, the second sentence was meant to be "There everything is zeros because that instance accepts predictions for the DEV subset only. "

---

From: Mariia Fedorova
Sent: 17 May 2024 20:02:19

[Zhengfei Ren]

Dear Organizer, 

Sure, thank you very much. I will do the correct evaluation in the carame read version. currently just based on the local machine.

Best wishes,

Zhenhgfei

To view this discussion on the web visit https://groups.google.com/d/msgid/axolotl-24/fdd118de034442bc8dec4a66478905a3%40ifi.uio.no.