Is Task 1 evaluation over?

80 views
Skip to first unread message

Talha Anwar

unread,
Feb 19, 2020, 2:33:00 AM2/19/20
to DeftEval 2020
Hi, Greetings Everyone. I believe the Task 1 and 2 evaluation start from 19 feb. But when i click on subtask 1 evaluation, it says 'This phase of the competition is closed. Here are the submissions you made'. 
Seems i am missing some thing
'

Sasha Spala

unread,
Feb 19, 2020, 11:17:02 AM2/19/20
to Talha Anwar, DeftEval 2020

Hi Talha,

Task one began Feb 19 at midnight, so it has just begun. It seems that Codalab doesn’t allow us to have concurrent phases, so I have updated our Codalab page so that you should be able to submit both Subtask 1 and Subtask 2 in the same phase tab since they have the same evaluation dates. Unfortunately Codalab also won’t let me remove a phase, so the original Subtask 1 tab will remain visible on the site. Let me know if you have any questions or if I can clarify further.

Best,
Sasha

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/6f4d2da0-5daf-4584-9916-bc552cef76b1%40googlegroups.com.

Адис Давлетов

unread,
Feb 19, 2020, 3:40:13 PM2/19/20
to DeftEval 2020
Hi, Sasha

Im constantly getting "Failed, competition phase is being migrated, please try again in a few minutes" for last two hours. Is it the problem only I have?

Best, Adis

среда, 19 февраля 2020 г., 19:17:02 UTC+3 пользователь Sasha Spala написал:

Hi Talha,

Task one began Feb 19 at midnight, so it has just begun. It seems that Codalab doesn’t allow us to have concurrent phases, so I have updated our Codalab page so that you should be able to submit both Subtask 1 and Subtask 2 in the same phase tab since they have the same evaluation dates. Unfortunately Codalab also won’t let me remove a phase, so the original Subtask 1 tab will remain visible on the site. Let me know if you have any questions or if I can clarify further.

Best,
Sasha

 

From: <semeval-202...@googlegroups.com> on behalf of Talha Anwar <chtalh...@gmail.com>
Date: Wednesday, February 19, 2020 at 2:33 AM
To: DeftEval 2020 <semeval-202...@googlegroups.com>
Subject: Is Task 1 evaluation over?

 

Hi, Greetings Everyone. I believe the Task 1 and 2 evaluation start from 19 feb. But when i click on subtask 1 evaluation, it says 'This phase of the competition is closed. Here are the submissions you made'. 

Seems i am missing some thing

'

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.

Sasha Spala

unread,
Feb 19, 2020, 4:06:11 PM2/19/20
to Адис Давлетов, DeftEval 2020

Hmm…. I’ve never heard that error before. I’m not totally sure how Codalab allocates resources for evaluation, but I have experienced times where it takes quite a long time (hours) to get my submission past the “running” phase. What tab are you submitting to? Subtask 1 & 2 evaluation or the All tasks: Training tab?

 

My best advice would be to give it a day and check back to see if you’re still getting the same error. If it doesn’t work then we can open a ticket with Codalab to see what’s up. You may also want to check their Github issues tracker to see if someone’s already logged the same ticket.

Best,
Sasha

 

From: <semeval-202...@googlegroups.com> on behalf of Адис Давлетов <adavl...@gmail.com>
Date: Wednesday, February 19, 2020 at 3:40 PM
To: DeftEval 2020 <semeval-202...@googlegroups.com>
Subject: Re: Is Task 1 evaluation over?

 

Hi, Sasha

 

Im constantly getting "Failed, competition phase is being migrated, please try again in a few minutes" for last two hours. Is it the problem only I have?

 

Best, Adis

среда, 19 февраля 2020 г., 19:17:02 UTC+3 пользователь Sasha Spala написал:

Hi Talha,

Task one began Feb 19 at midnight, so it has just begun. It seems that Codalab doesn’t allow us to have concurrent phases, so I have updated our Codalab page so that you should be able to submit both Subtask 1 and Subtask 2 in the same phase tab since they have the same evaluation dates. Unfortunately Codalab also won’t let me remove a phase, so the original Subtask 1 tab will remain visible on the site. Let me know if you have any questions or if I can clarify further.

Best,
Sasha

 

From: <semeval-202...@googlegroups.com> on behalf of Talha Anwar <chtalh...@gmail.com>
Date: Wednesday, February 19, 2020 at 2:33 AM
To: DeftEval 2020 <semeval-202...@googlegroups.com>
Subject: Is Task 1 evaluation over?

 

Hi, Greetings Everyone. I believe the Task 1 and 2 evaluation start from 19 feb. But when i click on subtask 1 evaluation, it says 'This phase of the competition is closed. Here are the submissions you made'. 

Seems i am missing some thing

'

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.

--

You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/356e426c-ca20-4182-a456-1f889fbff383%40googlegroups.com.

Адис Давлетов

unread,
Feb 19, 2020, 4:30:04 PM2/19/20
to DeftEval 2020
Im submitting to Subtask 1 & 2 evaluation and getting that error. And Im submitting zip archive with only task1 files from test set. And now I tried to submit the same archive to  All tasks: Training tab and got following:

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Traceback (most recent call last):
  File "/tmp/codalab/tmpyEmGcZ/run/program/semeval2020_06_evaluation_main.py", line 83, in 
    main(cfg)
  File "/tmp/codalab/tmpyEmGcZ/run/program/semeval2020_06_evaluation_main.py", line 62, in main
    task_1_report = task_1_eval_main(ref_path, res_path, output_dir, cfg['task_1']['eval_labels'])
  File "/tmp/codalab/tmpyEmGcZ/run/program/evaluation_sub1.py", line 183, in task_1_eval_main
    temp_y_gold, temp_y_pred = get_gold_and_pred_labels(ref_path.joinpath(child.name), child)
  File "/tmp/codalab/tmpyEmGcZ/run/program/evaluation_sub1.py", line 116, in get_gold_and_pred_labels
    validate_data(gold_rows, pred_rows)
  File "/tmp/codalab/tmpyEmGcZ/run/program/evaluation_sub1.py", line 96, in validate_data
    validate_sents(gold_rows, pred_rows)
  File "/tmp/codalab/tmpyEmGcZ/run/program/evaluation_sub1.py", line 68, in validate_sents
    raise ValueError("Sentence mismatch row {}: Pred {} Gold {}".format(row_index, pred_sent, gold_sent))
ValueError: Sentence mismatch row 0: Pred 3321 . A characteristic like cleft chin , for example , is influenced by a single gene from each parent . Gold  3321 . A characteristic like cleft chin , for example , is influenced by a single gene from each parent .

четверг, 20 февраля 2020 г., 0:06:11 UTC+3 пользователь Sasha Spala написал:

Hmm…. I’ve never heard that error before. I’m not totally sure how Codalab allocates resources for evaluation, but I have experienced times where it takes quite a long time (hours) to get my submission past the “running” phase. What tab are you submitting to? Subtask 1 & 2 evaluation or the All tasks: Training tab?

 

My best advice would be to give it a day and check back to see if you’re still getting the same error. If it doesn’t work then we can open a ticket with Codalab to see what’s up. You may also want to check their Github issues tracker to see if someone’s already logged the same ticket.

Best,
Sasha

 

From: <semeval-202...@googlegroups.com> on behalf of Адис Давлетов <adavl...@gmail.com>
Date: Wednesday, February 19, 2020 at 3:40 PM
To: DeftEval 2020 <semeval-202...@googlegroups.com>
Subject: Re: Is Task 1 evaluation over?

 

Hi, Sasha

 

Im constantly getting "Failed, competition phase is being migrated, please try again in a few minutes" for last two hours. Is it the problem only I have?

 

Best, Adis

среда, 19 февраля 2020 г., 19:17:02 UTC+3 пользователь Sasha Spala написал:

Hi Talha,

Task one began Feb 19 at midnight, so it has just begun. It seems that Codalab doesn’t allow us to have concurrent phases, so I have updated our Codalab page so that you should be able to submit both Subtask 1 and Subtask 2 in the same phase tab since they have the same evaluation dates. Unfortunately Codalab also won’t let me remove a phase, so the original Subtask 1 tab will remain visible on the site. Let me know if you have any questions or if I can clarify further.

Best,
Sasha

 

From: <semeval-202...@googlegroups.com> on behalf of Talha Anwar <chtalh...@gmail.com>
Date: Wednesday, February 19, 2020 at 2:33 AM
To: DeftEval 2020 <semeval-202...@googlegroups.com>
Subject: Is Task 1 evaluation over?

 

Hi, Greetings Everyone. I believe the Task 1 and 2 evaluation start from 19 feb. But when i click on subtask 1 evaluation, it says 'This phase of the competition is closed. Here are the submissions you made'. 

Seems i am missing some thing

'

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.

Sasha Spala

unread,
Feb 19, 2020, 4:54:18 PM2/19/20
to Адис Давлетов, DeftEval 2020

Looks like you might be missing a whitespace at the beginning of your line. It helps to look at the diff between the bare test files and the results files you’re trying to run. They should exactly match the bare files with the exception that you are adding a tab delimiter and your 0 or 1 prediction to the end of each line.

 

Make sure also that when you submit to the All tasks: Training tab you’re evaluating on the dev data. You can only evaluate test data in the evaluation tab and it will be considered an official submission.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.

--

You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/f2a2929d-3774-4d8b-90be-2d6d884faee4%40googlegroups.com.

Адис Давлетов

unread,
Feb 19, 2020, 5:18:05 PM2/19/20
to DeftEval 2020
Thanks, it seems the leading whitespace was the problem of the Sentence Mismatch error. Im more curious about the example which is in test and dev sets simultaneously.

Still there is submission error in tab Subtask 1 & 2 evaluation. Will attempt tomorrow.

Best, Adis

четверг, 20 февраля 2020 г., 0:54:18 UTC+3 пользователь Sasha Spala написал:

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.

Sasha Spala

unread,
Feb 21, 2020, 9:56:20 AM2/21/20
to Адис Давлетов, DeftEval 2020

I took a look around the Codalab github and I may have found a solution on our end to the "Failed, competition phase is being migrated, please try again in a few minutes” error. Please let me know if you still encounter it.

 

 

Im more curious about the example which is in test and dev sets simultaneously.

Can you clarify what you mean by this?

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.

--

You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/f1c317b8-5494-4795-bd68-2bd03362e222%40googlegroups.com.

Адис Давлетов

unread,
Feb 21, 2020, 10:07:11 AM2/21/20
to DeftEval 2020
There are only 5 submissions for test set and I don't want to waste them on testing if there is an error.

"Im more curious about the example which is in test and dev sets simultaneously."
Maybe I got it wrong, but in the following error encountered when I submitted the solution for the TEST set 
in the tab for the DEV set (All tasks training) there is Sentence mismatch row 0: Pred 3321 . A characteristic like cleft chin , for example , is influenced by a single gene from each parent . Gold 3321 . A characteristic like cleft chin , for example , is influenced by a single gene from each parent .
So I assumed that there is non empty intersection between test and dev examples for subtask 1.

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Traceback (most recent call last):
  File "/tmp/codalab/tmpyEmGcZ/run/program/semeval2020_06_evaluation_main.py", line 83, in 
    main(cfg)
  File "/tmp/codalab/tmpyEmGcZ/run/program/semeval2020_06_evaluation_main.py", line 62, in main
    task_1_report = task_1_eval_main(ref_path, res_path, output_dir, cfg['task_1']['eval_labels'])
  File "/tmp/codalab/tmpyEmGcZ/run/program/evaluation_sub1.py", line 183, in task_1_eval_main
    temp_y_gold, temp_y_pred = get_gold_and_pred_labels(ref_path.joinpath(child.name), child)
  File "/tmp/codalab/tmpyEmGcZ/run/program/evaluation_sub1.py", line 116, in get_gold_and_pred_labels
    validate_data(gold_rows, pred_rows)
  File "/tmp/codalab/tmpyEmGcZ/run/program/evaluation_sub1.py", line 96, in validate_data
    validate_sents(gold_rows, pred_rows)
  File "/tmp/codalab/tmpyEmGcZ/run/program/evaluation_sub1.py", line 68, in validate_sents
    raise ValueError("Sentence mismatch row {}: Pred {} Gold {}".format(row_index, pred_sent, gold_sent))
ValueError: Sentence mismatch row 0: Pred 3321 . A characteristic like cleft chin , for example , is influenced by a single gene from each parent . Gold  3321 . A characteristic like cleft chin , for example , is influenced by a single gene from each parent .

пятница, 21 февраля 2020 г., 17:56:20 UTC+3 пользователь Sasha Spala написал:

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.

Sasha Spala

unread,
Feb 21, 2020, 10:35:18 AM2/21/20
to Адис Давлетов, DeftEval 2020

Ah, I see. Context window 3321 is not in the test set, only in dev, so I’m not sure if you possibly were submitting the dev data or somehow your sentences are mixed together. You can validate this by looking for 3321 in the test set here: https://github.com/adobe-research/deft_corpus/tree/master/data/test_files/subtask_1

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.

--

You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/38c6f0a5-2331-4701-9311-9e386ed5857e%40googlegroups.com.

Адис Давлетов

unread,
Feb 21, 2020, 5:24:48 PM2/21/20
to DeftEval 2020
Thanks, indeed I forgot to delete dev files from predictions folder, so there was a mix of dev and test!

пятница, 21 февраля 2020 г., 18:35:18 UTC+3 пользователь Sasha Spala написал:

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.

Reply all
Reply to author
Forward
0 new messages