Important: Preparing for the evaluation phase
150 views
Skip to first unread message
Sapna Negi
Jan 1, 2019, 6:06:14 PM1/1/19
to SemEval 2019: Task 9
Hi all,
As we all gear up for the evaluation phase, here are a few important details and next steps.
1. Releasing trial test labels: We will be releasing the trial test dataset labels on January 5th. Therefore, we will be freezing the trial phase leaderboard that time onwards, and no trial phase submissions should be made thereafter. You can use the evaluation script provided on Github to calculate results outside CodaLab.
2. Declare your identity and team names: Your participation and submissions should be represented through your team.
- Create a team or add yourself to an existing team on Codalab before January 5th. You can use the ‘Team' tab for the same, eg. https://competitions.codalab.org/teams/19955/.
- Submission count limit applies at the team level, you will have to coordinate internally with your team members to ensure that the submission count from your team remains within the limit.
- All the participants are required to use their original names on CodaLab account with which they are registered/will register for the task. This can be added/edited through CodaLab account settings. Also, fill up the organisation/affiliation field in the form displayed during submission on CodaLab.
- All team members are required to register for the task before the evaluation phase starts. No additional members will be accepted thereafter.
3. Maximum submissions: Every team can make a maximum of 3 submissions for the evaluation phase, with a limit of only 2 submissions per day. Therefore, please do not plan to use all 3 tries on the last day.
4. Final submission: The best submission from your team will be regarded as the final submission. CodaLab will update the leaderboard with a new submission only when it beats the last submission.
5. Evaluation phase duration: We confirm that our evaluation phase will run from Jan 10th - Jan 24th
6. Leaderboard visibility: Leaderboard will be hidden for the duration of evaluation phase.
Please note that all the deadlines are scheduled at 23:59 hours GMT-12 for the given dates.
Regards
Organizers
As we all gear up for the evaluation phase, here are a few important details and next steps.
1. Releasing trial test labels: We will be releasing the trial test dataset labels on January 5th. Therefore, we will be freezing the trial phase leaderboard that time onwards, and no trial phase submissions should be made thereafter. You can use the evaluation script provided on Github to calculate results outside CodaLab.
2. Declare your identity and team names: Your participation and submissions should be represented through your team.
- Create a team or add yourself to an existing team on Codalab before January 5th. You can use the ‘Team' tab for the same, eg. https://competitions.codalab.org/teams/19955/.
- Submission count limit applies at the team level, you will have to coordinate internally with your team members to ensure that the submission count from your team remains within the limit.
- All the participants are required to use their original names on CodaLab account with which they are registered/will register for the task. This can be added/edited through CodaLab account settings. Also, fill up the organisation/affiliation field in the form displayed during submission on CodaLab.
- All team members are required to register for the task before the evaluation phase starts. No additional members will be accepted thereafter.
3. Maximum submissions: Every team can make a maximum of 3 submissions for the evaluation phase, with a limit of only 2 submissions per day. Therefore, please do not plan to use all 3 tries on the last day.
4. Final submission: The best submission from your team will be regarded as the final submission. CodaLab will update the leaderboard with a new submission only when it beats the last submission.
5. Evaluation phase duration: We confirm that our evaluation phase will run from Jan 10th - Jan 24th
6. Leaderboard visibility: Leaderboard will be hidden for the duration of evaluation phase.
Please note that all the deadlines are scheduled at 23:59 hours GMT-12 for the given dates.
Regards
Organizers
Tobias C
Jan 4, 2019, 1:32:27 AM1/4/19
to SemEval 2019: Task 9
Hi,
I have a question: After the trial data labels are released, is it allowed to use them as Training data for subtask A? And how about subtask B, since there should be performed a transfer learning?
Regards
Tobias
Sapna Negi
Jan 4, 2019, 5:43:13 AM1/4/19
to SemEval 2019: Task 9
Dear participants
We have come across some issues with each member having to register on CodaLab, and/or using their original names for the same.
Therefore, each team is required to fill this additional form by January 9th in order to register your teams for the task:
We will not accept your evaluation phase submission if you fail to do so by January 9th.
You still need to register your team on CodaLab.
In case you are participating in both the tasks, you will have to register on CodaLab separately for the two tasks.
Regards
Organizers
Sapna Negi
Jan 4, 2019, 6:21:51 AM1/4/19
to Tobias C, SemEval 2019: Task 9
Hi,
1. For subtask A, you can use it as the training data.
2. For subtask B, you cannot use any manually labeled training data from the same domain.
For your referral, it is clarified in the Terms and Conditions for the subtasks on CodaLab.
Please note that transfer learning is optional, we are not imposing any restrictions in terms of algorithms.
The main restriction for subtask B is to not use any hand labeled data from the same domain.
Regards
Organizers
--
You received this message because you are subscribed to the Google Groups "SemEval 2019: Task 9" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2019-ta...@googlegroups.com.
To post to this group, send email to semeval-2...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/semeval-2019-task-9/a636d36a-4b6c-481b-9551-69520b82929c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Sapna Negi
Jan 5, 2019, 10:47:40 AM1/5/19
to SemEval 2019: Task 9
Dear participants,
Trial test dataset labels are now available on GitHub,
No more trial submissions are allowed on CodaLab now onwards, till January 31st.
Regards
Organisers
Trial test dataset labels are now available on GitHub,
No more trial submissions are allowed on CodaLab now onwards, till January 31st.
Regards
Organisers
Sapna Negi
Jan 10, 2019, 6:57:54 AM1/10/19
to SemEval 2019: Task 9
Dear participants,
This is a reminder that you need to fill up the registration form in order to enter the evaluation phase. We will not accept any further entries after the evaluation data is released today.
This is our primary source of recognising the teams and mentioning them in the task description paper.
Please check your form again if filled already, and make the following corrections if needed:
1. Fill in your FULL NAMES where asked, not initials.
2. Provide these minimum details about your affiliation: Name of the organisation, Location
3. Paper title usually includes the team name, so please ensure that you are happy enough with your team name to be included in a formal publication.
Note that you are not obliged to travel and pay the registration fees for SemEval 2019 in order to include your system description paper in the proceedings as per the FAQs http://alt.qcri.org/semeval2019/index.php?id=frequently-asked-questions
Regards
Organisers
Sapna Negi
Jan 12, 2019, 4:38:33 PM1/12/19
to SemEval 2019: Task 9
Dear participants
The evaluation data is now available on GitHub, as SubtaskA_EvaluationData.csv and SubtaskB_EvaluationData.csv
The registration for official entry to SemEval2019 is closed now. Please contact us individually if you wish to make changes to your team and member names.
The evaluation phase will end on Jan 26th, 2019 (GMT-12)
Kindly be reminded of the evaluation phase conditions as described in the email below.
Regards
Organisers
On 1 Jan 2019, at 23:06, Sapna Negi <sapna...@insight-centre.org> wrote:
Reply all
Reply to author
Forward
0 new messages