Approaching Deadlines: Please Read

[Harish Tayyar Madabushi]

Dear all, 

This message contains important information including deadlines and instructions regarding the end of this competition. Please read it carefully and in full. If you have any questions, do not hesitate to ask either on Google Groups or by emailing me. 

Competition Ends: 31st of January 2022, 11:59pm UTC. 

This is the deadline to submit your results on the test set to CodaLab (make sure you submit to the "Evaluation Phase") 

If there are problems with your submission I will get in touch with you on the email you provide on CodaLab from my Sheffield email id (h.tayyar...@sheffield.ac.uk), so please add it to your address book so it doesn't end up in spam, and check your spam folders regularly until the results are released.

It's great to see so many submissions on the Evaluation phase! If you're yet to submit, there is still time but do keep an eye on the deadline! 

New Information: 

1. Please update your CodaLab accounts with a team name - you can do this by clicking on your name (right top) --> Settings and updating "Team name" under the "Competition Settings" option. If you do not set a team name, we will use your CodaLab username as the team name when we release the results. 
2. Please note that your team name will be part of the title of your paper - title format is: "<Team Name> at SemEval 2022 Task2: <title of paper here>". 
3. Please consider adding a description to each of your submissions on the Evaluation set - this will help you remember which configuration led to what score when we release rankings. You do NOT have to resubmit for this, you can do this anytime by clicking on the "+" next to an existing submission and "update description". 
4. ALL participants who submit predictions on the TEST set to the Evaluation Phase are invited to write a paper. The deadline to submit your paper is February 23, 2022. You are NOT required to present the paper for it to appear in the proceedings and there is NO fee to submit a paper. You are invited to submit a paper regardless of your performance (rank). Your description might offer insights into what does (and equally importantly what does not work) for this task. Read more about the Paper Submission Requirements and Guidelines for SemEval System Papers. 
5. As soon as the current phase ends, CodaLab will automatically start Phase 3 and we cannot control this.  However, the new phase will not process submissions until well after the competition results are released. 

Reminders:

1. The practice phase of the task (which has a visible leaderboard so you can improve your models) will be closed at 8pm UTC on the 30th of January. This is to make sure that there is no confusion on where to submit your final results on the last day of the competition. You can continue to improve your models on the development set for which the gold labels are provided (do keep an eye on the competition deadline though). 
2. The deadline to register your team using this Google Form is the 30th of January 2022 11:59pm UTC. If you make a mistake on the form, simply submit it again and I'll consider the last submission as final. 
3. The deadline to submit your results on the test set to CodaLab is the 31st of January 2022, 11:59pm UTC. 
4. You can submit up to 5 runs for each subtask during the evaluation period. Your best score across the five submissions will be used in the rankings. Failed submissions are not counted towards this 5. If you want me to delete a submission, please email me clearly specifying your CodaLab username and the data/time of the submission you want deleted and I can do that for you (don't forget to resubmit the deleted runs and please don't leave this to the last minute). 
   
5. If there is a problem with CodaLab just before the deadline (and you cannot submit there for some reason), please email me your submission (exactly as you would upload it) to "h.tayyar...@sheffield.ac.uk" with the subject "Submission: SemEval 2022 Task 2 <your team name>". This does not extend the deadline, I must receive your email before 11:59pm UTC on the 31st of January 2022.
   
6. If the status of your submission is "Finished", then your submission has been successfully submitted and (if you've submitted to the evaluation phase) will be considered for the evaluation phase of the competition. If your submission has errors, I will email you flagging this error.
   
7. Please remember that the baselines that we have provided are particularly strong benchmarks using very powerful pre-trained language models (mBERT). This is unlike typical baselines that tend to be weak. As such you should not be worried if your scores are below the baseline/benchmark and you should still aim to submit a paper.
   
8. Make sure you submit predictions on the correct data split to the correct phase (more here):
           1. dev.csv --> Do NOT submit to CodaLab, use associated gold labels and evaluation script.
           2. eval.csv --> "Practice Phase" (Leaderboard visible, used to tune your model)
           3. test.csv --> "Evaluation Phase" (Leaderboard NOT visible, used to rank teams on this task).
   
9. Your submission must consist of a zip file of a  folder called "submission" containing a separate file for each subtask (task2_subtaska.csv and task2_subtaskb.csv).
   

Best of luck! 

Best wishes, 

Harish

[Harish Tayyar Madabushi]

Dear all, 

Thank you for the active participation on the practice phase. The results look very interesting and I for one am looking forward to seeing what the results on the Evaluation phase will look like. 

Please note that the practice phase is now closed. 

Next approaching deadline: You have until midnight tonight (UTC) to register your team for the competition: https://forms.gle/ZQj9T5Zx7NF4okZ9A

The Evaluation phase will end tomorrow midnight (UTC). 

As always, don't hesitate to get in touch should you have any questions. 

Best

Harish