Subtask 3 results and paper submission instructions

[Sasha Spala]

Hello all,

 

Thank you to everyone who participated in DeftEval! I am including the final results of Subtask 3 below, and will add it to our Github and Codalab landing page.

 

 

Now that we have finished the evaluation period, we want to provide some instructions on paper submissions:

 

The paper submission deadline is 17 April 2020. System description papers may be up to 4 pages if you are submitting for only one subtask, and up to 6 pages if you are submitting for multiple subtasks. References do not count towards the page limit. Folks who have produced two significantly different systems for each subtask they participated in should submit separate papers for each significantly different system. System description papers should include steps for replication and should focus on the system approach rather than details about the task, which will be included in the task description paper.

 

Full details on submissions are available here, and the paper submission site is available here.

 

If you choose to submit to the SemEval conference, you will also be responsible for helping to review other papers between 17 April 2020 and 15 May 2020.

 

Submission decisions will be sent out by the SemEval co-chairs on June 10^th. Camera ready papers are due July 8^th.

 

Test data will be released this week, as well as access to the Codalab post-evaluation phase, where you can upload new results against the actual test data. We encourage you still to treat test data appropriately, and to only use test data at test time. We also encourage submissions from those who may have scored lower on the leaderboard but had architecturally interesting designs or otherwise technically significant systems. Remember also that the DEFT corpus is a work in progress, and the SemEval tasks we ran this year are the first of their kind – you may consider these things in context with the performance of your model. Critical analysis of models in the context of a newly designed corpus and novel metric is equally as important as model performance on the leaderboard.

 

Please feel free to send us your questions and to continue to use the google groups.

 

Best,
Sasha