WNUT Data release and Evaluation period starts!

[Leon Derczynski]

Hi all,

Thanks so much for your patience this far. We're ready to go!

On the shared task page, you can now download new items:

1. a README describing the entity types
   
2. development data, with BIO labels
   
3. test data, with no labels
   

In addition, the timeline is set! It's compact, so please be mindful of this. Papers should be around four pages in length, EMNLP format; formal details will go up soon.

• Test data released: 21 June 2017
  
• Result submission: 28 June 2017
  
• Shared-task results and gold annotations for test data: 30 June 2017
  
• System Description Papers Due: 7 July 2017
  
• Reviews Returned: 15 July 2017
  
• Camera Ready Deadline: 19 July 2017
  
• Workshop Date: Sep 7 2017
  

Good luck! The data is tough and dirty, and the scoring is aggressive, so please bring your best tools to the table. The test set draws from a mixture of sources: Twitter, Reddit and Stack Overflow. The development set draws from YouTube comments. Note that, as with most user-generated text, there is vulgar language in some of this data. All deadlines are in the latest timezone possible, so UTC+12. You have one week.

All the best,

Leon Eric Marieke and Nut

[pankaj gupta]

Hi Leon,

It would be really great if you could to release baseline scores on development set ?

Regards

Pankaj

[patrick.chri...@gmail.com]

Hi,

One remark regarding the dev data labels, there seems to be a few wrong tags, namely 'b-person' (lower case b), 'P-person', 'S-person' and 'B-people'.

Regards,
Patrick Jansson

[benjamin.heinzerling]

Looks like there are only a few of these:

$ grep -f <(comm -3 <(cut -f2 emerging.dev.conll | sort | uniq) <(cut -f2 wnut17train.conll | sort | uniq)) emerging.dev.conll

Bollywood       B-place
Ireland B-place
klay    P-person
AirPods B-people
trump   S-person
ldshadowlady    b-person


These should probably be:

Bollywood       B-location
Ireland B-location
klay    B-person
AirPods B-product
trump   B-person
ldshadowlady    B-person

[benjamin.heinzerling]

Not Leon, but here are some results on the dev set with a run-of-the-mill 3-layer LSTM and GoogleNews-vectors-negative300.bin embeddings:

surface forms precision:  55.90%; recall:  87.55%; FB1:  68.24
processed 15733 tokens with 836 phrases; found: 495 phrases; correct: 296.
accuracy:  94.18%; precision:  59.80%; recall:  35.41%; FB1:  44.48

[Leon Derczynski]

Thank you, Benjamin, for sharing these figures!

The dev data has now been updated on the site, fixing the tag issues.

--
You received this message because you are subscribed to the Google Groups "Workshop on Noisy User-generated Text (WNUT)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wnut+unsubscribe@googlegroups.com.
To post to this group, send email to wn...@googlegroups.com.
Visit this group at https://groups.google.com/group/wnut.

To view this discussion on the web visit https://groups.google.com/d/msgid/wnut/c36cf6f8-e93e-4499-a7ba-a45b46493c66%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Research Fellow, Department of Computer Science

University of Sheffield, UK

http://www.derczynski.com/

http://twitter.com/leonderczynski