Second Call for Participation - Shared Task on Automatic Identification of Verbal Multiword Expressions
Veronika Vincze
**SECOND CALL FOR PARTICIPATION**
Shared task on automatic identification of verbal multiword expressions
http://multiword.sourceforge.net/sharedtask2017
=======================================================================
*Apologies for cross-posting*
The PARSEME shared task on automatic identification of verbal multiword expressions (VMWEs) aims at identifying verbal MWEs in running texts. Verbal MWEs include idioms (*to let the cat out of the bag*), light verb constructions (*to make a decision*), verb-particle constructions (*to give up*), and inherently reflexive verbs (*se suicider* 'to suicide' in French). Their identification is a well-known challenge for NLP applications, due to their complex characteristics: discontinuity, non-compositionality, heterogeneity and syntactic variability.
The shared task is highly multilingual: we intend to cover up to 20 languages from as many countries. PARSEME members have elaborated annotation guidelines based on annotation experiments in 16 languages from several language families. These guidelines take both universal and language-specific phenomena into account. We hope that this will boost the development of language-independent and cross-lingual VMWE identification systems.
Participation
-------------
Participation is open and free worldwide.
We ask potential participant teams to register using the expression of interest form:
https://docs.google.com/forms/d/e/1FAIpQLSfnrAfr1B9IV3ftymUwxPGau91Zzc0ggt10U96tP7IpCkAG0g/viewform
Task updates and questions will be posted to our public mailing list:
http://groups.google.com/group/verbalmwe
More details on the annotated corpora can be found here:
The annotation guidelines used in manual annotation of the training and test sets are available here:
http://parsemefr.lif.univ-mrs.fr/guidelines-hypertext/
Publication and workshop
------------------------
Shared task participants will be invited to submit a system description paper to a special track of the EACL 2017 workshop on Multiword Expressions, to be held in Valencia, Spain on April 3 or 4. Submitted system description papers must follow the workshop submission instructions and will go through double-blind peer reviewing by other participants and selected MWE 2017 program committee members. Their acceptance depends on the quality of the paper rather than on the results obtained in the shared task. Authors of the accepted papers will present their work as posters/demos in a dedicated session of the workshop, collocated with EACL 2017. The submission of a system description paper is not mandatory.
Provided data
-------------
For each language, we will provide two corpora to the participants:
* Manually built **training corpora** in which VMWEs are annotated according to universal guidelines. These data will be made available to the participants in advance, in order to allow them to train their systems.
* Raw (unannotated) **test corpora** to be used as input to the systems. The VMWE annotations in this corpus will be kept secret.
A sample parseme-tsv training file in English is provided here:
http://typo.uni-konstanz.de/parseme/images/shared-task/format/parseme-tsv-sample.txt
When available, paired CONLL-U files containing parts of speech, lemmas, morphological features and/or syntactic dependencies will also be provided. Depending on the language, the information will come from treebanks (e.g., Universal Dependencies) or from automatic parsers trained on treebanks (e.g., UDPipe).
We are currently preparing corpora for the following languages: Bulgarian, Czech, German, Greek, English, Spanish, Farsi, French, Hebrew, Croatian, Hungarian, Italian, Lithuanian, Maltese, Polish, Brazilian Portuguese, Romanian, Swedish, Slovene, Turkish, Yiddish.
The amount of annotated data will depend on the language, and the list of covered languages may vary until the release of the training corpora.
Tracks
------
System results can be submitted in two tracks:
* **Closed track**: Systems using only the provided training data - VMWE annotations + CONLL-U files (if any) - to learn VMWE identification models and/or rules.
* **Open track**: Systems using or not the provided training data, plus any additional resources deemed useful (MWE lexicons, symbolic grammars, wordnets, raw corpora, word embeddings, parsers, etc.). This track includes notably purely symbolic and rule-based systems.
Teams submitting systems in the open track will be requested to describe and provide references to all resources used at submission time. Teams are encouraged to favor freely available resources for better reproducibility of their results.
Evaluation metrics
------------------
Participants will provide the output produced by their systems on the test corpus. This output will be compared with the gold standard (ground truth). Evaluation metrics are precision, recall and F1, both strict (per VMWE) and fuzzy (taking partial matches into account). The evaluation script will be released along with trial data. VMWE categories (e.g. LVC, ID, IReflV, VPC) will be ignored by the evaluation metrics. Categories are only provided in the training data to guide system design.
Important dates
-----------------
* Oct 14, 2016: first Call for Participation
* Nov 18, 2016: second Call for Participation
* Dec 13, 2016: trial data and evaluation script released
* Jan 6, 2016: training data released
* Jan 10, 2017: final Call for Participation
* Jan 20, 2017: blind test data released
* Jan 27, 2017: submission of system results
* Jan 30, 2017: announcement of results
* Feb 5, 2017: submission of shared task system description papers
* Feb 12, 2017: notification of acceptance
* Feb 19, 2017: camera-ready system description papers due
* 3 or 4 April 2017: shared task workshop colocated with MWE 2017
Organizing team
---------------