FINAL Call for Participation - Shared Task on Automatic Identification of Verbal Multiword Expressions
Carlos Ramisch
FINAL CALL FOR PARTICIPATION
Shared task on automatic identification of verbal multiword expressions
http://multiword.sourceforge.net/sharedtask2017
=======================================================================
Apologies for cross-posting
The PARSEME shared task on automatic identification of verbal multiword expressions (VMWEs) aims at identifying verbal MWEs in running texts. Verbal MWEs include idioms (to let the cat out of the bag), light verb constructions (to make a decision), verb-particle constructions (to give up), and inherently reflexive verbs (se suicider 'to suicide' in French). Their identification is a well-known challenge for NLP applications, due to their complex characteristics: discontinuity, non-compositionality, heterogeneity and syntactic variability.
The shared task is highly multilingual: we cover 18 languages from as many countries. PARSEME members have elaborated annotation guidelines based on annotation experiments in 16 languages from several language families. These guidelines take both universal and language-specific phenomena into account. We hope that this will boost the development of language-independent and cross-lingual VMWE identification systems.
Participation
-------------
Participation is open and free worldwide. We ask potential participant teams to register using the expression of interest form. Task updates and questions will be posted to our public mailing list. For more details on the annotation of the corpora visit the dedicated PARSEME page and check the annotation guidelines used in manual annotation of the training and test sets.
It should be noted that a large international community has been gathered (via the PARSEME network) around the effort of putting forward universal guidelines and performing corpus annotations. Our policy is to allow the same national teams, which provide annotated corpora, to also submit VMWE identification systems for the shared task. Our participation policy is described on the shared task website.
PARSEME travel grants
---------------------
PARSEME will fund travel and stay for over 30 MWE 2017 workshop participants from the Action's member countries. The modalities are available on a dedicated page.
Publication and workshop
------------------------
Shared task participants will be invited to submit input of two kinds to the SHARED TASK TRACK of the EACL 2017 workshop on Multiword Expressions via the dedicated START space:
System results (by January 27) obtained on the blind data (released on 20 January). The results for all languages should be submitted in a single .zip archive containing a single folder per language, named according to the ISO 639-1 code (e.g. FR for French, MT for Maltese, etc.). Each output file must be named test.system.parsemetsv and conform to the parsemetsv format. The format of each file should be checked before submission by the validation script as follows:
./checkParsemeTsvFormat.py test.system.parsemetsv
If one system participates both in the open and in the closed track, two independent submissions are required.
A system description paper (by February 5). These papers must follow the workshop submission instructions and will go through double-blind peer reviewing by other participants and selected MWE 2017 program committee members. Their acceptance depends on the quality of the paper rather than on the results obtained in the shared task. Authors of the accepted papers will present their work as posters/demos in a dedicated session of the workshop. The submission of a system description paper is not mandatory.
Provided data
-------------
For most languages, we provide two corpora to the participants:
Manually built training corpora in which VMWEs are annotated according to the universal guidelines.
Raw (unannotated) test corpora (to be released on 20 January) to be used as input to the systems. The VMWE annotations contained in this corpus, performed according to the same guidelines, will be kept secret.
Training and test corpora are provided in the parsemetsv format, inspired by the CONLL-U format. For most languages (all except BG, ES, HE and LT), paired files in the CONLL-U format - not necessarily using UD tagsets - containing parts of speech, lemmas, morphological features and/or syntactic dependencies are also provided. Depending on the language, the information comes from treebanks (e.g., Universal Dependencies) or from automatic parsers trained on treebanks (e.g., UDPipe).
The training data are available in our public GitLab repository for 16 languages: Bulgarian (BG), Czech (CS), German (DE), Greek (EL), Farsi (FA), French (FR), Hebrew (HE), Hungarian (HU), Italian (IT), Lithuanian (LT), Maltese (MT), Polish (PL), Brazilian Portuguese (PT), Romanian (RO), Slovene (SL), Turkish (TR). For 2 languages, we intend to provide only test data, but the trial data is available for training: Spanish (ES), Swedish (SV). Follow the repository link to access folders for individual languages. You can also download an archive containing all the training data directly using this shortcut link.
Tracks
------
System results can be submitted in two tracks:
Closed track: Systems using only the provided training data - VMWE annotations + CONLL-U files (if any) - to learn VMWE identification models and/or rules.
Open track: Systems using or not the provided training data, plus any additional resources deemed useful (MWE lexicons, symbolic grammars, wordnets, raw corpora, word embeddings, parsers, etc.). This track includes notably purely symbolic and rule-based systems.
Teams submitting systems in the open track will be requested to describe and provide references to all resources used at submission time. Each team may submit one result per track for each language. Teams are encouraged to favor freely available resources for better reproducibility of their results.
Evaluation metrics
------------------
Participants will provide the output produced by their systems on the blind test corpus. This output will be compared with the gold standard (ground truth). Evaluation metrics are precision, recall and F1, both strict (per VMWE) and fuzzy (taking partial matches into account). The evaluation script is available in our public data repository. Token-based F1 takes into account the fact that:
* discontinuities are allowed (take something into account)
* overlapping are allowed (take a walk and then a long shower)
* embeddings are allowed both at the syntactic level (take the fact that I didn't [give up] into account) and at the level of lexicalized components (let the cat out of the bag)
* multiword tokens may lead to one-token MWEs (ES suicidarse)
Therefore, we measure the best F1 score from all possible matches between the set of MWE token ranks in the gold and system sentences. We perform this by looking at all possible ways of matching MWEs in both sets.
VMWE categories (e.g., LVC, ID, IReflV, VPC - see the guidelines) are ignored by the evaluation metrics. Categories are only provided in the training data to guide system design. Systems focusing on selected VMWE categories only are also encouraged to participate.
Important dates
-----------------
* Oct 14, 2016: first Call for Participation
* Nov 18, 2016: second Call for Participation
* Dec 13, 2016: trial data and evaluation script released
* Jan 6, 2016: training data released
* Jan 10, 2017: final Call for Participation
* Jan 20, 2017: blind test data released
* Jan 27, 2017: submission of system results
* Jan 30, 2017: announcement of results
* Feb 5, 2017: submission of shared task system description papers
* Feb 12, 2017: notification of acceptance
* Feb 19, 2017: camera-ready system description papers due
* 4 April 2017: shared task workshop colocated with MWE 2017
Organizing team
---------------
Marie Candito, Fabienne Cap, Silvio Cordeiro, Antoine Doucet, Voula Giouli, Behrang QasemiZadeh, Carlos Ramisch, Federico Sangati, Agata Savary, Ivelina Stoyanova, Veronika Vincze.