SemEval 2016 Task 2 - Training data released

[Interpretable STS Semeval Task]

                    Call for Participation
                    SemEval 2016 Task 2

 Interpretable Semantic Textual Similarity (iSTS)
           http://alt.qcri.org/semeval2016/task2

                    Training data released

Semantic Textual Similarity (STS) measures the degree of equivalence in the underlying semantics of paired snippets of text.
Interpretable STS (iSTS) adds an explanatory layer. Given the input (pairs of sentences) participants need first to identify the chunks in each sentence, and then, align chunks across the two sentences, indicating the relation and similarity score of each  alignment.

For instance, given the following two sentences (drawn from a corpus of headlines):

•     12 killed in bus accident in Pakistan
•     10 killed in road accident in NW Pakistan
  

A participant system would split the sentence in chunks:

•     [12] [killed] [in bus accident] [in Pakistan]
•     [10] [killed] [in road accident] [in NW Pakistan]

And then provide the alignments between chunks, indicating the relation  and the similarity score of the alignment, as follows:

1.     [12] <=> [10] : (SIMILAR 4)
2.     [killed] <=> [killed] : (EQUIVALENT 5)
3.     [in bus accident] <=> [in road accident] : (MORE-SPECIFIC 4)
4.     [in Pakistan] <=> [in NW Pakistan] : (MORE-GENERAL 4)

Given such an alignment, an automatic system could explain why the two sentences are very similar but not equivalent, for instance, phrasing the differences as follows:

1.     the first sentence mentions "12" instead of "10", "bus
2.     accident" is more specific that "road accident" in the second,
3.     and "Pakistan" is more general than "NW Pakistan" in the second.

While giving such explanations comes naturally to people, constructing algorithms and computational models that mimic human level performance represents a difficult natural language understanding (NLU) problem, with applications in dialogue systems,  interactive systems and educational systems.

Please check the task website for more details on chunking, alignment, relation labels and scores.


== Datasets ==

Two datasets are currently covered, comprising pairs of sentences from news headlines and image captions. The pairs are a subset of the datasets released in the STS tasks. Please check the iSTS train dataset for details.


== New in 2016 ==

The 2015 STS task offered a pilot subtask on interpretable STS, which showed that the task is feasible, with high inter-annotator agreement and system scores well above baselines.

For 2016, the pilot subtask has been updated into a standalone task. The restriction to allow only 1:1 alignment has been lifted. Annotation guidelines have been updated, and new training has been released.
Please check out http://alt.qcri.org/semeval2016/task2/for more details.


== Participants ==

If you are interested in participating, you should:

1. join the mailing list for updates at https://groups.google.com/group/ists-semeval
2. check the guidelines and train data at http://alt.qcri.org/semeval2016/task2
3. register at the semeval website: http://goo.gl/forms/cGkRocFFph

• note that registration and mailing list management are independent, please do both of them
  

== Important dates ==

    Train data ready: NOW!
    Evaluation start: January 10, 2016
    Evaluation end: January 31, 2016
    Paper submission due: February 28, 2016 [TBC]
    Paper reviews due: March 31, 2016 [TBC]
    Camera ready due: April 30, 2016 [TBC]
    SemEval workshop: Summer 2016


== Organizers ==

 Eneko Agirre, Aitor Gonzalez-Agirre, Iñigo Lopez-Gazpio, Montse Maritxalar, German Rigau and Larraitz Uria.

 University of the Basque Country


== Reference ==

Agirre, E.  and Banea, C.  and Cardie, C.  and Cer, D.  and Diab, M.
  and Gonzalez-Agirre, A.  and Guo, W.  and Lopez-Gazpio, I. and
  Maritxalar, M. and Mihalcea, R.  and Rigau, G.  and Uria, L. and
  Wiebe, J. (2015). SemEval-2015 task 2: Semantic textual similarity,
  English, Spanish and pilot on interpretability. In Proceedings of
  the 9th International Workshop on Semantic Evaluation (SemEval
  2015), June. [http://anthology.aclweb.org/S/S15/S15-2045.pdf]