STS benchmark, official announcement

[Eneko]

Dear all,

please find below the official announcement of STS benchmark. Thanks to all of you for your assistance.

Incidentally, the page includes a link to the temporary url of the task paper.

best

eneko

-------------


Dear colleagues,

We are glad to release the Semantic Textual Similarity benchmark.

STS Benchmark comprises a selection of the English datasets used in the STS tasks organized in the context of SemEval between 2012 and 2017. The selection of datasets include text from image captions, news headlines and user forums.

The main goal is to provide a standard benchmark to compare among meaning representation systems in future years. Previously, several authors have reported results across different years, with a different mixture of genres and training conditions.

We organized STS benchmark into train, development and test. The development part can be used to develop and tune hyperparameters of the systems, and the test part should be only used once for the final system.

We already have results for some relevant systems. Please find all details in http://ixa2.si.ehu.es/stswiki/index.php/STSbenchmark

best

eneko, dan, mona

[AL-NATSHEH Hussein]

Dear Eneko,

I have sent our system results on May17 as in the below email but I could not see our system reported in the benchmark results table. Could you please double check and let us know if there is anything extra should we provide? Thanks.

---

From: AL-NATSHEH Hussein
Sent: 17 May 2017 17:04
To: e.ag...@ehu.eus
Subject: [STS] Re: STS benchmark

Hello Eneko,

I had the chance this week to run our system on STS Benchmark (with the settings of best run (Run1) of UdL Team as described in the system description paper, attached).

The results of 3 runs of our system were as follows:
run1: {'test_score': 0.72402797552941101, 'dev_score': 0.78979959199179273}
run2: {'test_score': 0.7240556918383606, 'dev_score': 0.78955107284385118}
run3: {'test_score': 0.72416070656332898, 'dev_score': 0.78937855000741974}

Average of the above 3 runs:
{'test_score': 0.724081458, 'dev_score': 0.7895764049}

I hope you can add this results to the STSBenchmak page with the following details:
STS 2017 Task System description Paper:
Al-Natsheh et al.

Type:
Constrained

Model:
RandomForest

Link to Paper (as in the provided BibTex file of SemEval):
http://www.aclweb.org/anthology/S17-2013


If you want to verify or reproduce the results, please goto:
https://github.com/natsheh/sensim 
and run (python sts_benchmark.py) with the default parameters.

Thanks a lot and best regards,

Hussein AL-NATSHEH
UdL Team @ STS Task1
Tel: +33 7 51 53 94 82 - Fax: +33 4 78 77 23 75

Best regards,

Hussein AL-NATSHEH

Doctorant

Institut des Sciences de l'Homme (ISH), Centre National de la Recherche Scientifique (CNRS)
Laboratoire ERIC, Université Lumière Lyon 2, email: hussein.a...@eric.univ-lyon2.fr
École Doctorale InfoMaths, Université de Lyon

Tel: +33 7 51 53 94 82  -  Fax: +33 4 78 77 23 75

---

From: sts-s...@googlegroups.com [sts-s...@googlegroups.com] on behalf of Eneko [e.ag...@gmail.com]
Sent: 30 June 2017 10:36
To: STS SemEval
Subject: [STS] STS benchmark, official announcement

--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com
To unsubscribe, send email to sts-semeval...@googlegroups.com
For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel Cer]

Hi Hussein,

You probably already got a reply off list, but it looks like the STS Benchmark wiki was updated to include the UdL results under Feature engineered and mixed systems.

Dan

To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval+unsubscribe@googlegroups.com.

[AL-NATSHEH Hussein]

Hi Dan,

Thanks a lot. I can see our system reported in the wiki page. However, Could you please include our system benchmark result in Table 14 in the workshop paper (http://nlp.arizona.edu/SemEval-2017/pdf/SemEval001.pdf )?

Thanks a lot,
Hussein

---

From: Daniel Cer [c...@google.com]
Sent: 07 July 2017 04:52
To: STS SemEval
Cc: AL-NATSHEH Hussein
Subject: Re: [STS] STS benchmark, official announcement

[Daniel Cer]

Hi Hussein,

I'll update the paper to include your STS benchmark numbers.

We'll be posting a version of the paper to arxiv, so the numbers can certainly be included there. We should also be able to update the version hosted at nlp.arizona.edu. I'm in the process of asking whether we can still update the ACL anthology version before the workshop. After the workshop, there's a good process for updating papers.

Dan

To unsubscribe, send email to sts-semeval...@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval...@googlegroups.com.

[Ergun Bicici]

Updates for papers in ACL anthology were done with a versioning system; so you have your version 1 and then version 2.

Maybe we'll see a new section next year for the STS task if the task is organized again. However, STS does not appear as a task in SemEval-2018 for now:

http://alt.qcri.org/semeval2018/index.php?id=tasks

Best Regards,
Ergun

Ergun Biçici

http://bicici.github.com/

On Fri, Jul 7, 2017 at 7:09 PM, 'Daniel Cer' via STS SemEval <sts-s...@googlegroups.com> wrote:

Hi Hussein,

I'll update the paper to include your STS benchmark numbers.

We'll be posting a version of the paper to arxiv, so the numbers can certainly be included there. We should also be able to update the version hosted at nlp.arizona.edu. I'm in the process of asking whether we can still update the ACL anthology version before the workshop. After the workshop, there's a good process for updating papers.

Dan

To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com

To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval+unsubscribe@googlegroups.com.

[Ergun Bicici]

Dear Eneko,

I am sending RTM's results on the test set:

                     r       MAE     RAE     MAER    MRAER

RTM mix weight 4  ​0.7061   0.8534   0.6524   0.5343   0.6876

​on the training set (10-fold cross-validation on the training and dev set combined, so not on the dev set and therefore different):​

                     r       MAE     RAE     MAER    MRAER

​

RTM mix weight 4

​    ​

​

0.7319   0.7833   0.6188   0.4575   0.6847​

where r is Pearson's correlation. RTM results are based on [1], which adds additional features to [2] and uses mix weight model introduced in [2].

Sentence representation: Other

supervision: Train

​Compared with other STS results with MRAER < 1, ​RTM on STS benchmark obtained the best results for non-specific domain (ALL):​

​

​Thank you again for preparing the STS benchmark dataset.​

Best Regards,
Ergun

Ergun Biçici

http://bicici.github.com/

References:

[1] Ergun Biçici. Predicting Translation Performance with Referential Translation Machines. In Proc. of the Second Conference on Statistical Machine Translation (WMT17), Copenhagen, Denmark, September 2017.

[2] Ergun Biçici. RTM at SemEval-2017 Task 1: Referential Translation Machines for Predicting Semantic Similarity. In Proc. of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, Canada, pages 194-198, August 2017. Association for Computational Linguistics. [PDF] [Abstract] [bibtex-entry]

[Daniel Cer]

Thanks, Ergun. Do you want the RTM numbers put in Table 14 of the task description paper?  

Dan

To unsubscribe, send email to sts-semeval...@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com

To unsubscribe, send email to sts-semeval...@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com

To unsubscribe, send email to sts-semeval...@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---

You received this message because you are subscribed to a topic in the Google Groups "STS SemEval" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sts-semeval/H3HFqz6GGfA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sts-semeval...@googlegroups.com.

[Ergun Bicici]

Yes, if you can; thank you.

Best Regards,
Ergun

Ergun Biçici

http://bicici.github.com/

Dan

To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com

To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com

To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---

You received this message because you are subscribed to a topic in the Google Groups "STS SemEval" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sts-semeval/H3HFqz6GGfA/unsubscribe.

To unsubscribe from this group and all its topics, send an email to sts-semeval+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com

To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval+unsubscribe@googlegroups.com.

[tienh...@gmail.com]

Dear enoko,

I found a mistake in the dataset statistic table. It should be:

               train  dev test total 
       -----------------------------
       news     3299  500  500  4299
       caption  2000  625  625  3250
       forum     450  375  254  1079
       -----------------------------
       total    5749 1500 1379  8628

Vào 17:36:34 UTC+9 Thứ Sáu, ngày 30 tháng 6 năm 2017, Eneko đã viết:

[Eneko Agirre]

thanks for spotting it, corrected!

best

eneko

09/08/2017 04:33 AM(e)an, tienh...@gmail.com igorleak idatzi zuen:

--

--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com

To unsubscribe, send email to sts-semeval...@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Eneko Agirre
Euskal Herriko Unibertsitatea
Universidad del Pais Vasco
University of the Basque Country
http://ixa2.si.ehu.eus/eneko