STS benchmark, official announcement

135 views
Skip to first unread message

Eneko

unread,
Jun 30, 2017, 4:36:34 AM6/30/17
to STS SemEval

Dear all,

please find below the official announcement of STS benchmark. Thanks to all of you for your assistance.

Incidentally, the page includes a link to the temporary url of the task paper.

best

eneko

-------------


Dear colleagues,

We are glad to release the Semantic Textual Similarity benchmark.

STS Benchmark comprises a selection of the English datasets used in the STS tasks organized in the context of SemEval between 2012 and 2017. The selection of datasets include text from image captions, news headlines and user forums.

The main goal is to provide a standard benchmark to compare among meaning representation systems in future years. Previously, several authors have reported results across different years, with a different mixture of genres and training conditions.

We organized STS benchmark into train, development and test. The development part can be used to develop and tune hyperparameters of the systems, and the test part should be only used once for the final system.

We already have results for some relevant systems. Please find all details in http://ixa2.si.ehu.es/stswiki/index.php/STSbenchmark

best

eneko, dan, mona

AL-NATSHEH Hussein

unread,
Jul 2, 2017, 4:55:09 AM7/2/17
to sts-s...@googlegroups.com
Dear Eneko,

I have sent our system results on May17 as in the below email but I could not see our system reported in the benchmark results table. Could you please double check and let us know if there is anything extra should we provide? Thanks.


From: AL-NATSHEH Hussein
Sent: 17 May 2017 17:04
To: e.ag...@ehu.eus
Subject: [STS] Re: STS benchmark

Hello Eneko,

I had the chance this week to run our system on STS Benchmark (with the settings of best run (Run1) of UdL Team as described in the system description paper, attached).

The results of 3 runs of our system were as follows:
run1: {'test_score': 0.72402797552941101, 'dev_score': 0.78979959199179273}
run2: {'test_score': 0.7240556918383606, 'dev_score': 0.78955107284385118}
run3: {'test_score': 0.72416070656332898, 'dev_score': 0.78937855000741974}

Average of the above 3 runs:
{'test_score': 0.724081458, 'dev_score': 0.7895764049}

I hope you can add this results to the STSBenchmak page with the following details:
STS 2017 Task System description Paper:
Al-Natsheh et al.

Type:
Constrained

Model:
RandomForest

Link to Paper (as in the provided BibTex file of SemEval):
http://www.aclweb.org/anthology/S17-2013


If you want to verify or reproduce the results, please goto:
https://github.com/natsheh/sensim 
and run (python sts_benchmark.py) with the default parameters.

Thanks a lot and best regards,

Hussein AL-NATSHEH
UdL Team @ STS Task1
Tel: +33 7 51 53 94 82 - Fax: +33 4 78 77 23 75
Best regards,

Hussein AL-NATSHEH
Doctorant
Institut des Sciences de l'Homme (ISH), Centre National de la Recherche Scientifique (CNRS)
Laboratoire ERIC, Université Lumière Lyon 2, email: hussein.a...@eric.univ-lyon2.fr
École Doctorale InfoMaths, Université de Lyon


From: sts-s...@googlegroups.com [sts-s...@googlegroups.com] on behalf of Eneko [e.ag...@gmail.com]
Sent: 30 June 2017 10:36
To: STS SemEval
Subject: [STS] STS benchmark, official announcement

--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com
To unsubscribe, send email to sts-semeval...@googlegroups.com
For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Daniel Cer

unread,
Jul 6, 2017, 10:52:12 PM7/6/17
to STS SemEval, Hussein.A...@ish-lyon.cnrs.fr
Hi Hussein,

You probably already got a reply off list, but it looks like the STS Benchmark wiki was updated to include the UdL results under Feature engineered and mixed systems.

Dan
To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval+unsubscribe@googlegroups.com.

AL-NATSHEH Hussein

unread,
Jul 6, 2017, 11:52:06 PM7/6/17
to Daniel Cer, STS SemEval
Hi Dan,

Thanks a lot. I can see our system reported in the wiki page. However, Could you please include our system benchmark result in Table 14 in the workshop paper (http://nlp.arizona.edu/SemEval-2017/pdf/SemEval001.pdf )?

Thanks a lot,
Hussein


From: Daniel Cer [c...@google.com]
Sent: 07 July 2017 04:52
To: STS SemEval
Cc: AL-NATSHEH Hussein
Subject: Re: [STS] STS benchmark, official announcement

Daniel Cer

unread,
Jul 7, 2017, 12:09:41 PM7/7/17
to AL-NATSHEH Hussein, STS SemEval
Hi Hussein,

I'll update the paper to include your STS benchmark numbers.

We'll be posting a version of the paper to arxiv, so the numbers can certainly be included there. We should also be able to update the version hosted at nlp.arizona.edu. I'm in the process of asking whether we can still update the ACL anthology version before the workshop. After the workshop, there's a good process for updating papers.

Dan

To unsubscribe, send email to sts-semeval...@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval...@googlegroups.com.

Ergun Bicici

unread,
Jul 7, 2017, 12:54:55 PM7/7/17
to sts-s...@googlegroups.com, AL-NATSHEH Hussein

Updates for papers in ACL anthology were done with a versioning system; so you have your version 1 and then version 2.

Maybe we'll see a new section next year for the STS task if the task is organized again. However, STS does not appear as a task in SemEval-2018 for now:


Best Regards,
Ergun

Ergun Biçici


On Fri, Jul 7, 2017 at 7:09 PM, 'Daniel Cer' via STS SemEval <sts-s...@googlegroups.com> wrote:
Hi Hussein,

I'll update the paper to include your STS benchmark numbers.

We'll be posting a version of the paper to arxiv, so the numbers can certainly be included there. We should also be able to update the version hosted at nlp.arizona.edu. I'm in the process of asking whether we can still update the ACL anthology version before the workshop. After the workshop, there's a good process for updating papers.

Dan

To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com
To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval+unsubscribe@googlegroups.com.

Ergun Bicici

unread,
Jul 9, 2017, 4:43:39 AM7/9/17
to sts-s...@googlegroups.com

Dear Eneko,

I am sending RTM's results on the test set:
                     r       MAE     RAE     MAER    MRAER
RTM mix weight 4  0.7061   0.8534   0.6524   0.5343   0.6876

​on the training set (10-fold cross-validation on the training and dev set combined, so not on the dev set and therefore different):​

                     r       MAE     RAE     MAER    MRAER
RTM mix weight 4
​    ​
0.7319   0.7833   0.6188   0.4575   0.6847

where r is Pearson's correlation. RTM results are based on [1], which adds additional features to [2] and uses mix weight model introduced in [2].
Sentence representation: Other
supervision: Train

​Compared with other STS results with MRAER < 1, ​RTM on STS benchmark obtained the best results for non-specific domain (ALL):​
Inline image 2

​Thank you again for preparing the STS benchmark dataset.​


Best Regards,
Ergun

Ergun Biçici

References:
[1] Ergun Biçici. Predicting Translation Performance with Referential Translation Machines. In Proc. of the Second Conference on Statistical Machine Translation (WMT17), Copenhagen, Denmark, September 2017.
[2] Ergun Biçici. RTM at SemEval-2017 Task 1: Referential Translation Machines for Predicting Semantic Similarity. In Proc. of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, Canada, pages 194-198, August 2017. Association for Computational Linguistics. [PDF] [Abstract] [bibtex-entry]

Daniel Cer

unread,
Jul 12, 2017, 2:51:31 AM7/12/17
to sts-s...@googlegroups.com
Thanks, Ergun. Do you want the RTM numbers put in Table 14 of the task description paper?  

Dan

To unsubscribe, send email to sts-semeval...@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com
To unsubscribe, send email to sts-semeval...@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com
To unsubscribe, send email to sts-semeval...@googlegroups.com
You received this message because you are subscribed to a topic in the Google Groups "STS SemEval" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sts-semeval/H3HFqz6GGfA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sts-semeval...@googlegroups.com.

Ergun Bicici

unread,
Jul 12, 2017, 3:45:42 AM7/12/17
to sts-s...@googlegroups.com

Yes, if you can; thank you.


Best Regards,
Ergun

Ergun Biçici


Dan

To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com
To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com
To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com
You received this message because you are subscribed to a topic in the Google Groups "STS SemEval" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sts-semeval/H3HFqz6GGfA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sts-semeval+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com
To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval+unsubscribe@googlegroups.com.

tienh...@gmail.com

unread,
Sep 8, 2017, 4:36:25 AM9/8/17
to STS SemEval
Dear enoko,
I found a mistake in the dataset statistic table. It should be:
               train  dev test total 
       -----------------------------
       news     3299  500  500  4299
       caption  2000  625  625  3250
       forum     450  375  254  1079
       -----------------------------
       total    5749 1500 1379  8628

Vào 17:36:34 UTC+9 Thứ Sáu, ngày 30 tháng 6 năm 2017, Eneko đã viết:

Eneko Agirre

unread,
Sep 8, 2017, 4:37:35 AM9/8/17
to sts-s...@googlegroups.com, tienh...@gmail.com




thanks for spotting it, corrected!

best

eneko


09/08/2017 04:33 AM(e)an, tienh...@gmail.com igorleak idatzi zuen:
--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com
To unsubscribe, send email to sts-semeval...@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Eneko Agirre
Euskal Herriko Unibertsitatea
Universidad del Pais Vasco
University of the Basque Country
http://ixa2.si.ehu.eus/eneko
Reply all
Reply to author
Forward
0 new messages