STS 2015 gold standard

59 views
Skip to first unread message

Shashi

unread,
Jul 4, 2017, 11:54:46 AM7/4/17
to STS SemEval
Hi Everyone,

How do you treat the blank lines in gold standard score files? Every file in STS 2015 task1 test files has blank lines in annotated score files.

Thanks,
Shashi

Ergun Bicici

unread,
Jul 5, 2017, 2:33:13 AM7/5/17
to sts-s...@googlegroups.com

This means score is not provided for those sentences. You only consider the lines that have a score.


Best Regards,
Ergun

Ergun Biçici


--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com
To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com
For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Ergun Bicici

unread,
Jul 5, 2017, 2:34:16 AM7/5/17
to sts-s...@googlegroups.com

This means score is not provided for those sentences.

Best Regards,
Ergun

Ergun Biçici


Daniel Cer

unread,
Jul 6, 2017, 10:30:51 PM7/6/17
to STS SemEval
Ergun is correct. 

For scoring purposes, only consider the pairs that have a non-blank line score in the gold standard file. This is done automatically by the provided scoring script (correlation-noconfidence.pl)

Dan
Reply all
Reply to author
Forward
0 new messages