STS 2015 gold standard

[Shashi]

Hi Everyone,

How do you treat the blank lines in gold standard score files? Every file in STS 2015 task1 test files has blank lines in annotated score files.

Thanks,

Shashi

[Ergun Bicici]

This means score is not provided for those sentences. You only consider the lines that have a score.

Best Regards,
Ergun

Ergun Biçici

http://bicici.github.com/

--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com
To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com
For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ergun Bicici]

This means score is not provided for those sentences.

Best Regards,
Ergun

Ergun Biçici

http://bicici.github.com/

[Daniel Cer]

Ergun is correct. 

For scoring purposes, only consider the pairs that have a non-blank line score in the gold standard file. This is done automatically by the provided scoring script (correlation-noconfidence.pl)

Dan