Ambiguous suggestion sentences

[potn...@gmail.com]

It seems as some of the training set features are wrongly classified, or that our logic regarding suggestions is at fault.

As we went through the training set we noticed several occasions in which the nature of the sentence (whether it is a suggestion or not) is rather ambiguous. You can see below some examples that we hope will explain our point best.

0_919

"Please support it if you want to make XAML capable of amazing animations."

0

0_920

"Please support SMIL animations in XAML."

0

0_362

"Didn't do this on Firefox or Chrome."

1

0_109

"It's not enough for devs :("

1

0_26

"Once we change the name, then everything works all right."

1

1400_4

"On all the ... buttom right, where I can share etc I would like to be able to print."

0

687_3

"make the mapicon of the windows phone sdk 8.1 able to be on top by the use of zindex make the event tapped accessible for mapicon too"

0

681_1

"It would be easier to sync calendars if they have a RemoteId like appointments."

0

Therefore, we would like to ask how exactly you have concluded which class each sentence belongs to, since following our logic the above examples should be classified the opposite.

[Sapna Negi]

Hi,

Some of the examples (not all) below do seem to be against the guidelines, but as per our observation it is quite an ambiguous annotation task for the annotators.

Since the first round of annotation were performed through crowdsourcing, it is likely to contain some mis-judged labels where 2 out of 3 annotators agreed.

For details please refer to this paper:

https://arxiv.org/pdf/1806.02179.pdf

Regards

Sapna

-- 
You received this message because you are subscribed to the Google Groups "SemEval 2019: Task 9" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2019-ta...@googlegroups.com.
To post to this group, send email to semeval-2...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/semeval-2019-task-9/14b8e1cc-053d-4040-b34d-9bb4d4db1274%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.