Labels of task b

[ghan]

Hi,

I have a question regarding task b:

In the codalab site you wrote:

"Although training threads for task B (verification) come with three labels, true, false and unverified, you should classify into two classes; true and false."

Then, few lines later:
"Unverified items should receive a confidence score of zero. For the F1 calculation, confidences below 0.5 will be considered a classification of unverified."

In the training data we have 3 labels, Should we discard "unverified" label in the produced predictions file or keep it but with confidence of ZERO?

Best,

[Leon Derczynski]

Keep the item, and give a conference score of zero.