New RumourEval scoring approach

[Genevieve Gorrell]

Hello all. After a number of requests we decided to switch scoring from accuracy to macro F. The Codalab site therefore now shows macro F, not accuracy, for both A and B tasks.

Accuracy is still calculated but it doesn't go to the leaderboard. RMSE is still calculated and does still go to the leaderboard.

The home scorer linked from the data page is now the updated version so you should download that.

Thanks to Elena Kochkina for the implementation!

Please let me know if you have any issues!

Genevieve

[Genevieve Gorrell]

Just made an update to the scorer (online and offline)--your unverified answers will come as a true or false with low confidence, which we forgot to accommodate in the macro F.

Now your answers with a confidence lower than 0.5 are treated as unverified for the purposes of the macro F. Hope that makes sense.