Update

[Genevieve Gorrell]

Hi all,

In case you didn't find it, the specification for how the answer should be is under the "Learn the Details" tab on the "Evaluation" page. I have just updated it to say that we are using macro F1 for the main evaluation metric. I also added that the answers file should be called "answer.json" (inside the zip file).

I have raised the number of submissions allowed to 2 since it seems to cause unnecessary stress to have it at 1. As I say though, if your submission fails completely, you can just try again as often as you need anyway to get it to succeed. If your submission succeeds but you have a reason why you need another one, you can ask us (rumoureval-20...@googlegroups.com).

You won't be able to see your score. Not all submissions appear on that graph.

Submissions all look basically okay though so far. Some people don't enter task B, just A, which is fine. However, don't forget that source tweets/posts are also included in task A! This has been discussed in a few previous threads, for example the thread titled "classification of the source tweet". So task A expects responses for all source and reply texts (1827). Task B expects responses for source texts (81).

Significant dates going forward are here, btw, including system paper deadline:

http://alt.qcri.org/semeval2019/

Best,

Genevieve

[Martin Fajčík]

Hi, lets say I have two model variants and they are about the same score on validation data. If I submit both, will the final score be the better submission, or the score from last submission?
Thank you for the information!
Martin

Dňa pondelok, 28. januára 2019 12:10:05 UTC+1 Genevieve Gorrell napísal(-a):

Hi all,

In case you didn't find it, the specification for how the answer should be is under the "Learn the Details" tab on the "Evaluation" page. I have just updated it to say that we are using macro F1 for the main evaluation metric. I also added that the answers file should be called "answer.json" (inside the zip file).

I have raised the number of submissions allowed to 2 since it seems to cause unnecessary stress to have it at 1. As I say though, if your submission fails completely, you can just try again as often as you need anyway to get it to succeed. If your submission succeeds but you have a reason why you need another one, you can ask us (rumoureval-2019-organizers@googlegroups.com).

[Leon Derczynski]

Hi Martin,

The best-scoring one will be used in the final ranking, though both will appear in the task report

Leon

On Mon, 28 Jan 2019 at 21:40, Martin Fajčík <blacke...@gmail.com> wrote:

Hi, lets say I have two model variants and they are about the same score on validation data. If I submit both, will the final score be the better submission, or the score from last submission?
Thank you for the information!
Martin

Dňa pondelok, 28. januára 2019 12:10:05 UTC+1 Genevieve Gorrell napísal(-a):

Hi all,

In case you didn't find it, the specification for how the answer should be is under the "Learn the Details" tab on the "Evaluation" page. I have just updated it to say that we are using macro F1 for the main evaluation metric. I also added that the answers file should be called "answer.json" (inside the zip file).

I have raised the number of submissions allowed to 2 since it seems to cause unnecessary stress to have it at 1. As I say though, if your submission fails completely, you can just try again as often as you need anyway to get it to succeed. If your submission succeeds but you have a reason why you need another one, you can ask us (rumoureval-20...@googlegroups.com).

You won't be able to see your score. Not all submissions appear on that graph.

Submissions all look basically okay though so far. Some people don't enter task B, just A, which is fine. However, don't forget that source tweets/posts are also included in task A! This has been discussed in a few previous threads, for example the thread titled "classification of the source tweet". So task A expects responses for all source and reply texts (1827). Task B expects responses for source texts (81).

Significant dates going forward are here, btw, including system paper deadline:

http://alt.qcri.org/semeval2019/

Best,

Genevieve

--
You received this message because you are subscribed to the Google Groups "RumourEval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rumoureval+...@googlegroups.com.
To post to this group, send email to rumou...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rumoureval/4291c5b6-490b-4cc9-9a1b-63cbd88c87f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.