Submission & Evaluation

[a.t.cig...@gmail.com]

Dear organizers,

I have a couple of questions concerning sumbission and evaluation:

1) how many runs are we allowed to submit per task?

2) considering that the 4 classes in task A are fairly unbalanced, is there a particular reason why you are using Accuracy as evaluation measure (instead of macro F-1)?

Any intuition or explanation about this matter would be of great help.

Thank you,

Alessandra * 

[Genevieve Gorrell]

Hi Alessandra,

We have set it to one run. If a team has a reason why they need another run, they can ask us and we can arrange it. Sound reasonable?

Regarding the evaluation metric, we have already discussed this previously in the organizer team and we see the merit of what you suggest. We will notify the list as soon as a firm decision is made.

Genevieve

[a.t.cig...@gmail.com]

Ok,

thank you for your reply Genevieve! 

I understand you're still discussing it. Of course as you can imagine, it is important to know it soon, in order to optimize our approach with regard to the specific measure that will be used in the final evaluation.

Thanks,

Alessandra*

[Genevieve Gorrell]

We are actively working on a new scorer that includes macro F.

It's taking a little while due to many other deadlines.

[A.T. Cignarella]

Very good, thank you!

A*

--
You received this message because you are subscribed to the Google Groups "RumourEval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rumoureval+...@googlegroups.com.
To post to this group, send email to rumou...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rumoureval/1de958d3-7380-4492-89e7-22e33d14fce0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.