Evaluation Goldstandard.csv

[Paulo Henrique da Silva]

Hello.

Can I do the evaluation of my result or only the organization of the competition does the evaluation?

How do I download the goldstandard.csv file to evaluate my result?

I searched the github file and did not find it.

Thanks

[Sapna Negi]

Hi,

You can currently evaluate/validate your system with the trial test dataset provided on Github (TrialData_SubtaskA_Test.csv or TrialData_SubtaskB_Test.csv ), and upload your prediction file on codalab under the ‘Participate’ tab.

https://competitions.codalab.org/competitions/19955#participate-submit_results

Instructions are provided in the Github repo, under 'Getting Started'

https://github.com/Semeval2019Task9/Subtask-A

Please note that the evaluation period approximately starts from Jan 10th and will end by Jan 30th, where there will be a fresh evaluation dataset provided.

Regards

Sapna

--
You received this message because you are subscribed to the Google Groups "SemEval 2019: Task 9" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2019-ta...@googlegroups.com.
To post to this group, send email to semeval-2...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/semeval-2019-task-9/4cc50ebf-bc60-41b5-bf74-f909a826931c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paulo Henrique da Silva]

Hi, Sapna.

Thanks for your quick reply.

I study at the Federal University of Goias, Brazil and I need to validate the results of my work for a NLP discipline.

The only way to evaluate is by sending the result to CodaLab?

You said that the evaluation period starts in Jan 10th and will end by Jan 30th.
If I send the result today when the Codalab organization can evaluate?

Regards

Paulo

Em quarta-feira, 12 de dezembro de 2018 13:26:44 UTC-2, Sapna Negi escreveu:

Hi,

You can currently evaluate/validate your system with the trial test dataset provided on Github (TrialData_SubtaskA_Test.csv or TrialData_SubtaskB_Test.csv ), and upload your prediction file on codalab under the ‘Participate’ tab.

https://competitions.codalab.org/competitions/19955#participate-submit_results

Instructions are provided in the Github repo, under 'Getting Started'

https://github.com/Semeval2019Task9/Subtask-A

Please note that the evaluation period approximately starts from Jan 10th and will end by Jan 30th, where there will be a fresh evaluation dataset provided.

Regards

Sapna

On 12 Dec 2018, at 19:12, Paulo Henrique da Silva <paul...@gmail.com> wrote:

Hello.

Can I do the evaluation of my result or only the organization of the competition does the evaluation?

How do I download the goldstandard.csv file to evaluate my result?

I searched the github file and did not find it.

Thanks

--
You received this message because you are subscribed to the Google Groups "SemEval 2019: Task 9" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2019-task-9+unsub...@googlegroups.com.

[Sapna Negi]

Hi Paulo,

You will see the scores in a few minutes after uploading your prediction file on Codalab, its an automated process. 

The final evaluation dataset will be released near Jan 10th, and the upload process will be the same on codalab.

As I said, the current evaluation dataset is for a 'trial evaluation’, therefore you can get some idea of you system performance through the trial evaluation.

Sapna

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2019-ta...@googlegroups.com.

To post to this group, send email to semeval-2...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/semeval-2019-task-9/06bfcd76-bd8c-49c0-bc1b-77f35d16f91c%40googlegroups.com.

[Paulo Henrique da Silva]

Hi Sapna,

Thank you for your help.

Paulo

[Paulo Henrique da Silva]

Hi Sapna,

I did my registration on Codalab like you suggest but it is still pending (message: Your request to participate in this challenge has been received and a decision is pending.).

The approval is very time consuming? So I can not send the result for evaluation.

I would like to know if there is anything I can do to validate the registration or if it is possible to evaluate with the evaluation_script.py file that is in github (https://github.com/Semeval2019Task9/Subtask-A/blob/master/evaluation_script.py) but I don't have the file Goldstandard.csv.

Thanks

Paulo