CodaLab competition site for Clinical TempEval 2017 now live

[Bethard, Steven John - (bethard)]

I have set up the CodaLab competition site where participants will be submitting their system output:

https://competitions.codalab.org/competitions/15621?secret_key=66712e6a-0548-4844-a16c-d8a336f4e4ab

The “Trial” phase is running right now, which means that it is evaluating system output against the THYME colon cancer test set (i.e., the same test set that was used for Clinical TempEval 2016).

Please give it a try; the sooner we can iron out any bugs, the smoother the real evaluation will run in January!

Steven Bethard
Assistant Professor
School of Information
University of Arizona

[André Lamúrias]

Hi Steven,

I'm not sure if I am using this tool correctly. When I click on "View scoring output log" I only get "None" (same with error log). However clicking on  "Download evaluation output from scoring step" I can see my results. Is this supposed to happen?

Also, is there a way to delete submissions?

Thanks,

[Bethard, Steven John - (bethard)]

Yes, that’s correct. The “scoring output log” and “scoring error log” only show up if the evaluation script writes something to standard output or standard error, which would typically only happen if there was something wrong with your submission. The scores that your submission got are indeed in the “evaluation output from scoring step” download, specifically, in the “scores.txt” file.

 

There’s no way for a participant to delete a submission, but you do get to select which of your submissions gets posted to the leaderboard; there should be a “Submit to Leaderboard” button alongside each submission on the “Participate” page. The “Trial” phase is set to allow you up to 100 submissions though, so hopefully that should be plenty of submissions to test out your setup. If you still really need to delete a submission, let me know; as an administrator, I can do that.

 

Steve

--
You received this message because you are subscribed to the Google Groups "clinical-tempeval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clinical-tempe...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Manikandan R]

Hi Steven,

We have registered in codalab, but however there is no option I could see for submission, also under participate section it says "Your request to participate in this challenge has been received and a decision is pending".

Please note that we had signed DUA during SemEval 2016 evaluation. Should we again do a DUA for 2017? 

Any help on this would be appreciated.

best,

Man

[Bethard, Steven John - (bethard)]

Right, with the CodaLab registration, you have to wait until the task organizer approves your request. Looks like you requested a few hours ago; I just got into work and have approved your request. You should be able to submit now.

Steve

[Manikandan R]

Steven,

Thank you for replying to queries regarding Codalab. 

Since we have 2 phases namely  Phase 1-unsupervised domain adaptation and Phase 2-supervised domain adaptation this time.

Is it necessary to participate on phase 1 to participate in phase 2 or can a team participate in phase 2 directly?

Best,

Mani

[Bethard, Steven John - (bethard)]

A team should be able to participate in just phase 2, skipping phase 1.

Steve