please test your systems on CodaLab

[Bethard, Steven John - (bethard)]

Happy holidays!

This is just a gentle reminder to start testing your systems on CodaLab (https://competitions.codalab.org/competitions/15621). The official evaluation period will begin in about 10 days, but you can already try your systems out on the trial data. Once the official evaluation begins, you will only be able to upload system output twice, so it’s very important that you debug your system’s output format now on the trial data.

The trial data is just the “Test” portion of the colon cancer notes, so you should train your system on “Train” and “Dev” colon cancer notes, run your system on the raw text of the “Test” colon cancer notes, and upload the Anafora XML files that your system produces on that “Test” data.

Steven Bethard
Assistant Professor
School of Information
University of Arizona

[Manikandan R]

Steve,

Just some last minute queries regarding evaluation and data release period 

Evaluation period:

·         During SemEval 2016 we had around a week time for each  evaluation phase, this time in Codalab only start date is mentioned. Just wanted to confirm if we similar evaluation period or should we submit on the date mentioned in Codalab.

Target domain data availability

·         Just wanted to know when would be each of train and test sets be released. 

Any updates on time frame(Start and End) of data release and evaluation phases would be helpful.

[Bethard, Steven John - (bethard)]

On 1/2/17, 00:05, Manikandan R wrote:
> During SemEval 2016 we had around a week time for each  evaluation phase, this time in Codalab only start date is mentioned. Just wanted to confirm if we similar evaluation period or should we submit on the date mentioned in Codalab.

The Phases tab shows all the dates:

https://competitions.codalab.org/competitions/15621#phases

In short, there will be one week to submit systems to the “Unsupervised Domain Adaptation” phase, i.e., based on all the annotated colon cancer that has already been released and the unlabeled brain cancer data. Then, there will be two weeks to submit systems to the “Supervised Domain Adaptation” phase.

> Just wanted to know when would be each of train and test sets be released. 

The training data for the “Supervised Domain Adaptation” phase will be released when that phase begins, on Jan. 16, 2017. The plain text for the test data has already been released (it is the doc* files in the Test zip file); the annotations for the test data will be released once the competition is complete.

Note that since the plain text for the test set has been already released (due to the format in which the Mayo Clinic distributed the data), we ask participants to be extra careful not to somehow tune their systems to that data, and when we ask you all for system descriptions to give some details exactly how you used any of the unlabeled data (if you used it).

Steve