feedback needed: phases of evaluation for Clinical TempEval 2017

[Bethard, Steven John - (bethard)]

I’d like your feedback on what kind of phases we’d like in the evaluation for Clinical TempEval 2017. There are two choices:

1. In phase 1, only the raw text is provided for the target domain (brain cancer) test data to be annotated, and the evaluation is on events, times, and temporal relations. In phase 2, manual EVENT and TIMEX3 annotations for the test data are also provided, and the evaluation is restricted to temporal relations.

2. In phase 1 and phase 2, only the raw text is provided for the target domain (brain cancer) test data to be annotated, and the evaluation is on events, times, and temporal relations. For phase 2, we would separately release additional training data in the target domain (brain cancer).

I personally prefer option 2, since that would allow people to try both unsupervised and supervised forms of domain adaptation, while option 1 would only allow unsupervised forms of domain adaptation.

What is your preference?

Steven Bethard
Assistant Professor
School of Information
University of Arizona

[Manikandan R]

Steven,

1. According to my understanding of option 2 we get two sets of brain cancer sets in phase 2, one  set for training (with annotation) and other set for testing(without annotations).
   
2. If so option 2 feels fine as we can test our so  model(trained on colon cancer data only) with brain cancer data set & it would be a better form of evaluation of domain adaptability of our systems considering system has not seen data of brain cancer at all during system development phase. Any reason for having a additional set of brain cancer other than to have supervision of brain cancer data in training? 
   
3. Yet I can see some loss of flexibility in participating only in phase 2 like last year given annotations of event and timex3, which was a major plus point of clinical tempeval.

[Bethard, Steven John - (bethard)]

Yes, with option 2, there would be brain cancer annotated data released as additional training data during phase 2 of the evaluation (probably 16 Jan 2016 through 30 Jan 2016). With option 1, there would never be any brain cancer annotated data available for training.

 

One way to address the concern about not having to build a system for EVENTs and TIMEX3s might be to release the predicted EVENTs and TIMEX3s of some system. I could do that for the baseline system, and if someone with a better system was willing to do so as well, I’m sure we could find a way to host them somewhere.

 

Steve

--
You received this message because you are subscribed to the Google Groups "clinical-tempeval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clinical-tempe...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bethard, Steven John - (bethard)]

I only heard agreement with my preference for option 2, so I’ve updated the task website accordingly:

http://alt.qcri.org/semeval2017/task12/index.php?id=evaluation

Steve