> * Several possible TIMEX3 are sometimes annotated with SECTIONTIME or DOCTIME entities. If our system annotate these with TIMEX3 entities, does it impact the evaluation score ? In other words, does the evaluation script take care of this situation ?
The scoring script ignores SECTIONTIME and DOCTIME, and scores only the TIMEX3s. The overlapping SECTIONTIME and TIMEX3 is a result of changing annotation guidelines over time, but I’ve checked the test data, and there don’t appear to be any SECTIONTIMEs overlapping with TIMEX3s in the test data (which I believe is correct, given the final annotation guidelines). So you should tune your systems accordingly.
> * As mentioned in the Annotation Guidelines, several parts related to medication are not annotated. Does the evaluation script ignore annotations that could be provided by the system for these parts ? Should we make sure that these parts stay empty ?
Great question! The evaluation script does not know about the different section types, so yes, your systems should not produce annotations on those parts. If you want to see an example of doing that, there’s one in Apache cTAKES:
https://github.com/apache/ctakes/blob/trunk/ctakes-temporal/src/main/java/org/apache/ctakes/temporal/ae/TemporalEntityAnnotator_ImplBase.java
If you follow that code through, the specific segments to skip are "20104", "20105", "20116", and "20138".
> * Some parts of the annotation are annotated with DUPLICATE entities. How the evaluation script deals with these parts if they exist in the test corpus ?
The only entities that are evaluated by the evaluation script are TIMEX3, EVENT, and TLINK:
https://github.com/bethard/clinical-tempeval/blob/master/program/evaluate.py
Everything else (including DUPLICATE entities) is ignored.
Steve