Issues with the Evaluation Script

[Usama Yaseen]

Hi All,

We encountered strange issues with the evaluation script, I am having trouble understanding how does the pairing works; our situation is that we are getting an error with debug level

SERIOUS with error 'type error'. On forcefully evaluating with the detailed flag I noticed a strange behavior with how pairing was done,

document: BB-norm+ner-9745160

This is extracted from the prediction file:

T1 Habitat 59 73 duodenal ulcer

T2 Microorganism 79 98 Helicobacter pylori

T22 Microorganism 2358 2366 H pylori

N1 OntoBiotope Annotation:T1 Referent:OBT:001533

N2 NCBI_Taxonomy Annotation:T2 Referent:210

N22 OntoBiotope Annotation:T14 Referent:OBT:003269

And the corresponding relevant entries in the annotation file are:

T7 Habitat 59 73 duodenal ulcer

T8      Microorganism 79 98 Helicobacter pylori

T31 Microorganism 2358 2366 H pylori

N2 OntoBiotope Annotation:T7 Referent:OBT:001533

N3 NCBI_Taxonomy Annotation:T8 Referent:210

N26 NCBI_Taxonomy Annotation:T31 Referent:210

And this is how the detailed output looks like (only the relevant part):

--   T2-83fd973b-c6ed-4945-b3fc-7d0df7419a6c 0.0000

--   T1-b0479f43-e4ac-475f-93f7-52abe8ffdc5d 0.0000

T31  T22  1.0000

T1 & T2 cannot be paired but T31 & T22 does which is weird because T1 should be paired with T7 and T2 should be paired to T8, every thing is same except the entity index in the beginning which should not matter anyway.

Also what exactly is this 'T1-b0479f43-e4ac-475f-93f7-52abe8ffdc5d' ? (after T1-, is it some kind of hash?)

Unfortunately, with this error we don't know how to proceed further as apparently everything seems right.

Also in the ground annotations the starting index of entities is somewhat random, for some it's T3 for some it's T6, but that should not make any difference?

Would be great if anyone can give a hint of what could be wrong.

PS: This post is very specific to the error we are getting and probably should only be discussed with email. I have emailed Robert as well but since it's a weekend and there is not enough time left so I am also spamming here; sorry if it's not a relevant post.

Many Thanks!

Kind Regards,

Usama

[Louise Deleger]

Hi Usama,

We are sorry about this. I don't think Robert will have time to do anything about it before Monday. But it will be fixed for the evaluation of your predictions on the test set.

In the meantime, I suggest you start the index of your entities at a higher number, say 10. Since the issue seems to occur with the first entities, changing their index might solve the problem.

Hope that helps.

Best regards,

Louise

[Louise Deleger]

Following up on the issue:

if you look at the .a1 files, I think you'll find some pre-existing "Title" and "Abstract" annotations, with T1, T2... indices. I think the issue occurs if you re-assign their indices to the entities you're predicting. So if you start your index at a higher number, you should be fine.

[Usama Yaseen]

Hi Louise,

I am trying with the increased indices and hopefully it will work (unfortunately I won't be available for next couple of hours so I will give an update afterwards).

Thanks alot.

Best,

Usama

[Robert Bossy]

Hi Usama,

sorry I couldn't handle your issue earlier.

I'm nearly hundred percent positive that the fact that predictions reuse identifiers of annotations given as input causes your problems.

Try starting your predicted entities at T3 and tell me if it's better.

Cheers

Robert

[Usama Yaseen]

Hi Robert & Louise,

Sorry for extremely late reply. I changed the starting index of entities to 11 and it fixed the problem as you guys thought (in the annotation file some entities start with 6 index as well so it's probably safer to set the starting index to 10/11)

Many Thanks for your support.

Best,

Usama