question on creating answer file in 2010 format from gold standard

[Ted Pedersen]

I'd like to create an answer file using the 2010 format, so I can run
that answer file against the gold standard and hopefully get the best
case (gold) scoring.

What I did to do that was take the STRel.txt file (portion below to
give the idea) :

1.1 1.1
1.1 1.2
1.1 1.3
1.1 1.4
1.1 1.5
1.1 1.9
1.1 1.10
1.1 1.11
1.1 1.13
1.1 1.14
1.1 1.23
1.1 1.32
1.1 1.39
1.1 1.49
1.1 1.50
1.1 1.61
1.2 1.6
1.2 1.16
1.2 1.20
1.2 1.21
1.2 1.22

and then convert that into the 2010 format...

aetna.n aetna.n.1 aetna.n.1
aetna.n aetna.n.2 aetna.n.1
aetna.n aetna.n.3 aetna.n.1
aetna.n aetna.n.4 aetna.n.1
aetna.n aetna.n.5 aetna.n.1
aetna.n aetna.n.9 aetna.n.1
aetna.n aetna.n.10 aetna.n.1
aetna.n aetna.n.11 aetna.n.1
aetna.n aetna.n.13 aetna.n.1
aetna.n aetna.n.14 aetna.n.1
aetna.n aetna.n.23 aetna.n.1
aetna.n aetna.n.32 aetna.n.1
aetna.n aetna.n.39 aetna.n.1
aetna.n aetna.n.49 aetna.n.1
aetna.n aetna.n.50 aetna.n.1
aetna.n aetna.n.61 aetna.n.1
aetna.n aetna.n.6 aetna.n.2
aetna.n aetna.n.16 aetna.n.2
aetna.n aetna.n.20 aetna.n.2
aetna.n aetna.n.21 aetna.n.2
aetna.n aetna.n.22 aetna.n.2

Does it look like I'm doing the conversion correctly? If I call my
2010 formatted answer file newKey.txt and then I run against the
actual gold standard, I get values that don't really look like "best
case" :

java -jar WSI-Evaluator.jar ../gold/ ../gold/newKey.txt


average F1 = 0.58453125
average Rand Index = 0.5220634920634919
average Adj Rand Index = 0.0021988323527870374
average Jaccard Index = 0.22428047555683117
============ average number of created clusters: 7.69
============ average cluster size: 11.562983289843968

What makes me think I'm not totally wrong in what I'm doing is that my
average number of clusters and average cluster size do correspond to
what the gold standard gets in the evaluation results, but the values
for F1, ARI, etc. are very different indeed.

What have I missed here??

Thanks!
Ted

[Ted Pedersen]

BTW, I have put my entire "key" file here:

http://www.d.umn.edu/~tpederse/newKey.txt

So the idea is to create an answer file in 2010 format that should get
a "perfect" score on the gold data, just as a kind of sanity check....
Any ideas why this isn't working quite as I am hoping...?

Thanks!
Ted

[Daniele Vannella]

Ted, I guess you should follow the following format:

> Date: Fri, 5 Apr 2013 22:36:06 -0500
> Subject: Re: question on creating answer file in 2010 format from gold standard
> From: tped...@d.umn.edu
> To: semeval-2013-ws...@googlegroups.com

> --
> You received this message because you are subscribed to the Google Groups "Semeval-2013 Task 11: WSI & Disambiguation within An Application" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2013-wsi-in-a...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Ted Pedersen]

Ah, thank you! I see my mistake - I had mapped the queries assuming an
alphabetical ordering (in my gold standard) rather than following the
order as shown in topics.txt. So I had assumed, for example, that
topic 1 was aetna, when in fact it was polaroid. It would have taken
me quite a while to figure that out on my own I am sure. Thanks for
the quick response!

Cordially,
Ted