NEEL Evaluation Framework

[#Microposts2014 Chairs]

Dear #Microposts2014 NEEL Challenge participants,

Please find attached the Evaluation Framework.

How to run:
java -java neeleval.jar /path/goldstandard /path/testset

Both the gold standard (GS) and the test set (TS) have to be well
formatted tab separated files, where each line consists of:
tweet_id
entity_mention_1
entity_uri_1
...
entity_mention_n
entity_uri_n

The source code of the evaluation framework is available at:
https://github.com/giusepperizzo/neeleval.

Examples are provided in:
- test package:
https://github.com/giusepperizzo/neeleval/tree/master/src/test/java/neeleval
- utils folder: https://github.com/giusepperizzo/neeleval/tree/master/utils

We encourage you to open a new issue on GitHub if you spot a bug
https://github.com/giusepperizzo/neeleval/issues .
We do encourage patches, so in case you quickly come up with the patch,
please do a pull. We will merge the changes if they are needed.

Thanks for your collaboration and we wish good luck for your submissions.

Cheers,
#Microposts2014 NEEL Challenge crew

[Chang Ming-Wei]

I have some questions about the evaluation framework. I feel that the evaluation script should handle the insertion or the deletion of the entities in a better way.

For example, assume the tweet is

nfl: giants vs packers

, and the gold annotation is as follows:

0001         nfl         NFL        giant           New_York_Giants        packers      Green_Bay_Packers

Assume a system predicts:

0001         giant           New_York_Giants        vs       VS (maganize) packers      Green_Bay_Packers

What kind of score will a system get? From the code snippet (included below), it seems that it will get a zero score. However, I am not sure that is a fair comparison, given that the system in fact gets the two entities correctly. Maybe some forms of alignment are needed here.

 for (int i = 0; i < gs_pairs.size() && i < ts_pairs.size(); i++)

                    no_correct += gs_pairs.get(i).compareTo(ts_pairs.get(i));    

[Mena Badieh Habib Morgan]

I agree with Chang ... the evaluation method considers correct pair a false one if other false  pairs appeared before the correct one.
I think you should not assume that both the gs_pairs and ts_pairs are of the same size.

Mena

[#Microposts2014 Chairs]

Dear Chang, Mena, #Microposts2014 NEEL Challenge participants,

we have updated the evaluation framework with a logic that considers the
longest ordered common subsequence.
Given:
GS = A B C
TS = B D C
The new logic counts 2 TPs.

Attached you can find the library, while the source code is available at:
https://github.com/giusepperizzo/neeleval

The logic has been co-updated together with Stefano Parmesan (thanks
Stafano).

Cheers,
#Microposts2014 NEEL Challenge crew

On 19/02/14 09:40, Mena Badieh Habib Morgan wrote:
> I agree with Chang ... the evaluation method considers correct pair a
> false one if other false pairs appeared before the correct one.

> I think you should not assume that both the gs_pairsand ts_pairsare of

> <https://github.com/giusepperizzo/neeleval>.

>
> Examples are provided in:
> - test package:
> https://github.com/giusepperizzo/neeleval/tree/master/src/test/java/neeleval
> <https://github.com/giusepperizzo/neeleval/tree/master/src/test/java/neeleval>
>
> - utils folder:
> https://github.com/giusepperizzo/neeleval/tree/master/utils
> <https://github.com/giusepperizzo/neeleval/tree/master/utils>
>
> We encourage you to open a new issue on GitHub if you spot a bug
> https://github.com/giusepperizzo/neeleval/issues
> <https://github.com/giusepperizzo/neeleval/issues> .
> We do encourage patches, so in case you quickly come up with the
> patch,
> please do a pull. We will merge the changes if they are needed.
>
> Thanks for your collaboration and we wish good luck for your
> submissions.
>
> Cheers,
> #Microposts2014 NEEL Challenge crew
>

> --
> You received this message because you are subscribed to the Google
> Groups "microposts2014" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to microposts201...@googlegroups.com.
> Visit this group at http://groups.google.com/group/microposts2014.
> For more options, visit https://groups.google.com/groups/opt_out.