Gold test data

[Maria Pontiki]

The evaluation datasets with the gold annotations for all four subtasks are now available at http://metashare.ilsp.gr:8080/repository/browse/semeval-2014-absa-test-data-gold-annotations/b98d11cec18211e38229842b2b6a04d77591d40acd7542b7af823a54fb03a155/

Maria

[Saif Mohammad]

Thanks Maria and the organizers of ABSA for getting this data out and running the evaluations.

We very much appreciate the efforts.

Best wishes.

-Saif

[Pablo N. Mendes]

+1 Thanks to organizers!

--
You received this message because you are subscribed to the Google Groups "SemEval-ABSA" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-absa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Pablo N. Mendes

http://pablomendes.com

[Hussam Hamdan]

thanks for the efforts.

could you please provide us with the scores of baselines for the 4 subtasks?

2014-04-11 17:14 GMT+02:00 Maria Pontiki <mpon...@gmail.com>:

The evaluation datasets with the gold annotations for all four subtasks are now available at http://metashare.ilsp.gr:8080/repository/browse/semeval-2014-absa-test-data-gold-annotations/b98d11cec18211e38229842b2b6a04d77591d40acd7542b7af823a54fb03a155/

Maria

--
You received this message because you are subscribed to the Google Groups "SemEval-ABSA" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-absa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Best Regards
Hussam Hamdan

[Dimitris Galanis]

Hello Hussam,

The current version of the baselines get the following scores:

Laptops     term extraction: F1= 35.64
Restaurants term extraction: F1= 47.15
Restaurants category extraction: F1= 63.89

Laptops     term polarity: Accuracy = 51.07
Restaurants term polarity: Accuracy = 64.28
Restaurants category polarity: Accuracy =65.65

Dimitris

[Hussam Hamdan]

ok, thanks

[blinof...@gmail.com]

Hi,

Can anyone tell what are the reasons for choosing Accuracy as primary measure for the polarity detection subtasks? It seems that Accuracy not very suitable (compared to F-measure) for skewed test data.

Could organisers provide F-measure results for polarity detection subtasks for the baseline system?

Thank you!

[Joachim Wagner]

Hi organisers,

Will the baseline systems that produce the above results be described in the task overview paper?

If yes, can you paste the reference entry here? I remember seeing it in another thread but it would be handy for all to have it here as well.

Joachim

[Dimitris Galanis]

Hello Pavel,

We use accuracy as the primary evaluation measure for the polarity detection subtasks, because it is widely used
and it provides an overall score for all four categories together (positive, negative, neutral, conflict).
The evaluation script that we provided also calculates F1/Precision/Recall for each class, as well as the confusion matrix.
Attached you will find the F1/Precision/Recall scores for the baselines.


Dimitris

[blinof...@gmail.com]

Thank you!

[Maria Pontiki]

Hello Joachim,

Yes, the baseline systems will be described in the task overview paper.
The reference should look like this:
 
Maria Pontiki, Dimitrios Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar. SemEval-2014 Task 4: Aspect Based Sentiment Analysis. Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval), 2014.

Maria

[Aitor García]

Hello Dimitris,

I have a small question. On which particular data have you run the baselines to obtain those scores?
When I run the provided baselines scripts on the test set with the gold annotations I get different scores. Same happens if I run them on the train_v2 datasets.

Best regards,

Aitor.

[Dimitris Galanis]

Hello

The script for laptops is the following:

#!/bin/bash
clear
trainf=Laptop_Train_v2.xml
testf=Laptops_Test_DataD_finalD.xml.gold
A=absa--test.predicted-aspect.xml
B=absa--test.predicted-aspectPolar.xml
echo -e "\n\n--- Running baseline for SB 1"
python ../semeval_base.py --train $trainf -k $testf --task 1
echo -e "\n\n--- Validating " $A
java -cp ../eval.jar Main.Valid $A  ../SemEvalSchema.xsd
echo -e "\n\n--- Evaluating " $A
java -cp ../eval.jar Main.Aspects $A $testf
echo -e "\n\n--- Running baseline for SB 2"
python ../semeval_base.py --train $trainf -k $testf --task 3
echo -e "\n\n--- Validating " $B
java -cp ../eval.jar Main.Valid $B  ../SemEvalSchema.xsd
echo -e "\n\n--- Evaluating " $B
java -cp ../eval.jar Main.Polarity $B $testf

The baseline learns from Laptop_Train_v2.xml (training data) and predicts the terms and polarities for the test sentences.

Then it compares the predicted with the gold and calculates the evaluation measures.

Dimitris.

[Dimitris Galanis]

I forgot to say that the detailed evaluation scores (F1/P/R per class) are calculated by the eval.jar.

Dimitris

[Aitor García]

Thank you very much :-)

[Pramodith Bprum]

Hi, I can't seem to download the Gold Test data, I've created an account on metashare but when I try to login it says that the account is inactive, is there some other url I can use to download the Gold tags?