Semantic Sentiment Analysis Challenge @ ESWC 2016 - Task #2 Training Dataset Released

[Mauro Dragoni]

Dear All,

the training dataset for the Task #2 of the Semantic Sentiment Analysis challenge of ESWC 2016 has been released.

The dataset contains reviews from the Laptop and Restaurant domains.

For each review, the set of aspects and the associated polarities are provided.

The test set will be released during the month of April and it will contain reviews coming from the same domains.

Apart from this training set, you are allowed to use any other resource and knowledge base for building your models.

Please follow the links below for downloading the dataset and for further resources.

Challenge information: http://2016.eswc-conferences.org/eswc-16-challenge-semantic-sentiment-analysis

Training dataset: https://www.maurodragoni.com/research/opinionmining/datasets/challenge2016/task2.zip

Resources (not limited to these ones)

Sentic.Net: http://sentic.net/downloads/

General Inquirer: http://www.wjh.harvard.edu/~inquirer/homecat.htm

Even if you are not planning to compete in the challenge, please consider to adopt the dataset also for training and evaluating your approaches.

Submissions to the Semantic Sentiment Analysis workshop are welcome.

Workshop information: http://www.maurodragoni.com/research/opinionmining/events/

For any question, do not hesitate to contact the chairs.

Kind regards.

[Soufian Jebbara]

Hi Mauro,

I am very interested in Task #2 of this years challenge but still have trouble understanding the task in full detail.

Here are a few of my questions:

The github wiki page of this task describes it as an aspect extraction task where each sentence is assigned a set of aspect and polarity pairs.

The full set of the possible aspects (e.g. "speaker", "touchscreen","camera", ...) seems to be predefined and the individual aspects need not be explicitly mentioned in the text (no "from" and "to" attributes), correct?

Polarity is said to be binary ("positive" and "negative").

However, the released training data (from the github repo) seems to suggest that it is not relevant to categorize the aspect mentions into predefined categories (as mentioned before).

However, the exact position of an aspect in the sentence (if applicable) needs to be extracted.

Polarity can be either "positive", "neutral" or "negative".

Which on of the two settings is it going to be?

Last question:

Is the aspect extraction evaluated separately from the polarity extraction or are they evaluated jointly as one?

I hope you can help me with some of my questions.

Kind regards,

Soufian

[Mauro Dragoni]

Hi Soufian,

thanks for your interest in participating the challenge.

The second setting is the right one.

The list of aspect is not predefined, but you need to extract the aspect from the text.

The position of the aspect is not mandatory because in the test set, we will not provide ambiguous sentences.

Concerning polarity, the "neutral" one will not be considered in the test set.

Aspect extraction and polarity will be evaluated separately, but systems will be ranked by aggregating the two f-measures in the same way as f-measure is computed.

Example:

given Fa the f-measure computed on the aspect extraction and Fp the f-measure computed on the polarity inference computed only on the aspect you extracted correctly;

the system score will be computed as:

S = 2 * ((Fa * Fp) / (Fa + Fp))

Hope to solve your doubts.

Kind regards,

Mauro.

[Soufian Jebbara]

Hi again,

thank your for your clarifying answer. It helped a lot!

Yet, I do have another question if that is Ok:

I see that the dataset, especially the restaurant domain, contains entries like:

<sentence id="1014458:3">

   <text>The wine list is interesting and has many good values.</text>

   <Opinions>

        <Opinion aspect="wine list" polarity="positive" from="4" to="13"/>

        <Opinion aspect="wine list" polarity="positive" from="4" to="13"/>

   </Opinions>

</sentence>

<sentence id="1014458:4">

   <text>For the price, you cannot eat this well in Manhattan.</text>

   <Opinions>

        <Opinion aspect="NULL" polarity="positive" from="0" to="0"/>

        <Opinion aspect="NULL" polarity="positive" from="0" to="0"/>

   </Opinions>

</sentence>

<sentence id="1632445:5">

   <text>Even though its good seafood, the prices are too high.</text>

   <Opinions>

        <Opinion aspect="seafood" polarity="positive" from="21" to="28"/>

        <Opinion aspect="seafood" polarity="negative" from="21" to="28"/>

   </Opinions>

</sentence>

I assume, a competing system will need to reproduce all annotations, including seemingly identical and conflicting ones for a given text part, am I correct?

So, missing one of the annotations, say

        <Opinion aspect="wine list" polarity="positive" from="4" to="13"/>

would result in a lower score for both the aspect and the polarity extraction!? 

I just have some difficulties implementing myself an adequate evaluation measure.

Kind regards,

Soufian

[Mauro Dragoni]

Hi Soufian,

we will consider ONLY distinct annotations in our test set.

Therefore, your interpretation is correct and your result will not generate a lower score.

Kind regards,

Mauro.