SemEval-2014 Task 4 overlapped categories and aspect terms

[omar mustafa]

Hello

 

As matter of fact I'm currently working forward to extract the aspect/opinion-target from SemEval 2014 dataset, I’m trying to extract and aggregate the aspects from online reviews using topic modeling clustering algorithm LDA. In such algorithm labeled dataset is required to evaluate the performance of the proposed model, and you have clear idea that SemEval 2014 is unlabeled dataset except Restaurant review dataset it has category label. But, the big elephant in this dataset which I have realised recently is that there are multiple aspect category for the same aspect terms as shown in the attached photo from Restaurant reviews dataset, there are multiple categories for multiple aspect terms. This thing drive me crazy. In addition to that, I have to use Rand index evaluation measure to evaluate the proposed clustering algorithm. where Rand index was used to evaluate the similarity of the LDA output which is clusters in this case against the gold standard dataset. in other word, it represents the number of times two data points belong to the same cluster in both produced clusters of LDA and gold/ standard dataset SemEval 2014. in the literature authors have claimed that they have used manually labeled dataset by labelling 6 clusters which is equal to the number of categories in SemEval 2014. The aspect categories are FOOD, SERVICE, PRICE, AMBIENCE and ANECDOTES. The big question here is that how to manage to label overlapped aspect terms to their specific category or cluster???. 

 

I would be thankful if you can help me understand how to evaluate the performance of LDA using Rand index against  SemEval 2014 dataset, where there are multilabel for same aspect term. ??

 

Please help me to find labeled SemEval 2014 dataset, or suggest a solution for this matter!!

 

Regards

Omar

<sentence id="11432316#715219#2">

        <text>It was pleasantly uncrowded, the service was delightful, the garden adorable, the food (from appetizers to entrees) was delectable.</text>

        <aspectTerms>

            <aspectTerm term="service" from="33" to="40"/>

            <aspectTerm term="garden" from="61" to="67"/>

            <aspectTerm term="food" from="82" to="86"/>

            <aspectTerm term="appetizers" from="93" to="103"/>

            <aspectTerm term="entrees" from="107" to="114"/>

        </aspectTerms>

        <aspectCategories>

            <aspectCategory category="ambience"/>

            <aspectCategory category="service"/>

            <aspectCategory category="food"/>

        </aspectCategories>

    </sentence>