Agreement : "Error: An annotation study needs at least two different categories; otherwise there is no decision for the raters to agree on"

[Lucie G.]

Hello,

I am trying to compute inter annotator agreements on a custom polarity layer and tagset
(annotation granularity set to sentence-level and is allowed to cross sentence boundaries) and I get the following error for Cohen's and Fleiss Kappa (no problem for Kripendorff's Alpha and Kappa) :

Error: An annotation study needs at least two different categories; otherwise there is no decision for the raters to agree on

I don't understand because the polarity tagset contains more than two categories.

Any help ?

Thanks

[Richard Eckart de Castilho]

Hi,

the current agreement calculation code looks at the actually annotated data but not at
the tagset. So if you actually annotate only "positive", then it only sees one label
and yields the error.

I think this is a bug and the way it should be is that the agreement code should look
at a potentially existing tagset and take the additional labels defined by it into
account. Hopefully, that would fix the problem.

Here is an issue describing the problem: https://github.com/inception-project/inception/issues/2191

You can subscribe to the issue to get updates for it and/or provide additional information
if you would like.

Cheers,

-- Richard

[Lucie G.]

Hi Richard,

Thank you very much for your answer.

However, different tags have been annotated, and annotators do not always agree.

I have since thought that the setting of the project might be the problem: I have named the layer, the feature layer and the tagset all with the same name. Might be the issue maybe ?

Thanks

Lucie

[Richard Eckart de Castilho]

Hi,

> On 13. Apr 2021, at 12:11, Lucie G. <lucie....@gmail.com> wrote:
>
> However, different tags have been annotated, and annotators do not always agree.
>
> I have since thought that the setting of the project might be the problem: I have named the layer, the feature layer and the tagset all with the same name. Might be the issue maybe ?

Shouldn't be a problem. Could you for convenience provide a project export by which I could reproduce your scenario?

You could mail it to me directly.

Cheers,

-- Richard

[Richard Eckart de Castilho]

Hi,

> On 13. Apr 2021, at 13:24, Richard Eckart de Castilho <richard...@gmail.com> wrote:
>
>> On 13. Apr 2021, at 12:11, Lucie G. <lucie....@gmail.com> wrote:
>>
>> However, different tags have been annotated, and annotators do not always agree.

thanks for the data. I check and my fix seems to work for it. The issue is that
the agreement is calculated pairwise between the annotators and in some cases,
there is no disagreement in a particular pair - then you get the error.

The fix will be in the next release.

Cheers,

-- Richard