Calculating IAA outside of Inception

[Raia Abu Ahmad]

Hi all! 

We had a recent problem where we lost our inception backup, and so to get the annotations we extracted the tables from the mariaDB.

Long story short - we were able to retrieve the annotations from there and have them in a format similar to this one (note that this is a multi-label annotation task):

(annotator1, document_id, list of labels (e.g. [1, 2, 5, 6])),

(annotator2, document_id, list of labels (e.g. [4, 2, 5])), 

etc...

We want to calculate the inter-annotator agreement score for this data outside of inception (we don't have a project to load it back to inception). We tried to use the nltk AnnotationTask class implementation of Kripendorff's alpha, but turns out that it only counts agreements if the lists of labels for the same document are exactly the same (meaning that the agreement for the example above will be 0, even though there are two labels in common). 

Maybe someone can help us out with how we can compute the agreement? Maybe there is a library that we can use for that that you know about?

Alternatively, maybe inception developers have some code that we can use for this?

Thanks a lot to anyone who can help!

Raia Abu Ahmad

[Richard Eckart de Castilho]

Hi,

> On 26. Sep 2023, at 16:24, Raia Abu Ahmad <raya.ab...@gmail.com> wrote:
>
> Hi all!
>
> We had a recent problem where we lost our inception backup, and so to get the annotations we extracted the tables from the mariaDB.

Sorry for that :( I hope it wasn't INCEpTION's fault... Great that you managed to recover from the data in the database!

> Long story short - we were able to retrieve the annotations from there and have them in a format similar to this one (note that this is a multi-label annotation task):
>
> (annotator1, document_id, list of labels (e.g. [1, 2, 5, 6])),
> (annotator2, document_id, list of labels (e.g. [4, 2, 5])),
> etc...
>
> We want to calculate the inter-annotator agreement score for this data outside of inception (we don't have a project to load it back to inception). We tried to use the nltk AnnotationTask class implementation of Kripendorff's alpha, but turns out that it only counts agreements if the lists of labels for the same document are exactly the same (meaning that the agreement for the example above will be 0, even though there are two labels in common).
>
> Maybe someone can help us out with how we can compute the agreement? Maybe there is a library that we can use for that that you know about?
> Alternatively, maybe inception developers have some code that we can use for this?

Maybe you could reformulate your data into a binary problem like this?

(annotator1, document_id, label_1, true)
(annotator2, document_id, label_1, false)

(annotator1, document_id, label_2, true)
(annotator2, document_id, label_2, true)

(annotator1, document_id, label_3, false)
(annotator2, document_id, label_3, false)

(annotator1, document_id, label_4, false)
(annotator2, document_id, label_4, true)

(annotator1, document_id, label_5, true)
(annotator2, document_id, label_5, true)

(annotator1, document_id, label_6, true)
(annotator2, document_id, label_6, false)

Agreement would be primarily if both annotators have a "true". You could also consider the case that both have "false" as agreement, but you might care to exclude this case.

Cheers,

-- Richard

[Torsten Zesch]

I would agree with Richard that binarization is one way to go.

In https://github.com/catalpa-cl/inceptalytics we are using the Kripendorff's alpha implementation from here https://pypi.org/project/krippendorff/
You could give that a try.

-Torsten

On 26.09.23, 18:04, "incepti...@googlegroups.com <mailto:incepti...@googlegroups.com> on behalf of Richard Eckart de Castilho" <incepti...@googlegroups.com <mailto:incepti...@googlegroups.com> on behalf of richard...@gmail.com <mailto:richard...@gmail.com>> wrote:


Hi,

--
You received this message because you are subscribed to the Google Groups "inception-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to inception-use...@googlegroups.com <mailto:inception-use...@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/inception-users/3BA791BB-038D-4D0C-84B8-1ED55669A84A%40gmail.com <https://groups.google.com/d/msgid/inception-users/3BA791BB-038D-4D0C-84B8-1ED55669A84A%40gmail.com>.