Questions on agreement scores

[Martin Lentschat]

Hello,

I have a annotation project with three annotators. We use Spans and Relations to annotate our texts. Reviewing the agreement scores raised a few questions.

As I understand it, most of the different agreement scores used do not include incomplete annotations (when a position has only been annotated by one annotator and not the second). Krippendorff's Alpha may or may not include these cases. So why do all my scores have the same value ? The Krippendorff (with or without the incomplete annotations), Cohen and Fleiss all have exactly the same values.

Also, is there any way to consider agreement between overlapping annotations? This is an important issue because we have annotated many phrases, resulting in different annotations for different annotators that overlap.

Regards,

Martin

[Richard Eckart de Castilho]

Hi Martin,

> Also, is there any way to consider agreement between overlapping annotations? This is an important issue because we have annotated many phrases, resulting in different annotations for different annotators that overlap.

For features on span layers, there should be the option to select the "Krippendorff's Alpha (unitizing / character offsets)" agreement which should take partial overlap into account.

> As I understand it, most of the different agreement scores used do not include incomplete annotations (when a position has only been annotated by one annotator and not the second). Krippendorff's Alpha may or may not include these cases. So why do all my scores have the same value ? The Krippendorff (with or without the incomplete annotations), Cohen and Fleiss all have exactly the same values.

For a real-world annotation project, I would assume that not all the agreement measures yield the same results. Might be you hit a bug. But before looking into that: which version of WebAnno are you using?

Cheers,

-- Richard

[Martin Lentschat]

Hi Richard,

I use the 3.5.5 version of WebAnno.

I have only the "Exclude incomplete" option with Krippendorff's Alpha (nominal), no "Krippendorff's Alpha (unitizing / character offsets)" measure shows.

Thank you for your help.

[Richard Eckart de Castilho]

Hi,

> On 23. Jun 2020, at 15:10, Martin Lentschat <martinl...@gmail.com> wrote:
>
> I use the 3.5.5 version of WebAnno.
>
> I have only the "Exclude incomplete" option with Krippendorff's Alpha (nominal), no "Krippendorff's Alpha (unitizing / character offsets)" measure shows.

The unitizing measure is only available in WebAnno 4.0.0-beta-6 or higher
or in INCEpTION 0.13.0 (current is 0.16.1).

I would suggest you export your project from WebAnno, download the latest
INCEpTION release [1], log in to INCEpTION as admin, import your project
via the "Administration -> Projects" page with the option "import missing
users" enabled and then check out the agreement page.

Cheers,

-- Richard

[1] https://inception-project.github.io

[Martin Lentschat]

Thank you, I will try that immedatly