Questions on agreement scores

21 views
Skip to first unread message

Martin Lentschat

unread,
Jun 23, 2020, 8:26:58 AM6/23/20
to webanno-user

Hello,

I have a annotation project with three annotators. We use Spans and Relations to annotate our texts. Reviewing the agreement scores raised a few questions.

As I understand it, most of the different agreement scores used do not include incomplete annotations (when a position has only been annotated by one annotator and not the second). Krippendorff's Alpha may or may not include these cases. So why do all my scores have the same value ? The Krippendorff (with or without the incomplete annotations), Cohen and Fleiss all have exactly the same values.

Also, is there any way to consider agreement between overlapping annotations? This is an important issue because we have annotated many phrases, resulting in different annotations for different annotators that overlap.

Regards,
Martin

Richard Eckart de Castilho

unread,
Jun 23, 2020, 9:02:31 AM6/23/20
to Martin Lentschat, webanno-user
Hi Martin,

> Also, is there any way to consider agreement between overlapping annotations? This is an important issue because we have annotated many phrases, resulting in different annotations for different annotators that overlap.

For features on span layers, there should be the option to select the "Krippendorff's Alpha (unitizing / character offsets)" agreement which should take partial overlap into account.

> As I understand it, most of the different agreement scores used do not include incomplete annotations (when a position has only been annotated by one annotator and not the second). Krippendorff's Alpha may or may not include these cases. So why do all my scores have the same value ? The Krippendorff (with or without the incomplete annotations), Cohen and Fleiss all have exactly the same values.

For a real-world annotation project, I would assume that not all the agreement measures yield the same results. Might be you hit a bug. But before looking into that: which version of WebAnno are you using?

Cheers,

-- Richard

Martin Lentschat

unread,
Jun 23, 2020, 9:10:35 AM6/23/20
to webanno-user

Hi Richard,

I use the 3.5.5 version of WebAnno.

I have only the "Exclude incomplete" option with Krippendorff's Alpha (nominal), no "Krippendorff's Alpha (unitizing / character offsets)" measure shows.

Thank you for your help.

Richard Eckart de Castilho

unread,
Jun 23, 2020, 9:35:25 AM6/23/20
to Martin Lentschat, webanno-user
Hi,

> On 23. Jun 2020, at 15:10, Martin Lentschat <martinl...@gmail.com> wrote:
>
> I use the 3.5.5 version of WebAnno.
>
> I have only the "Exclude incomplete" option with Krippendorff's Alpha (nominal), no "Krippendorff's Alpha (unitizing / character offsets)" measure shows.

The unitizing measure is only available in WebAnno 4.0.0-beta-6 or higher
or in INCEpTION 0.13.0 (current is 0.16.1).

I would suggest you export your project from WebAnno, download the latest
INCEpTION release [1], log in to INCEpTION as admin, import your project
via the "Administration -> Projects" page with the option "import missing
users" enabled and then check out the agreement page.

Cheers,

-- Richard

[1] https://inception-project.github.io

Martin Lentschat

unread,
Jun 23, 2020, 9:40:46 AM6/23/20
to webanno-user
Thank you, I will try that immedatly
Reply all
Reply to author
Forward
0 new messages