How to calculate krippendorff alpha with MASI distance

[elgesto elge]

Hello,

I have a task to calculate inter-annotator agreement in multi-label classification, where for each example more than one label can be assigned. I found that NLTK can measure agreement based on distance metric.

I am looking for an example of calculating krippendorff alpha with MASI distance.

This is what I have.

import nltk
from nltk.metrics import masi_distance


toy_data = [['1', 5723, [1,2]],['2', 5723, [2,3]]]

task = nltk.metrics.agreement.AnnotationTask(data=toy_data, distance=masi_distance)
print task.alpha()

This code fails with 

TypeError: unhashable type: 'list'

Obviously, the following doesn't work to.

toy_data = [['1', 5723, set([1,2])],['2', 5723, set([2,3])]]

Do you have a working example?

Thank you!

[Steven Bird]

From the documentation, it looks like each item needs to be immutable, i.e. your `toy_data` shouldn't consist of lists but tuples.

http://www.nltk.org/_modules/nltk/metrics/agreement.html

It would be great if someone could contribute more documentation, e.g. here:

http://www.nltk.org/_modules/nltk/metrics/agreement.html#AnnotationTask.alpha

--
You received this message because you are subscribed to the Google Groups "nltk-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nltk-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

--
Sent from my phone, please excuse miscorrected typos

[elgesto elge]

Tuples are not appropriate.

It fails with error message

AttributeError: 'tuple' object has no attribute 'intersection'

[Alexis Dimitriadis]

Sounds like the code expects `frozenset` objects: They have an intersection() method and are immutable.

It also sounds like it is badly in need of more documentation...

Alexis

--

Dr. Alexis Dimitriadis | Senior research fellow | Utrecht Institute of Linguistics OTS | Utrecht University | Trans 10, 3512 JK Utrecht, room 2.33 | +31 30 253 65 68 | a.dimi...@uu.nl | www.hum.uu.nl/medewerkers/a.dimitriadis 

[elgesto elge]

You are right, it's frozenset. For some reason it still doesn't work.

toy_data = [['1', 5723, frozenset([2,3])],['2', 5723, frozenset([2,3])]]

fails with

ret = 1.0 - (self.Do_alpha() / De)
ZeroDivisionError: float division by zero