Using agreement metrics with missing data?

Skip to first unread message

P Resnik

Aug 24, 2021, 12:19:33 PM8/24/21
to nltk-users
The documentation in the Agreement Metrics package indicates that for computation of inter-annotator agreement, every coder needs to have coded every item:

Note that the data list needs to contain the same number of triples for each individual coder, containing category values for the same set of items.

However,  there's also this tantalizing note in the documentation:

TODO: Describe handling of multiple coders and missing data

I'm attaching code illustrating the error that gets raised when you have some coder who did not code some items.  Curious whether anyone has a solution or work-around for using this package under those circumstances?  My specific goal is to evaluate agreement in a multi-label setting using MASI.

Thanks for any thoughts!

Reply all
Reply to author
0 new messages