Using agreement metrics with missing data?

11 views
Skip to first unread message

P Resnik

unread,
Aug 24, 2021, 12:19:33 PMAug 24
to nltk-users
The documentation in the Agreement Metrics package indicates that for computation of inter-annotator agreement, every coder needs to have coded every item:

Note that the data list needs to contain the same number of triples for each individual coder, containing category values for the same set of items.

However,  there's also this tantalizing note in the documentation:

TODO: Describe handling of multiple coders and missing data

I'm attaching code illustrating the error that gets raised when you have some coder who did not code some items.  Curious whether anyone has a solution or work-around for using this package under those circumstances?  My specific goal is to evaluate agreement in a multi-label setting using MASI.

Thanks for any thoughts!

  Philip

reproducing_problem.py
Reply all
Reply to author
Forward
0 new messages