The documentation in the Agreement Metrics
package indicates that for computation of inter-annotator agreement, every coder needs to have coded every item:
Note that the data list needs to contain the same number of triples for each individual coder, containing category values for the same set of items.
However, there's also this tantalizing note in the documentation:
TODO: Describe handling of multiple coders and missing data
I'm attaching code illustrating the error that gets raised when you have some coder who did not code some items. Curious whether anyone has a solution or work-around for using this package under those circumstances? My specific goal is to evaluate agreement in a multi-label setting using MASI.
Thanks for any thoughts!