Issue 13 in dkpro-statistics: NaN result of expected agreement matches actual agreement

[dkpro-st...@googlecode.com]

Status: Accepted
Owner: richard.eckart
Labels: Type-Defect Priority-Medium

New issue 13 by richard.eckart: NaN result of expected agreement matches
actual agreement
https://code.google.com/p/dkpro-statistics/issues/detail?id=13

If the expected agreement (EA) matches the actual agreement (AA), the final
result is NaN. E.g. if EA=1 and AA=1, the result is NaN. Shouldn't the
result be 1 in this case - or in general if EA = AA, shouldn't the result
be the value of AA?

Observed in DKPro Statistics 2.0.0 using CohenKappaAgreement on a study
containing only one item on which the annotators agreed.

--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[dkpro-st...@googlecode.com]

Updates:
Labels: Component-Agreement Component-Docs

Comment #1 on issue 13 by chmeyer.de: NaN result of expected agreement

matches actual agreement
https://code.google.com/p/dkpro-statistics/issues/detail?id=13

This is a difficult case, see issue 1.

Returning 1.0 as chance-corrected agreement is not correct, IMHO. I leave
the issue open, such that documentation can be improved.

[dkpro-st...@googlecode.com]

Comment #2 on issue 13 by richard.eckart: NaN result of expected agreement

matches actual agreement
https://code.google.com/p/dkpro-statistics/issues/detail?id=13

Ok. I had this in a unit test and worked around it by simply an adding
additional non-different line to the study.

Still, maybe throwing some "InsufficientDataException" would be sensible.