MTurk quality analysis implementation & web demo

[Brendan O'Connor]

Hi,

Panos Ipeirotis recently released an implementation of worker quality analysis; it appears to use the Dawid & Skene (1979) EM (weighted voting) approach, which jointly estimates worker quality/accuracies along with aggregate label confidences.  There's even a web demo.

http://qmturk.appspot.com/

http://code.google.com/p/get-another-label/

Brendan

--
http://anyall.org