ROC or DET curves

[Stephanie Rickett]

Has anyone tried to create ROC or DET curves to compare the performance of the classifiers on a single data set? To do this I would need to play with the thresholds. Is this possible in all supervised methods?

[Luis Pedro Coelho]

Hi,

I think this only really works with methods which return a numeric estimate of
the output rather than simply the binary value.

What exactly were you trying to do?

HTH,
--
Luis Pedro Coelho | Institute for Molecular Medicine | http://luispedro.org

[Stephanie Rickett]

I am trying to get a better sense of performance for my models. With a DET graph I would be able to see how many FPs I have if I only 25% miss rate. I got this from a paper.
"A detection error tradeoff (DET) graph is a graphical plot of error rates for binary classification systems, plotting false reject rate vs. false accept rate."

 http://www.itl.nist.gov/iad/mig/publications/storage_paper/det.pdf

[Stephanie Rickett]

Or an ROC is good too http://en.wikipedia.org/wiki/Receiver_operating_characteristic

[Luis Pedro Coelho]

Hi,

Sorry for the long silence (Holidays & all).

Anyway, ROC is something I've been meaning to put into milk for a while.
I just did, but only for the github version for the moment:

https://github.com/luispedro/milk/commit/559f0ab9de3dc61f0061ab5e8da3d2d35fc6a790

HTH
Luis

[Stephanie Rickett]

Awesome! I will check it out!

[Stephanie Rickett]

Thanks :o)