what to do with zigzaggy ROC curve?

Michael

unread,

Sep 9, 2006, 1:17:31 AM9/9/06

to

Hi all,

I need your help! When I draw ROC curve, I have the problem that the number
of samples is too small:

nFalseAlarm -- the number of False alarms in
prediction/testing
nActualDataNotDiseased -- the number of NonDiseased samples in actual/test
data
nTrueAlarm -- the number of True alarms in
prediction/testing
nActualDataDiseased -- the number of Diseased samples in actual/test
data

falseAlarmRate=nFalseAlarm / nActualDataNotDiseased;
trueAlarmRate=nTrueAlarm / nActualDataDiseased;

Due to the lack of samples, the curve plotted was very ziggyzaggy...

Even worse, in some of my tests, the denominators were actually 1 or 0...

I just don't know how to handle these situations...

Please help me!

Thanks a lot

Glen M. Sizemore

unread,

Sep 9, 2006, 9:34:44 AM9/9/06

to

Why don't you explain what you are doing?

"Michael" <michael.monkey...@gmail.com> wrote in message
news:edtioh$904$1...@news.Stanford.EDU...

Michael

unread,

Sep 9, 2006, 2:34:18 PM9/9/06

to

"Glen M. Sizemore" <gmsiz...@yahoo.com> wrote in message
news:4502c34a$0$14693$ed36...@nr2.newsreader.com...

> Why don't you explain what you are doing?
>
>

I think I've explained very clearly. You want me to show what disease, or
etc.? That's not relevant... I want to keep the question short and clear,
and cut right to the point -- the key question is due to the lack of number
of diseased samples in actual data, the denominators of the false alarm
rate,etc. are only 0 or 1... so the ROC curve is very zigzaggy...

philbr...@hotmail.com

unread,

Sep 9, 2006, 8:25:25 PM9/9/06

to

Glen,

How is the relevant to neural networks?
You should probably post to a different newsgroup.

Phil

Glen M. Sizemore

unread,

Sep 10, 2006, 4:46:42 AM9/10/06

to

<philbr...@hotmail.com> wrote in message
news:1157847925.1...@e3g2000cwe.googlegroups.com...

> Glen,
>
> How is the relevant to neural networks?
> You should probably post to a different newsgroup.

How is my asking "Why don't you explain what you are doing?" relevant to
neural networks? Is that what you ar asking?

Greg Heath

unread,

Sep 10, 2006, 8:22:57 AM9/10/06

to

The ROC is obtained by combining the probabilities of CDF1
vs T(threshold) and CDF2 vs T. You cannot construct a CDF
with a sample size of 0 or 1. Therefore the solution is either
get more data or use another means to summarize your results.

Hope this helps.

Greg

Michael

unread,

Sep 17, 2006, 5:19:44 PM9/17/06

to

"Greg Heath" <he...@alumni.brown.edu> wrote in message
news:1157890977.8...@h48g2000cwc.googlegroups.com...

Definitely. What other power means do you recommend to summarize my
data/results?

The goodness of ROC is that it is powerful and concise, one plot tells all.

Do I have other alternatives to fullfill the same goal?

Thanks

Greg Heath

unread,

Sep 18, 2006, 3:54:52 AM9/18/06

to

> Definitely. What other power means do you recommend to summarize my
> data/results?
>
> The goodness of ROC is that it is powerful and concise, one plot tells all.
>
> Do I have other alternatives to fullfill the same goal?

With a sample size of N2 = 0 or 1, the best summary is just CDF1
complemented with a statement about the class 2 sample.

Hope this helps.

Greg