> > The Clopper-Pearson confidence limits for a binomial proportion > > are the smallest and largest hypothetical values of p > > that would not be rejected by the observed data. > > For 9/10 the 95% limits are (.554984, .997471); > > for 880/1000 they are (.858233, .899499).
> Thanks. This sounds like what I need. As I have data which I can use > to evaluate the performance of different rule selection strategies, > can probably apply optimisation techniques to come up the the scalar > utility function.
You can take advantage of the econometric concept of "value at risk": pick the model that guarantees better performance, neglecting the worst X% of situations.
If X=5%, then you would pick the 880/1000 proportion, as the guaranteed performance is 0.86, while the guaranteed performance for 9/10 is 0.55 (much worse).
This same "value at risk" logic underlies significance testing in general: prefer the null model except if the alternative will guarantee you lower error in 95 or 99 or 99.5 percent of samples of the same size.
-- mag. Aleks Jakulin http://www.ailab.si/aleks/ Artificial Intelligence Laboratory, Faculty of Computer and Information Science, University of Ljubljana, Slovenia.