Metrics for evaluate data stream classification algorithms using MOA

[Jorge Luis Rivero]

Hi!!!

 I would like to comment some things related with the evaluation and comparative assesment of algorithms. I have been studied some papers that address that issues, among them: On evaluating stream learning algorithms that is an extension of Issues in evaluation of stream learning algorithms, and Pitfalls in Benchmarking Data Stream Classification and How to Avoid Them. I think that its are a very good base for these issues. Regarding it I have some doubts I want to ask.

Well I am using MOA for development the new algorithm and I want to compare it with IBLStream algorithms. MOA provide Evaluate Prequential task, with options for slidding window size and fading factors. Can I switch between use slidding window or fading factors? I try with the window size = 0 and it didn´t work. There are some evaluators for choice, by default is WindowClassificationPerformanceEvaluator and there are others like FadingFactorsPerformanceEvaluator, then for switch if I want to evaluate Prequential error with fading factors or over sliding window I must to switch among those evaluators?


 When I running it in the evaluation panel are some metrics such as: accuracy (if I chose prequential, it is your refered prequential accuracy??), Kappa, Ram-Hours, Time and Memory, and MOA plot it. In the Gama article is refered the prequential error that is the acummulate sum of the loss function per evaluations, then if the algorithm is good then the prequential error is close to the Bayes error. Gama address three prequential errors: the prequential error, the prequential error over slidding windows and using fading factors. Does MOA implement it? How can I get its values using MOA?. In section 4 Comparative assesment Gama proposes Q. How can I to compute S ?  In section 4.1 He explain McNemar Test as a 0-1 loss function. How can I to compute the n0,1 and n1,0 ?

sorry if these are many question, I am trying to understand all issues related with the evaluation tasks.

Thank you for your time!!!
 Best Regards,
Jorge

[Albert Bifet]

> Well I am using MOA for development the new algorithm and I want to compare
> it with IBLStream algorithms. MOA provide Evaluate Prequential task, with
> options for slidding window size and fading factors. Can I switch between
> use slidding window or fading factors? I try with the window size = 0 and it
> didn´t work. There are some evaluators for choice, by default is
> WindowClassificationPerformanceEvaluator and there are others like
> FadingFactorsPerformanceEvaluator, then for switch if I want to evaluate
> Prequential error with fading factors or over sliding window I must to
> switch among those evaluators?

Yes, there are three evaluators:

- BasicClassificationPerformanceEvaluator: uses statistics from all the stream
- WindowClassificationPerformanceEvaluator: uses statistics of a sliding window
- FadingFactorsPerformanceEvaluator: updates statistics using fading factors

You have to run the experiments with each evaluator independently.

> When I running it in the evaluation panel are some metrics such as:
> accuracy (if I chose prequential, it is your refered prequential
> accuracy??), Kappa, Ram-Hours, Time and Memory, and MOA plot it. In the Gama
> article is refered the prequential error that is the acummulate sum of the
> loss function per evaluations, then if the algorithm is good then the
> prequential error is close to the Bayes error. Gama address three
> prequential errors: the prequential error, the prequential error over
> slidding windows and using fading factors. Does MOA implement it? How can I
> get its values using MOA?.

Yes, they correspond to

- BasicClassificationPerformanceEvaluator: uses statistics from all the stream
- WindowClassificationPerformanceEvaluator: uses statistics of a sliding window
- FadingFactorsPerformanceEvaluator: updates statistics using fading factors

> In section 4 Comparative assesment Gama proposes
> Q. How can I to compute S ? In section 4.1 He explain McNemar Test as a 0-1
> loss function. How can I to compute the n0,1 and n1,0 ?

MOA does not compute Q and the McNemar Test. You can implement them
easily in Java.

Cheers, Albert