How to apply prequential evaluation with sliding window using the MOA API

[damianos melidis]

Dear all,

Currently, I want to perform prequential evluation, for each incoming instance firstly test and then learn, but using the MOA api instead of the command line or GUI.

Until now I do the following pseudocode:

windowSize = 10;

//inside the prequential evaluation

evaluator = new WindowClassificationPerformanceEvaluator();
evaluator.widthOption(windowSize)
firstTimeSlidingWindow = true;

while(trainingSet.hasMoreInstances){

   trainingInsance = getTrainingInstance
  
   //test the model using current instance
   trainingInstanceExample = //create a training instance example from the training instance
   votes = selectedClassifier.getVotesForInstance(trainingInstance);
   evaluator.addResult(trainingInstanceExample)
  
->   if ((firstTimeSlidingWindow && windowCount == 0) || (!firstTimeSlidingWindow && windowCount == 1)){
         firstTimeSlidingWindow = false
         accuracy = evaluator.getFractionCorrectlyClassified()
         windowCount = windowSize
->      //evaluator.reset()
   }

   //train the model on the current instance

   windowCount--
}

Now, I am not sure for the lines with "->" that is,

for the first -> using this condition I get the correct number of statistics, accuracy, for the sliding windows (tested with a sample of 5 instances and window size of 2)
but the alternative simpler if (windowCount == 0){..} does not give me the correct number of statistics for the previous small test of sliding window (5 instances, window size 2).
Which of the two conditions is the correct?

For the second -> when the window size is reached do I need to reset my evaluation? I guess not as the sliding window always resets the head of the window and add the instance next to its' tail. But am I correct?

Lastly, a fundamental question (to understand) if I have set the window size of evaluation to X and I ask for the accuracy (inside the if with "->") at an instance number Y < X, is the resulted accuracy of any validity?

Grateful for your time and suggestions!

Many thanks,
Damianos

[Albert Bifet]

I think that the lines with -> are not needed, the result of accuracy should be computed correctly independently of the number of instances.

Cheers, Albert

--
You received this message because you are subscribed to the Google Groups "MOA users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moa-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[damianos melidis]

Dear Albert,

Thank you for your fast reply!

Sure, I have also seen the respective code of WindowClassificationPerformanceEvaluator.add() and .estimation(). I understand that as you keep track of the lenWindow to get the sum always the accuracy is correct.
However, I still need to collect the accuracy value, at the end of each sliding window, in order to have the plot of accuracy vs. sliding window.

An extra question, I would like also to run the evaluator with fading factor just to compare the results with the book "Knowledge Discovery from Data Streams" by Prof. Jao Gama, I use it the same way as the previous window without the smoothing factor. Am I right?

Many thanks in advance,
Damianos

[Albert Bifet]

What do you mean by "the end of each sliding window"? You can get the
accuracy of the sliding window at each instance.

You can take a look how the fading factor is implemented in
FadingFactorClassificationPerformanceEvaluator.

Cheers, Albert

> --
> You received this message because you are subscribed to the Google Groups
> "MOA users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to moa-users+...@googlegroups.com.

[damianos melidis]

Dear Albert, I mean that for the instances for the very first window I was not sure if I have report (plot) the accuracy of the model. For all other next windows I understand an incoming instance is a new sliding window (by the definition).
But after discussion with my supervisor, I will keep your thinking reporting for each instance.

Sure, I will have a look to that implementation.

Many thanks, Damianos