Re: [learnlib-qa] Counting Membership Queries posed by the Learner

[Markus Frohme]

Hi John,


the counter oracles work as a wrapper around the original oracle, so every query that is passed through them is counted. If you pass the same counter oracle to your learner and equivalence oracle then the counter will count both types of queries. If you use separate counter oracle instances you can count them separately.

You can also chain the counter oracles with caches so that you can differentiate between total queries and unique queries depending on whether you count before or after the cache. I wrote a small example that should give you the basic idea of how you can plug them together [0].


Regarding your second question: The oracle creates batches of queries depending on your specified batchSize (default size is 1). Then it poses each batch sequentially until a counterexample is found (while not exceeding your upper limit). Let's say you have a batch size of 20 and set an upper limit of 1000 queries, then the oracle would

* pose 20 queries -> no counterexamples found?
* pose another 20 queries -> no counterexamples found?
* pose another 20 queries -> counterexample found? Return CE and terminate


Kind regards,
Markus


[0] - https://gist.github.com/mtf90/77cfd55083b5a402b9efc7c94cdd1626

On Sun, Apr 26, 2020 at 05:04:58AM -0700, John Huynh wrote:
> Hello,
>
> I am trying to count the total number of membership queries that are posed
> by the learner.
>
> This is what I did:
>
> MembershipOracle<Integer,Boolean> oracle = new MyDFAMembershipOracle1();
>
> DFACounterOracle<Integer> mqOracle =
> new DFACounterOracle<Integer>(oracle, "membership queries");
>
> ...
>
> System.out.println(mqOracle.getStatisticalData());
>
> Furthermore, I used "DFARandomWordsEQOracle" as equivalence oracle that produces randomly generated words.
>
> I observe that the number of membership queries goes up, as the number of randomly generated words go up as well.
>
> So this
>
>
> System.out.println(mqOracle.getStatisticalData());
>
> is actually printing out all the membership queries that were posed by the learner and by the equivalence oracle?
>
> Do I have to implement a cache to be able to count the number of membership queries that the learner posed, as shown in the example [1]?
>
>
> Another question is: If I set the number of randomly generated words to
> 1000 for example,
> does the equivalence oracle produce overall 1000 words or does it produce
> 1000 words for every invocation of the method "findcounterexample" ?
>
>
> Kind regards,
>
> John
>
> [1]:
> https://github.com/LearnLib/cav2015-example/blob/master/src/main/java/de/learnlib/example/cav2015/coffee/Example1.java
>
> --
> You received this message because you are subscribed to the Google Groups "LearnLib Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to learnlib-qa...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/learnlib-qa/5a61d034-3251-42e7-92e0-ad0f8a52224e%40googlegroups.com.

[John Huynh]

Hi Markus,

first of all, thank you.

regarding  the answer to my second question, I am still confused. Let's take the following example: 

DFARandomWordsEQOracle<>(eqCounter, 0, 20, 1000, random);

Does that mean that the EQ oracle has set 1000 queries as upper limit that have a length between 0 and 20 ? 

And the batch size is in this case random?

Kind regards,

John 

> To unsubscribe from this group and stop receiving emails from it, send an email to learn...@googlegroups.com.

[Carlos Diego Nascimento Damasceno]

Dear John,

You're 80% right. 

The random variable among the parameters is the seed the EQ oracle takes. 

That's useful when you want to guarantee the reproducibility of your experiments. 

In my experiments, I often keep the seeds of my experiments logged so I can repeat them if I need them.  

The batch size, according to the javadocs, would go after the random parameter as an integer value.  

--

Carlos Diego Nascimento Damasceno 

PhD candidate @ University of Sao Paulo

https://damascenodiego.github.io/

To unsubscribe from this group and stop receiving emails from it, send an email to learnlib-qa...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/learnlib-qa/de535934-0483-4470-a650-3c930e36288f%40googlegroups.com.

[John Huynh]

Thank you very much!

To view this discussion on the web visit https://groups.google.com/d/msgid/learnlib-qa/de535934-0483-4470-a650-3c930e36288f%40googlegroups.com.