Drools memory footprint and are we using Drools the right way?

[Christian L]

Hello,

A couple of months ago I joined a company which uses Drools for event processing. We recently had more time to benchmark the application and found that we spent a significant amount with garbage collection. It seems even at a moderate load we write a couple of GB/s to memory which then gets freed again by the GC. The main contributor was identified to be Drools and we would have a couple questions regarding this, but also beyond that. I will start with some background and prepared an example project for illustration under: https://github.com/liebharc/JavaRules

As a background: Our application frequently receives events. Events belong to a certain entity - in the example with students and classes that could be a school. The event is then routed to the corresponding processing unit for that entity. The processing unit starts a Drools session and processes the event. We typically have 500 to 1500 of such processing units instantiated and events for each of them come every couple of seconds. In the example there is only one processing unit called DroolsEngine [1]. As you can see in DroolsEngine a new session is started and there is also an interesting fact inserted which is the DataStore [1]. The data store is a large cache which holds state information which must be maintained between events. In the real application (not the illustration project) DataStore periodically gets persisted so that in case of a system restart the last state of DataStore will be restored and only the events which came after the last persist will be processed again to bring the application up to date.

The first question is that we found the option to reset a stateful session. Drools itself seems to use this for the session cache. Using the reset method instead of creating a new session with every event appears to significantly improve the memory footprint of Drools in our application. Is it okay to use the reset method to bring session back into life [1]? And will the session behave different after the reset compared to the case where it was created with new session? The last question is mainly directing towards this: Are all facts removed from memory and will the order of execution of rules will be the same? Our knowledge base often heavily depends on rules being executed in a certain order. Tokens and salience are used to get a certain order in Drools, but in the past we had bugs were a developer forgot to properly ensure order between two rules which depend on each other. If the order could potentially change then we would need to be more diligent with our testing. I recognize that the last part might be weird as we should never really on the order unless we take the necessary means to ensure it, but if you have a large code base then all kind of errors happen .

The second and more complex question is if Drools is the right tool for us to use or what we can do do benefit more from Drools? We also use jBPM and have a lot of logic components. Each logic component only has a few rules (20 at most). The order of execution within a rule file is almost always fixed with salience and tokens. With the relatively small number of rules per logic component and the fixed order I believe that we don't profit a lot from the Rete algorithm and a sequential rule engine might be better for us. However I haven't found a performance graph or guidelines at which point the Rete algorithm starts to outperform sequential rule engines or hand written if statements. So it's hard to base that decision on facts. If you have some insight in this then I would be happy to hear your opinion! A point where I feel we don't really use Drools very well is our DataStore. We also didn't had time yet to make it ready for property reactivity so that feature is deactivated. From my understanding having a large object as this DataStore which is used in most of the rules as fact would cause Drools to reevaluate a lot of LHS conditions with every modification of the DataStore. Again I'm not sure how big the impact is. Also I'm not sure if there would be a much better way to do it? Like extracting everything from the DataStore into the Drools session as a fact and then keep the session alive between events? That idea would require us to rewrite a lot of our Drools code. So I would first be interested to hear thoughts about this or other ideas on how we can improve our usage of Drools.

I recognize that this message became rather long and that the questions raised still aren't as concrete as I wished them to be. But I hope to get some answers and to gain some insight so that we are able to set the right course for our application.

Thanks and best Regards,
Christian

[1] Drools engine: https://github.com/liebharc/JavaRules/blob/master/src/main/java/com/github/liebharc/JavaRules/DroolsEngine.java
[2] Data store: https://github.com/liebharc/JavaRules/blob/master/src/main/java/com/github/liebharc/JavaRules/sharedknowledge/DataStore.java

[Christian L]

One update: It seems resetting the session isn't enough to reuse a session. If we do that then some rules don't get fired anymore. I wasn't yet able to reproduce that in a smaller project but we saw that the node memory after a session reset still contained PathMemory nodes. If we clear the memory after the reset ( session.getNodeMemories().clear() ) then reusing an existing session seems to work fine.

[Mark Proctor]

On Mon, Aug 13, 2018 at 3:29 PM, Christian L <christi...@gmail.com> wrote:

One update: It seems resetting the session isn't enough to reuse a session.

It should be, but I'm not sure how well tested it is. I think, recalling from memory, it's also  an internal method... which might be why it's not tested so well. We'll look into that soon, as either way it is desirable to work.

If we do that then some rules don't get fired anymore. I wasn't yet able to reproduce that in a smaller project but we saw that the node memory after a session reset still contained PathMemory nodes. If we clear the memory after the reset ( session.getNodeMemories().clear() ) then reusing an existing session seems to work fine.

I suspect when the path memory was added, the reset was not updated. Could you make a JIRA, and well make sure that's addressed.

Mark 

--
You received this message because you are subscribed to the Google Groups "Drools Usage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to drools-usage+unsubscribe@googlegroups.com.
To post to this group, send email to drools...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/drools-usage/9cf9156f-2f35-4494-9c67-2a8611e4c1d8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Christian L]

Thanks for the feedback! I've opened DROOLS-2892 to address this point.

If you or someone else would have some feedback or ideas about how we can adjust our usage of Drools to possibly get a faster processing speed and/or less memory consumption then any pointer would be appreciated. I've also updated the example project (see link in my earlier message) to illustrate how we use Tokens to control which rules are fired.

[Mark Proctor]

It looks like your usage is actually stateless. Correct?

We do have the StatelessKieSession which might help, although it doesn't use reset either.

I have some ideas on how to improve StatelessKieSession if we move to a completely immutable design - so that listeners, channels etc are only set once on creation. We can probably look at some sort of thread local caching and reset(), for situations where GC is being hit.

I'll see if we can find time to look into this over the coming months. So maybe around the Oct/Nov time frame.

Mark

On Tue, Aug 14, 2018 at 2:36 PM, Christian L <christi...@gmail.com> wrote:

Thanks for the feedback! I've opened DROOLS-2892 to address this point.

If you or someone else would have some feedback or ideas about how we can adjust our usage of Drools to possibly get a faster processing speed and/or less memory consumption then any pointer would be appreciated. I've also updated the example project (see link in my earlier message) to illustrate how we use Tokens to control which rules are fired.

--

You received this message because you are subscribed to the Google Groups "Drools Usage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to drools-usage+unsubscribe@googlegroups.com.
To post to this group, send email to drools...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/drools-usage/8cd207c7-4b6d-4c92-b487-384e0491ec64%40googlegroups.com.

[Mario Fusco]

Hi,

I fixed the reset and now it should be ok, but I cannot be 100% sure since I didn't have a reproducer for the problem you experienced. Drools 7.11.0.Final will be out next week with this fix, could you please check if this solves the problem also for you? If so we will add that reset() also to our public API.

Thanks,

Mario

On Tue, Aug 14, 2018 at 3:36 PM Christian L <christi...@gmail.com> wrote:

Thanks for the feedback! I've opened DROOLS-2892 to address this point.

If you or someone else would have some feedback or ideas about how we can adjust our usage of Drools to possibly get a faster processing speed and/or less memory consumption then any pointer would be appreciated. I've also updated the example project (see link in my earlier message) to illustrate how we use Tokens to control which rules are fired.

--

You received this message because you are subscribed to the Google Groups "Drools Usage" group.

To unsubscribe from this group and stop receiving emails from it, send an email to drools-usage...@googlegroups.com.

To post to this group, send email to drools...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/drools-usage/8cd207c7-4b6d-4c92-b487-384e0491ec64%40googlegroups.com.

[Christian L]

Hello,

Thanks for the fix! Yes we will test the new version. If there are snapshot builds available then we could also start earlier with our tests.

Regarding your other message: We tried to use the StatelessKieSession but didn't find a significant improvement (again in terms of memory consumption) by switching to it.

Thanks

Christian

[Christian L]

Thanks for https://issues.jboss.org/browse/DROOLS-2968 . Looks promising!

[Christian L]

Hello,

I would like to summarize our findings for anyone who followed or found this thread. In our case we have three issues:

1. Our application frequently receives new information which must be interpreted and for that it also creates frequently new Drools sessions
2. In the Drools code of the application control facts are used as primary method to avoid infinite loops
3. Most rules used the MVEL dialect for consequences

First benchmarks indicate that if we address those issues Drools performs (in terms of memory footprint and execution time) as well as our hand written Java comparison code in a test scenario which is worst case for Drools (fast LHS and consequences and a small number of rules,
14 rules to be precise). So the overhead Drools adds seems to be small, while you of course get a performance boost as your number of rules increase.

With that in mind these are the actions we are considering to improve the performance in our application:

The first issue has been addressed by the Drools team with the addition of session pools in Drools 7.13. This clearly improves the memory footprint of our application as we could measure with micro-benchmarks and macro-benchmarks. Throughput in the macro-benchmark roughly doubled.

The second issue requires work on our side. Here is a blog article on different ways how to prevent infinite loops in Drools: [1] As of of now the majority of our rules use control facts (our control facts are called tokens). The issue here is that the control facts are inserted into the Drools session which causes a few things which can hit
performance: The control fact needs to be created and garbage collected. In addition to the control fact which is created by the rule, Drools itself will create some internal objects to manage the new fact.

Drools offers alternatives to control facts as the blog article describes as well. So wherever possible we will replace the control fact with one of the following strategies:

lock-on-active: The LHS of a rule is only evaluated once when the agenda group becomes active. So if a rule in the same agenda group inserts or modifies a fact then the LHS of any rule with lock-on-active set will never be reevaluated. In our case I think there are a lot of cases where this alone is enough to prevent infinite loops without having to rely on control facts. The reason for that is that many rules react to an event fact (a custom class) which is never modified or inserted by any rule.

property reactivity: We have property reactivity disabled by now as two of our classes don't follow simple bean convention (there methods are more complicated than simple getters and setters). We will therefore apply "@Modifies" annotations as described in the section "Fine grained property change listeners" of the Drools documentation [2] or rely on traits are also described in the Drools documentation [3] and discussed for maps and map like structures in more details here [4, 5, 6].

In addition almost all of our rules used the MVEL dialect. Changing to the default Java dialect also improved the performance.

Micro-benchmarks indicate another considerable performance improvement if this is done. In our real application the improvement was roughly a factor of 4.

[1]
https://ilesteban.wordpress.com/2012/11/16/about-drools-and-infinite-execution-loops/
[2]
https://docs.jboss.org/drools/release/7.13.0.Final/drools-docs/html_single/#_left_hand_side_when_syntax
[3]
https://docs.jboss.org/drools/release/7.13.0.Final/drools-docs/html_single/#_traits

[4] groups.google.com/forum/#!topic/drools-usage/G7rlz3oI1HQ
[5]
https://github.com/kiegroup/drools/blob/master/drools-compiler/src/test/java/org/drools/compiler/factmodel/traits/TraitMapCoreTest.java
[6]
http://blog.athico.com/2011/12/dynamic-typing-in-rules-traits-part-2.html