Drools as CEP engine: good or bad practices

[Yoenn Burban]

Hello,

We are using Drools fusion within a Spring Boot application to monitor streams of metrics and trigger alerting / remediation procedures if a rule matches. We are exclusively using events entrypoints to insert facts in the working memory, and  our application is designed so that the user can load new rules packages at runtime while the session is running.

We have been using it for a few months on low frequency streams and it's working very well so far. I would like to clarify a few points with anyone who has more experience of it to be sure we make a proper usage of Drools before scaling it up:

1. Is it okay to use a single, long-lived stateful session, do we need to halt and restart it occasionally to clear up the drools memory?

2. Is is acceptable to occasionally insert new packages in the session kbase while processing events in parallel? Adding a package takes around a few hundred milliseconds on a small ruleset, but I don't know how to measure the actual internal downtime

2. The RHS of the rules does not directly modify the working memory by inserting or removing facts, but it may be quite time-consuming, because it can trigger HTTP calls. I'm not sure whether Drools can execute RHS in separate threads or if we should do it manually? In such case, is the "kcontext" variable thread-safe? I pass this variable to java components involved in RHS to get information about current match.

Additionally, we have activated multithreadEvaluation (that seems relevant since our rules are independent for now), but the engine logs the following message at startup:

"The rete network cannot be partitioned: disabling multithread evaluation".

We are not using salience, agendas or queries, is there any other explanation for this?

3. Metrics samples (i.e. events)  usually come in batches of tens to thousands. Is it recommended to insert batches all at once for performance considerations using an atomicAction, or inserting them one-by-one does not make much difference?

Thanks for the help!

[Toshiya Kobayashi]

Hi,

> 1. Is it okay to use a single, long-lived stateful session, do we need to halt and restart it occasionally to clear up the drools memory?

It's okay to have a single, long-lived stateful session. But you should expire (@expires) events and delete facts which are no longer needed. If you keep adding events/facts without expiring/deleting, it would lead to out of memory.

Anyway, I recommend to monitor the heap under your long-run test.

> 2. Is is acceptable to occasionally insert new packages in the session kbase while processing events in parallel? Adding a package takes around a few hundred milliseconds on a small ruleset, but I don't know how to measure the actual internal downtime

You can incrementally update a kbase using KieScanner.

> 2. The RHS of the rules does not directly modify the working memory by inserting or removing facts, but it may be quite time-consuming, because it can trigger HTTP calls. I'm not sure whether Drools can execute RHS in separate threads or if we should do it manually? In such case, is the "kcontext" variable thread-safe? I pass this variable to java components involved in RHS to get information about current match.

You need to do it manually. kcontext state is changed by an engine thread so it's not thread-safe. You should extract the information from kcontext and give it to the separate thread.

> Additionally, we have activated multithreadEvaluation (that seems relevant since our rules are independent for now), but the engine logs the following message at startup:
> "The rete network cannot be partitioned: disabling multithread evaluation".
> We are not using salience, agendas or queries, is there any other explanation for this?

The message is raised usually when rules are tightly coupled so cannot be partitioned. But you said "our rules are independent". If you can share a small reproducer, I can check it.

> 3. Metrics samples (i.e. events)  usually come in batches of tens to thousands. Is it recommended to insert batches all at once for performance considerations using an atomicAction, or inserting them one-by-one does not make much difference?

Sorry that I'm not sure if it makes much difference. It would be nice to measure the performance in your test environment.

I hope it helps.

Toshiya

2021年11月20日土曜日 1:23:19 UTC+9 gloo...@gmail.com:

[Yoenn Burban]

Hi,

Thank you so much for your detailed reply. I've heard about KieScanner in the meantime and I'll investigate this as soon as I can.

Comments point by point:

1. I understood from the documentation that events are freed from the Drools memory once no rule can be activated by it:

By default, an event expires when the event can no longer match and activate any of the current rules

So if I insert a MyEvent instance in an entrypoint on which one or more rules are listening:

• either the content of MyEvent does not match the LHS constraint and it can immediately be freed
• or the LHS relies on other constraint that cannot immediately be met, thus all eligible MyEvent instances inserted are retained until the rest of the LHS eventually becomes true?

Did I get this right? If I set an @expire annotation on each event class, is this exclusive with the default behaviour? (i.e events would be retained until @expires anyway)

2. OK

3.1. Done, it appears to work well (if kcontext is read before starting the thread). The only point is that it requires declaring a closure in the RHS (executed in a separate thread), which does not give access to the globals. They must be copied to a final variable in the RHS before use. That's OK for me though.

3.2. "The rete network cannot be partitioned: disabling multithread evaluation"

Actually, the message immediately shows up after I insert a single rule in the kbase to declare the entrypoints:

rule "_"

enabled false

when

eval ( false )

Object() from entry-point "DeviceAlarms"

...

then

end

It likely has to do with the environment on which is running the engine: I noticed that I don't see this message on my machine, but it shows up on docker environment used for CI testing and production (we're running it on

openjdk:11.0.6-jre-slim-buster)

4. Sure, I'll test it someday. Adding events one-by-one seems OK for the moment.

Thanks!

[Yoenn Burban]

EDIT: I found in documentation section 4.5.10 and 21.52.1.5 the answer to point 1.

So I should use @Expires( value = "1d", policy = TIME_SOFT ) to get what I'm looking for.

[Yoenn Burban]

There's another thing I can't figure out as I migrate from KnowledgeBaseFactory to KieContainers so that I can perform incremental updates on the KieSession.

I understand that KieBases created with kcontainer.newKieBase(conf) ar not linked to the container, so that container updates are not applied to these sessions.

I don't see how to programmatically define specific KieBaseConfiguration options on the predefined KieBase(s) of the kcontainer. I understand that we can use a KieModuleModel.newKieBaseModel for that, but how can I set a custom

ConsequenceExceptionHandler, the same way I would do

config.setOption(ConsequenceExceptionHandlerOption.get(ExceptionHandler.class));

?

there's no such KieBaseModel method for that...

[Toshiya Kobayashi]

> 3.2. "The rete network cannot be partitioned: disabling multithread evaluation"
>
> Actually, the message immediately shows up after I insert a single rule in the kbase to declare the entrypoints:

Yeah, one rule cannot be partitioned. Adding single rule and building a kbase is a normal use case in your application? Maybe you are going to change the use case by using KieContainer.

> there's no such KieBaseModel method for that...

Ohh, you are right. I filed a JIRA for that : https://issues.redhat.com/browse/DROOLS-6763

In the meantime, you can use system property "drools.consequenceExceptionHandler" although it cannot be changed at runtime...

Toshiya

2022年1月5日水曜日 2:07:13 UTC+9 gloo...@gmail.com:

[Yoenn Burban]

> Yeah, one rule cannot be partitioned. Adding single rule and building a kbase is a normal use case in your application? Maybe you are going to change the use case by using KieContainer.

Aah, OK. Well, the business rules are only added by the user later during runtime, after the session is configured and fired. We use this "empty" rule to declare entrypoints from the DRL, because we immediately start to feed the entrypoints regardless of whether a rule uses them already.  entrypoint.insert(someEvent) raises a NullPointerException if it's not the case.
I'm still not sure why I don't get this message on my machine but only in a docker container. I'll try to investigate this more in detail later.

> In the meantime, you can use system property "drools.consequenceExceptionHandler" although it cannot be changed at runtime...

All right, I'll go this way then.

Thank you so much for your time!

Yoenn