Understanding events retraction

[Yoenn Burban]

Hi,

I am trying to understand the conditions over which an event is retracted from Drools memory when running in stream mode.

considering the following class

@Expiration(value="1h", policy=TIME_SOFT)

@Role(Role.Type.EVENT)

public class MyEvent {

      public String message;

}

With the following DRL:

rule "_"

when

    $ev: MyEvent()

    MyEvent(this after[0s, 10S] $ev)

then

end

If I insert a MyEvent(), then move the session clock 1 minute forward and insert a second event, I can see from the EventFactHandle that the first event is retracted after firing the rule, which is expected.

Now if I use the following DRL:

rule "_"

when

    $ev: MyEvent( message="foo" )

then

end

Then the inserted event for which message != "foo" is not retracted until I move the clock 1 hour forward (i.e. it reaches the explicit expiration time).

Why so? As the event cannot match the above rule (since it is supposed to be immutable), it could be deleted as well?

We may often encounter rules as the last one with our drools usage and I'm concerned about how we can remove events from the drools memory as soon as they are not usable, if that's possible.

Thanks for your help!

[Ivan Kostyuk]

The events will stay in memory until they expire or explicitly retracted. If there are a lot of garbage events and they can be identified, it will be good to have a rule which retracts the useless for the logic facts.

Regarding any rule not matching on event occurrence, once an expiration is defined the engine expects the implementer to be in charge.

This is from the documentation for @expires

By default, an event expires when the event can no longer match and activate any of the current rules.

https://docs.drools.org/7.63.0.Final/drools-docs/html_single/#drl-declarations-metadata-tags-ref_decision-engine

in DRL, the command delete can be used (retract can is supported too and does absolutely the same).

https://docs.drools.org/7.63.0.Final/drools-docs/html_single/#drl-rules-THEN-con_drl-rules

Be careful with the "window" and the pace of incoming events - if more than 1 million of events is in memory performance degrades a lot. When it is about 6M of them, an insertion of a new event took for me 30+ minutes. Eventually Out Of Memory came.

[Ivan Kostyuk]

I replied and got another thought. Week ago I asked what is the difference between @duration and @expires.

https://docs.drools.org/7.63.0.Final/drools-docs/html_single/#drl-declarations-metadata-tags-ref_decision-engine

I think the difference is in what you are looking for. Could you try with @duration instead?

On Tuesday, January 18, 2022 at 7:44:57 AM UTC-6 gloo...@gmail.com wrote:

[Yoenn Burban]

Actually, reading my initial question I think I could've simplified first since putting the @expires tag does not change the result of the test. If there's no time constraint on the event in the rule, it is not retracted.

So basically i can insert a MyOtherEvent() object that is not used in any rule, and it won't be removed from memory.

@duration from my understanding seems to be intended for a different purpose: it is involved in some temporal operators, such as finishes, coincides, etc (as explained here, slides 24-25: https://www.slideshare.net/ge0ffrey/applying-cep-drools-fusion-drools-jbpm-bootcamps-2011)

So that does not sound like what I'm looking for.

I just realized that if I set a soft explicit timeout of 0, like this:

@Expires(value="0s", policy=TIME_SOFT)

 then the event is retracted as wanted in both cases, i.e. it is immediately retracted if no longer used by any rule, but preserved if a rule uses it (as in the second case).

[Yoenn Burban]

That's still not perfect though.

If I have such DRL:

rule "_"

when

    $ev: MyEvent(message=="foo")

    MyEvent(this after[0s, 10S] $ev, message=="foo"))

then

end

Then a MyEvent(message="bar") is not retracted before 10s are elapsed.

[Ivan Kostyuk]

That did not help to understand. I understand interval logic, but not getting what is the main difference on event deletion.

What I understood, that by adding interval into the rules you implemented the same logic - you set duration in the rule, by making the fact valid for the rule longer.

In this case, I guess, @duration can be set on event for the same purpose you use interval [0s,10s]

Expiration is very hard thing (according to the source code I read).

The TIME_SOFT policy is explained here

https://docs.drools.org/7.63.0.Final/drools-docs/html_single/#_soft_expiration_for_events

but I fail to understand the meaning from description.

If I understood right from the description and your tests this statement

@Expires(value="0s", policy=TIME_SOFT)

will make events to remain in the memory forever if at least one rule can be triggered by it even it was already triggered once.

But, according to the source code. the expiration offset is set to 1ms (unless there are other expirations defined on rules or context).

drools/drools-core/src/main/java/org/drools/core/reteoo/builder/PatternBuilder.java method attachObjectTypeNode

I would use this with caution

@Expires(value="0s", policy=TIME_SOFT)

If events are not needed if not covered by any rule, I will delete them. If events are needed only once, I will use them as default facts. If there is interval logic, I will define duration.

Thanks for your question, I think I concluded answer for mine. :-)

[Yoenn Burban]

I don't think @duration is related to event lifecycle. If you check the slides I linked above, it allows to express interval-based events that have a start and end date, and operators that derive from this property, such as "overlaps" which does not make sense for point-in-time events (default). Exceeding the end date of duration does not mean the event is to be deleted from memory. So that's different from @expires which gives a delay after which the event must be removed from memory.

As you point it, it sounds like the TIME_SOFT policy might lead to events stuck in memory for a while if a rule has a temporal operator on them, and I'm worried about it. I'm thinking about periodically calling getFactHandles() on each entrypoint and enforce event expiration for old events using the EventFactHandle. Don't know if that's very idiomatic but I don't see any other way except using @expires with the default TIME_HARD policy, either with a long delay that would involve high memory usage, or a short delay which will break longer temporal operators.

[Ivan Kostyuk]

This is from documentation about @duration

Interval-based events have a duration time and persist in the working memory of the Drools engine until their duration time has lapsed.

If you will to try. I struggle to set up to run a simple rule - I use Workbench with KIE Server.

[Yoenn Burban]

I just tested with adding a duration. It seems to override the expiry because if I set @expires(value="0s") then the rule with the temporal condition still fires (if the duration is < 10s, otherwise eventB happens before the end of eventA).

Furthermore, if I remove any @expires and set a duration of 20s (btw, the duration is expressed in milliseconds), use the following rule:

rule "_"

when

    $ev: MyEvent()

    not(MyEvent(this overlaps $ev, this != $ev))

then

end

then do the following actions:

- insert 1 event

- wait 10s

- insert 1 event

- fire session

Then the rule is not triggered (because the second event overlaps the first). However the first event is not expired after that.

Now things get a bit weird:
- if there's no @expires and I wait 10 more seconds, the event is still not retracted

- if there's @expires with TIME_SOFT policy and I wait 10 more seconds, then the event is retracted.

So, at least that goes with what the documentation says: the duration takes precedence over inferred or explicit expiration. Once the duration is over, the expiration rules apply.

[Ivan Kostyuk]

The first event was not retracted because it has duration and you created another event which depends on it. Its time lapsed, but there is still rule which activated by it.

If you use temporal conditions while there are garbage events, you need to add a rule which explicitly retracts garbage events before expiration.

If we keep altering initial scenario we will never find a solution.

What can be done to make it more efficient to have two different events/bases/entry points.

At first events are being filtered and expire immediately. Good events are written to another entry point, where they have normal expiration and you do not worry that they are garbage.

DirtyEvent is the input, MyEvent is output with expiration.

I'm not an expert in Drools, still trying to find a practical application for it.

[Yoenn Burban]

The first event was not retracted because it has duration and you created another event which depends on it. Its time lapsed, but there is still rule which activated by it.

> More exactly, I'd say that the event is still valid because it could still match the rule if another event comes later. So my example was quite not relevant

If you use temporal conditions while there are garbage events, you need to add a rule which explicitly retracts garbage events before expiration.

> "garbage event" for you means an event that cannot match any rule ? How do you identify an event as "garbage" at runtime ?

If we keep altering initial scenario we will never find a solution.

> I was trying to answer you questions about duration with another example. As I told you, adding a duration on the initial scenario basically works like an offset on the @expires tag and prevents scenario expiration until duration is elapsed. That is not really what I'm looking for, because I expect events to be retracted immediately if no rule retains them (like the second example of my first post).

What can be done to make it more efficient to have two different events/bases/entry points.

At first events are being filtered and expire immediately. Good events are written to another entry point, where they have normal expiration and you do not worry that they are garbage.

DirtyEvent is the input, MyEvent is output with expiration.

I'm not an expert in Drools, still trying to find a practical application for it.

> I'm not sure of what you intend by "good events are written to another entry point". If by that you mean "events that may match a rule should be inserted into another entrypoint" then I'm not sure how you would actually filter out these events from the others.

[Yoenn Burban]

More exactly, I'd say that the event is still valid because it could still match the rule if another event comes later. So my example was quite not relevant

Erratum: that is incorrect. I thought the not(Event(this overlap...)) might be understood as "another event comes that does not overlap the first one" but the actual meaning from my experiment is "there's no other event that overlaps the first one", which makes more sense. So the proper explanation would be, I guess: the first event is retained since it prevents the second one from firing the rule.

My former observations on adding @Expires with TIME_SOFT might be incorrect as well: the first event is indeed retracted, but the rule is actually fired by the second event. In fact, there's no difference between a TIME_SOFT and TIME_HARD policy here: once the first event reaches end of duration, it is retracted, allowing the second to match the rule.

[Ivan Kostyuk]

You build the logic the rules. So, you can know ahead of time what events are "garbage" not needed to be kept and retract them explicitly. Temporal rules create complexity.

From your example,

rule "_"

when

    $ev: MyEvent( message!="foo" )

then

end

You decided (by your expectation) that "foo" is good while !"foo" is "garbage" and it should be expired immediately. So, retract it explicitly.