Executable Model Questions

[KimJohn Quinn]

We have a KieBase, with about 4k rules and a few custom types, that is built off of a spreadsheet into a single DRL file now.  The rules are relatively simple, basic constraints and consequences.

As we experiment with moving towards the ExecutableModel (potentially in prep to use Kogito) I wanted to make sure the steps we do now would be the same?

    final KieFileSystem kFilesystem = kServices.newKieFileSystem();

    kFilesystem.write(newFileResource(newRuleResource("executable-all.drl")));

    final Stopwatch sw = Stopwatch.createStarted();

    final KieBuilder kBuilder = kServices

        .newKieBuilder(kFilesystem)

        .buildAll(ExecutableModelProject.class);

    log.debug("Compiled in {}", sw.stop());

    final InternalKieModule kModule = (InternalKieModule)kBuilder.getKieModule();

    writeByteArrayToFile(newOutputFile("executable-all.jar"), kModule.getBytes());

    final Results results = kBuilder.getResults();

    assertThat(results, notNullValue());

    //Get session and execute (either throught container or kbase)

    KieContainer kContainer = kServices.newKieContainer(kModule.getReleaseId());

    KieBase kBase = kContainer.getKieBase();

    StatelessKieSession session = kBase.newStatelessKieSession();

            

It seems the time to "compile" is much longer.  About 60s vs. <20s and the JAR size is about 6MB vs. 1.5MB (I can understand this).

Questions:

1) Is this expected that the compile time would take considerably longer?  We are going from the DRL to the new lambda functions.

2) Is usage of the model similar?  Create a session from a container or kiebase and fire it?

3) Once I "compile" the KJAR can I manually load it into the filesystem and use it from there still (I ask this because I am confused how Kogito does it)?

4) In trying to line up for using the Executable Model, and potentially Kogito, is it better to work directly off the KieBase instead of the KieContainer going forward?

5) We use custom classloaders in the build (above is a test snippet) and container creation, this "released" resources a little nicer according to the profiler and MAT.  The same approach does not apply with the executable model?  It does not look to be release any resources.  Where usually we sit around 20k classes loaded I now show about 115k (could be me but just wondering since I can reliably run the same scenario without the new model and see the lower resource usage).

[Pavel Tavoda]

For using compiled KieBase you need following in your RULES project:

<plugin>
 <groupId>org.kie</groupId>
 <artifactId>kie-maven-plugin</artifactId>
 <version>${drools-version}</version>
 <extensions>true</extensions>
 <configuration>
<generateModel>YES</generateModel>
 </configuration>
</plugin>

Parameter generateModel is important. Than in your runtime project you need only dependency for:

                <dependency>

                        <groupId>org.drools</groupId>
                        <artifactId>drools-model-compiler</artifactId>
                        <version>${drools-version}</version>
                </dependency>

And than load it (if you have static dependency on RULE project):

KieServices kieServices = KieServices.Factory.get();

ReleaseId releaseId = kieServices.newReleaseId( "org.acme", "myartifact", "1.0" );
KieContainer kieContainer = kieServices.newKieContainer( releaseId );

This is most performant way how to load rules to runtime.

/Pavel Tavoda

--
You received this message because you are subscribed to the Google Groups "Drools Usage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to drools-usage...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/drools-usage/8402a644-e87f-413a-8eb0-c9b24c71ffa7%40googlegroups.com.

[KimJohn Quinn]

That works if I am compiling the KJar upfront and including it with the application.  In my case, rules can be modified and compiled at runtime on a node and pushed to the repository (they may be for completely different domains). Then, for execution, which is on handled by a different node, they are pulled from the repository, loaded, and executed.

To unsubscribe from this group and stop receiving emails from it, send an email to drools...@googlegroups.com.

[KimJohn Quinn]

I should also point out the two steps above do not happen on the same machine.  The compile happens on one and the execute happens on another (so the executing node manages pulling down the artifact and loading it into the KieRepository before executing it).

[Pavel Tavoda]

Scanner doesn't work?

To unsubscribe from this group and stop receiving emails from it, send an email to drools-usage...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/drools-usage/6de3fbc6-bbef-41a0-9398-3aced810c5fe%40googlegroups.com.

[KimJohn Quinn]

Problem is not with the execution side.  More the "compilation" side when i switch over to using the Executable Model.  It still executes fine.

The questions I have are maybe a little broader and for my own clarity rather than a problem:

- The considerable time difference between not using (buildAll()) and using the ExecutableModel (buildAll(ExecutableModel.class)) - <20s vs. ~60s.

- Whether or not usage of a "executable model" is the same (i.e. get the container or kiebase and create the session from it vs. some other approach)?

- How does this align with Kogito or even Drools going forward?  Is the way to go going to be the ExecutableModel? 

- Resources, memory-wise, it does not seem to release when I use the ExecutableModel.  Same exact process without it does - is this something I need to dig deeper into?

KJQ

Scanner doesn't work?

To view this discussion on the web visit https://groups.google.com/d/msgid/drools-usage/6de3fbc6-bbef-41a0-9398-3aced810c5fe%40googlegroups.com.

[Pavel Tavoda]

From my knowledge of drools:

On Mon, Dec 16, 2019 at 9:56 PM KimJohn Quinn <k...@logicdrop.com> wrote:

Problem is not with the execution side.  More the "compilation" side when i switch over to using the Executable Model.  It still executes fine.

The questions I have are maybe a little broader and for my own clarity rather than a problem:

- The considerable time difference between not using (buildAll()) and using the ExecutableModel (buildAll(ExecutableModel.class)) - <20s vs. ~60s.

Compiled model contains compiled expressions only (no MVEL necessary) as lambda expressions. It's still need compiler to fully create KieBase in memory.

 

- Whether or not usage of a "executable model" is the same (i.e. get the container or kiebase and create the session from it vs. some other approach)?

YES. Only conditions are precompiled to lambda expressions.

 

- How does this align with Kogito or even Drools going forward?  Is the way to go going to be the ExecutableModel? 

I guess they are working on 'full' compilation. Rules than can be loaded without compiler.

 

- Resources, memory-wise, it does not seem to release when I use the ExecutableModel.  Same exact process without it does - is this something I need to dig deeper into?

I'm (re)loading rules via InternalKieModule and using StatelessSession. Rules are garbage collected and memory released. I have special testCase for this for our DroolsService.

/Pavel

[KimJohn Quinn]

Thank you for the responses.  Mine are inline in red...

On Monday, December 16, 2019 at 4:18:04 PM UTC-5, Pavel Tavoda wrote:

From my knowledge of drools:

On Mon, Dec 16, 2019 at 9:56 PM KimJohn Quinn <k...@logicdrop.com> wrote:

Problem is not with the execution side.  More the "compilation" side when i switch over to using the Executable Model.  It still executes fine.

The questions I have are maybe a little broader and for my own clarity rather than a problem:

- The considerable time difference between not using (buildAll()) and using the ExecutableModel (buildAll(ExecutableModel.class)) - <20s vs. ~60s.

Compiled model contains compiled expressions only (no MVEL necessary) as lambda expressions. It's still need compiler to fully create KieBase in memory.

Still does not explain why the inclusion of only the "ExecutableModel" in the build all would take so much longer. 

 

- Whether or not usage of a "executable model" is the same (i.e. get the container or kiebase and create the session from it vs. some other approach)?

YES. Only conditions are precompiled to lambda expressions.

This seem to work for me.  Only issue I had was with a property that had a collection inside the mody(){} block. 

 

- How does this align with Kogito or even Drools going forward?  Is the way to go going to be the ExecutableModel? 

I guess they are working on 'full' compilation. Rules than can be loaded without compiler.

So, will Drools and Kogito eventually converge or will they remain separate?

 

- Resources, memory-wise, it does not seem to release when I use the ExecutableModel.  Same exact process without it does - is this something I need to dig deeper into?

I'm (re)loading rules via InternalKieModule and using StatelessSession. Rules are garbage collected and memory released. I have special testCase for this for our DroolsService.

I am curious as to why it does not seem to release for me.  Same as point #1 above, nothing is different except the inclusion of the ExecutableModel in the buildAll. 

/Pavel

[Luca Molteni]

Good morning Kim

On Mon, Dec 16, 2019 at 8:53 PM KimJohn Quinn <k...@logicdrop.com> wrote:

We have a KieBase, with about 4k rules and a few custom types, that is built off of a spreadsheet into a single DRL file now.  The rules are relatively simple, basic constraints and consequences.

...

            

It seems the time to "compile" is much longer.  About 60s vs. <20s and the JAR size is about 6MB vs. 1.5MB (I can understand this).

Questions:

1) Is this expected that the compile time would take considerably longer?  We are going from the DRL to the new lambda functions.

Compile times are definitely longer, as with the executable model you have to generate some Java source code files and compile them using an embedded Java compiler to memory. Expect also higher memory consumption.

The whole idea of the executable model is to do this upfront and create a KJar that is faster to boostrap.

Expect this mode of execution to be much faster at initialization (creating a KieBase and a new KieSession), given a compiled kjar.

2) Is usage of the model similar?  Create a session from a container or kiebase and fire it?

The API is exactly the same.

 

3) Once I "compile" the KJAR can I manually load it into the filesystem and use it from there still (I ask this because I am confused how Kogito does it)?

Kogito uses the same API but use some internals to provide incremental code compilation.

Take a look at it source code if you need more details

https://github.com/kiegroup/kogito-runtimes/blob/master/kogito-codegen/src/main/java/org/kie/kogito/codegen/rules/IncrementalRuleCodegen.java

4) In trying to line up for using the Executable Model, and potentially Kogito, is it better to work directly off the KieBase instead of the KieContainer going forward?

 Kogito may provide a different API in addition to the old one, but it's still a work in progress. Anyway if your code supports the executable model expect the migration to be effortless

5) We use custom classloaders in the build (above is a test snippet) and container creation, this "released" resources a little nicer according to the profiler and MAT.  The same approach does not apply with the executable model?  It does not look to be release any resources.  Where usually we sit around 20k classes loaded I now show about 115k (could be me but just wondering since I can reliably run the same scenario without the new model and see the lower resource usage).

The problem with the not released resources are related to the fact that we used to provide the lambdas for the rules  inside the compiled rules file, forcing the runtime to keep the instance of the rule definition. After this commit (https://github.com/kiegroup/drools/commit/0ac6d1d80c66a2a943b1f1106bb8638215e1cb0a) the lambdas are externalised (and shared) among the rules, therefore the Rules class are garbage collected.

If you use the executable model in 7.31.Final you should see much better results in memory occupation.

I hope I answered all your questions, let us know if you have more.

 

[Luca Molteni]

On Mon, Dec 16, 2019 at 9:43 PM KimJohn Quinn <k...@logicdrop.com> wrote:

That works if I am compiling the KJar upfront and including it with the application.  In my case, rules can be modified and compiled at runtime on a node and pushed to the repository (they may be for completely different domains). Then, for execution, which is on handled by a different node, they are pulled from the repository, loaded, and executed.

Kim,

Given this scenario I can see why you're considering Kogito, you're on the right track.

In your case, the compilation on the first machine should take longer but bootstrapping each node should be much faster.

With the executable model you're doing only the work that it was used to be done in each done at the startup time. so it should definitely scale better especially when you have many nodes.

[Luca Molteni]

On Mon, Dec 16, 2019 at 10:28 PM KimJohn Quinn <k...@logicdrop.com> wrote:

- Whether or not usage of a "executable model" is the same (i.e. get the container or kiebase and create the session from it vs. some other approach)?

YES. Only conditions are precompiled to lambda expressions.

This seem to work for me.  Only issue I had was with a property that had a collection inside the mody(){} block.

I'm not sure I'm following you both here, but not only constraints are transformed to a lambda, also consequences. But yes I agree with the fact that MVEL is not necessary anymore. Mvel consequences (including the usage of the modify block) are compiled to Java code using the drools-mvel-compiler module.

The problem currently with consequences is that we cannot externalise the one using modify statements as they're capturing lambdas but we're working on it.

Please refer to https://issues.redhat.com/browse/DROOLS-4870 if you need such functionality.

[Pavel Tavoda]

Kim I guess Luca answered all your questions. I would like to add some more thoughts which are important for me as architect:

- I don't like to run Java compiler inside my service runtime -> use precompiled KJAR

- You never know errors before you compile, you have to compile DRL, XLS, ... and decide if you can switch to new KBase. Sometimes warnings are important sometimes not (type conversions Integer -> Long, ...), you can't decide by automatic algorithm

- I don't like to run Maven inside my VM (that's what scanner is doing) for downloading new versions from central repository

- Be careful about maven settings.xml setup because it checking artifacts only once a day (with default setup) even for SNAPSHOTs

- I don't even like to run Maven repository in my production environment

- Exclude sources (DRLs, XLS, ...) from result KJAR

- Much less dependencies in server runtime (only drools-model-compiler required) if you don't use scanner and compilation from sources it's about 25MB less for resulting server JAR

Because of this I decided to have precompiled KJARs available for production environment. We distribute KJARs to server our own way (separate thread which check and download new version from central administration server)

Than we build KBase from precompiled KJAR:

File kjarFile = ....
ReleaseId kieRelease = new ReleaseId("aaa", "bbbb", "vvvv"); // Use real GroupId and ArtifactId of your KJAR, version (vvvv) is ignored ;-)
InternalKieModule kieModule = InternalKieModule.createKieModule(kieRelease, kjarFile);
KieProject kProject = new KieModuleKieProject(kieModule, kieModule.getModuleClassLoader());
kProject.init();

return new KieContainerImpl("RULESXXX", kProject, KieRepositoryImpl.INSTANCE, kieRelease);

It's maybe little bit 'hacky' but it's working and GC returns RAM used by previous RuleBase. Everything run fast and smooth with no additional RAM required for Maven and Java Compiler. We are using stateless sessions.

Hope this helps

/Pavel

--
You received this message because you are subscribed to the Google Groups "Drools Usage" group.

To unsubscribe from this group and stop receiving emails from it, send an email to drools-usage...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/drools-usage/CAFRyLGibZqAUTG0o5vZC1nD21XU5NuCUbraKkqy032peX9H7aA%40mail.gmail.com.

[KimJohn Quinn]

Thank you everyone.  Best answers ever!

On Monday, December 16, 2019 at 2:53:04 PM UTC-5, KimJohn Quinn wrote: