How to remove the Rules in KieBase from memory?

[Suhas S Kumar]

Hi All,

We are using drools in one of our batch jobs where this a different set of rules submitted for each job. We are facing a incremental growth in ram after each run. Is there a way to dispose the rules in KieBase after each run as they wont be required for the next job.

Our code to read the rule file(Before each batch job  and create container is as follows :

private static KieFileSystem getKieFileSystem(String rulesFileName,String tenant,String filePath) throws Exception {

KieFileSystem kieFileSystem = kieServices.newKieFileSystem();

kieFileSystem.write(ResourceFactory.newFileResource(filePath+tenant+"//"+rulesFileName+""));

return kieFileSystem;

}

 public static KieContainer getKieContainer(String rulesFileName, String tenant, String filePath) throws Exception {

KieBuilder kb = kieServices.newKieBuilder(getKieFileSystem(rulesFileName,tenant,filePath));

kb.buildAll();

KieModule kieModule = kb.getKieModule();

KieContainer kContainer = kieServices.newKieContainer(kieModule.getReleaseId());

return kContainer; 

}

Code we tried to remove the rules from memory  after each batch job (None of it worked) :

1. KieFileSystem.delete(filePath+tenant+"//"+rulesFileName+"");
2. kieServices.getRepository().removeModule(kContainer. getReleaseId());
3. kiesession.dispose() and kieContainer.dispose() (after the run)

Any help regarding this would be great.

Regards,

Suhas

[Toshiya Kobayashi]

Hi,

It might be the cache in KieRepositoryImpl. Please try to tune system property "kie.repository.project.cache.size" and "kie.repository.project.versions.cache.size".

for example)

        System.setProperty("kie.repository.project.cache.size", "1");  // default is 100

        System.setProperty("kie.repository.project.versions.cache.size", "0"); // default is 10

Regards,

Toshiya

2021年9月29日水曜日 17:33:35 UTC+9 suh...@gmail.com:

[Wigglesworth Moses]

Hi,
I have developed a REST API Service using Drools and Spring boot.
Apart from Firing Rules , This REST API Service loads Rules from a Drools Configured Excel Spreadsheet twice.
1.While starting the REST API Service.
2.Whenever the Rules in the Spreadsheet gets Updated.

For this API Service , I have set 4GB as the Max Heap Size.

Over the time , I found that the Heap Size has been consumed completely and the Swap Memory is being used.

While checking for a solution in the internet,
This Issue can be resolved by Configuring Cache Size of the Project and Project Versions of KIE Repository --> https://blog.kie.org/2022/09/drools-trouble-shooting-memory-issues.html
The same can be provided in Java using System.getProperty() --> https://groups.google.com/g/drools-usage/c/NhrQhuHeYOU/m/8QisTXohAwAJ

There was no Change in Heap Memory Consumption after providing these System Properties either using "-D" Option while starting the application or providing it via Java Class

Please find the below regarding the Heap Memory Consumed (Used Jconsole Tool) by this Service for loading Rules (Updated from the Spreadsheet)

1st Rule Update :
Initial Heap Size : 1.2 GB
Max Heap Size Consumed : 2.5 GB
Heap Size (When Loading Updated Rules gets Completed) : 1.3 GB

2nd Rule Update :
Initial Heap Size : 1.35 GB
Max Heap Size Consumed : 3.1 GB
Heap Size (When Loading Updated Rules gets Completed) : 3.2 GB

3rd Rule Update :
Initial Heap Size : 3.1 GB
Max Heap Size Consumed : 4 GB
Heap Size (When Loading Updated Rules gets Completed) : 2.5 GB

4th Rule Update :
Initial Heap Size : 2.6 GB
Max Heap Size Consumed : 3.9 GB
Heap Size (When Loading Updated Rules gets Completed) : 3.75 GB

5th Rule Update :
Initial Heap Size : 3.75 GB
Max Heap Size Consumed : 4 GB
Heap Size (When Loading Updated Rules gets Completed) : 5.0 GB

I took the Heap Dump and when Analyzed using MAT Tool by Eclipse , found there are 2 Objects which are alive and consumed large memory

1.org.drools.compiler.kie.builder.impl.MemoryKieModule - Occupied 357.8 MB out of 775.8MB
2.org.drools.compiler.kie.builder.impl.KieRepositoryImpl - Occupied 357 MB out of 775.8MB

Note : This Heap Memory Consumption and Heap Dump Details are the same - before and after providing the Cache Size as System Property

Please help me on this issue

Thanks in advance for spending your valuable time to analyze this issue and providing any info/solution.

Regards,
Wigglesworth

[Toshiya Kobayashi]

Hi Wigglesworth,

You have already started analyzing the heap dump with MAT.

Then, let's drill down into KieRepositoryImpl.

In the Histogram tab, right click the KieRepositoryImpl [List objects]->[with outgoing references]. It opens list_objects tab.

In the tab, you can expand the outgoing references by clicking a small triangle on the left side of the class name. Keep expanding... Then you will find what makes the KieRepositoryImpl big. e.g. Unexpected number of map entries (larger than the configured system property value).

Regards,
Toshiya

2023年12月6日水曜日 1:02:15 UTC+9 wiggles...@gmail.com:

[Wigglesworth Moses]

Hi Toshiya,
Thanks for your valuable assistance on this issue.
Will check and get back to you.

Regards,
Wigglesworth

[Wigglesworth Moses]

Hi Toshiya ,
As you suggested , I checked the Outgoing references of KieRepositoryImpl object. Found Old Kie Modules were present inside the KieRepositoryImpl and while looking on the internet to clear this issue , this link helped me to flushed the Older Kie Modules from KieRepositoryImpl Object.Took the Heap Dump and when analyzed in Eclipse MAT Tool , I wasn't able to see the KieRepositoryImpl Object in Histogram Tab.Thanks a lot Toshiya for your suggestions and guidance on this Issue.

But Still I could see org.drools.compiler.kie.builder.impl.MemoryKieModule is available [Holds 331.7 MB in Top Consumer Tab] unless and until the Rest API Service goes down. As mentioned in my previous mails I am using 2 Spreadsheets , whose total rule count is 30,000. Does MemoryKieModule holds this much memory because of the rule count?

Kindly help me on this.
Thanks once again Toshiya and Thanks in advance to anyone for spending your valuable time to analyze this issue and providing any info/solution.

Regards,
Wigglesworth

[Toshiya Kobayashi]

Hi Wigglesworth,

It depends on the complexity of rules, but 331.7 MB for 30,000 rules is not surprising to me. You would need some sufficient memory.

Regards,

Toshiya

2023年12月13日水曜日 1:41:54 UTC+9 wiggles...@gmail.com: