DMN with 0.3 million rules

[rishab raina]

Hi,

We have a case where i have 0.4 million rules approx in a single dmn table. 

When running maven build the build itself fails with java heap space issue.

We tried running the build with scaling maven java heap space to almost 10 Gb, which in itself is not scalable. 

We wanted to know the most optimised way to run such dmn's, since we cant keep such high memory for containers, 

Also our services dont talk to db, so in case you think we can lazy load from db. 

What do you think ?

Regards

Rishab 

[Matteo Mortari]

Hi Rishab,

Are you by chance using the kie-maven-plugin as part of your build? If so, I would make sure the "validateDMN" is set to "disable" (example). Or, if you are using DMN Validation via another mechanism, to disable it.

If that is confirmed being the case, the actual reason being: DMN Validation makes sense for human-authored tables, and I find it hard to believe a single table has been manually authored by a human with 0.4 million lines :)

At the time, we tested extensively for complex cases with many rows/columns, but still in the realms of human-authored and to focus on best practices which might be overlooked or missed by the author, according to literature guidance and best practices.

Hope this helps!

MM

--
You received this message because you are subscribed to the Google Groups "Kogito development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kogito-developm...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/kogito-development/5d55b9f6-2ee2-47e5-a545-4865080fbe27n%40googlegroups.com.

--

Matteo Mortari

youtube.com/@MatteoMortari

linkedin.com/in/matteomortari

[Rhett S]

consider looking for ways to reduce the size of your table. food for thought:

- in database design, normalization adds reduces duplication and adds integration checks. I've found that large tables can be reduced to many smaller tables through normalization techniques. 

- similar to nested conditionals, consider if statement literals before your large decision table. 

I've been able to load a DMN with 100k decision rows a few years ago. attempt to load 100k, and 200k etc. until hit a limit. it is possible you can have 4 DMNs with 100k each, chained together.

When I see DMNs that big I start asking, does this data belong in a database.

[rishab raina]

Hi, 

I am using quarkus maven plugin, i dont seem to find a public property that could disable it. 

<dependency>
            <groupId>org.drools</groupId>
            <artifactId>drools-quarkus-decisions</artifactId>
        </dependency>

my plugin 

- <plugin>
                <groupId>${quarkus.platform.group-id}</groupId>
                <artifactId>quarkus-maven-plugin</artifactId>
                <version>${quarkus.platform.version}</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>build</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

[Matteo Mortari]

in Kogito/Quarkus, iirc, there is the analogous in application.properties

kogito.decisions.validation=DISABLED

Reference: https://docs.kogito.kie.org/latest/html_single/#con-dmn-validation_kogito-dmn-models:~:text=Configuration%20to%20disable,decisions.validation%3DDISABLED

but it has been a while so you might want to check the latest developments where this property needs to be set in a Quarkus app :)

Hope this helps, let us know!

MM

To view this discussion visit https://groups.google.com/d/msgid/kogito-development/634a9c91-152d-4131-ad28-2ef186f12582n%40googlegroups.com.

[rishab raina]

thanks this worked, memory consumption has reduced, from 6gb - 3gb for maven build process, but i wanted it to be under 1.5gb, do you think any other optimisation other than this can reduce maven build memory footprint ?

[Matteo Mortari]

Hi Rishab,

I'm glad to hear that helped, and that unblocked you move further.

About your last question, difficult to say without profiling and understanding if there is anything to be done on your specific use case.

Additionally, there is still the point that millions rows in a single table is not human friendly, so I'd recommend looking at your use case with decision mgt best practices, beyond what Rhett suggested.

Also, for your additional considerations, there are companies specialising in consulting services on both those 2 dimenstions; I can think a few:

• IBM https://www.ibm.com/products/business-automation-manager-open-editions/decision-manager
• Aletyx https://aletyx.com
• Trisotech https://www.trisotech.com
• Blue Polaris https://bluepolaris.com

Hope this helps!

--
Matteo Mortari
youtube.com/@MatteoMortari
linkedin.com/in/matteomortari

To view this discussion visit https://groups.google.com/d/msgid/kogito-development/db3df9e7-b151-46d9-96c1-990eed55e13en%40googlegroups.com.

[Gabriele Cardosi]

Hi Rishab,

could you please share:

1. what is the kind of project you are building, i.e. it is a standalone jar/project or a "kogito" application?

2. could you please share the pom itself (if possible) ?

3. could you please share the command you issue to execute the build ?

4. last, could please let us know if those problems arise only during build time or also during execution ?

I would like to understand if the issue you're facing during build time may, somehow, happen also at runtime.

Besides that, I agree with Matteo that, without profiling, it is very hard to go much deeper...

Many thanks!

Best Regards

Gabriele

To view this discussion visit https://groups.google.com/d/msgid/kogito-development/CAHOoiEo1pi_3MwtT_zkapaHBr6_BJn-83xRDg-oO3nNmb6GNQw%40mail.gmail.com.

[Joseba Ercilla Olabarri]

Hi Rishab!

The default codegen approach is probably not the best option for your case. I would suggest exploring other options, like bypassing Kogito's build-time code generation by using the underlying kie-dmn-core to load and evaluate DMN models dynamically at runtime, with LRU caching for parsed runtimes and a filesystem-backed repository for model storage.

Regards.

_

Joseba

[Matteo Mortari]

Just to provide a bit of context, one potential way to achieve what Joseba is suggesting is analogous to this: https://www.ibm.com/docs/en/ibamoe/9.3.x?topic=scenarios-executing-decisions-in-embedded-java-mode#_kiecontainer

I'm positive we had also other simple ways to simply load it from FS like in the DMN JIT executor too

hth,

To view this discussion visit https://groups.google.com/d/msgid/kogito-development/5a353fca-7ce1-4d9d-b676-1ff50df3aa0cn%40googlegroups.com.