Quarkus + Kogito extension Maven project with generated DMN resources from another Maven dependency

[Bertrand GILLIS]

Hi,

I would like to manage Quarkus files and DMN files in different Maven projects because both projects will be maintained by different teams and will have also different lifecycles:

• The DMN files will be authored and maintained by Business teams.
• The Quarkus files will be authored and maintained by IT teams.

Both Maven projects are available on my shared folder on box.com.

https://app.box.com/s/6fsemy7ggqxdhv3cxl3v10bity0swg97

DMN files

DMN files are authored by a business team and persisted in a git repository.

This Maven project ("com.mainsysgroup.dmn:mainsysgroup-dmn-allocation-score-resources") produces a jar file containing a DMN file only:

• \pom.xml
• \src\main\resources\AllocationScore.dmn

It is not a Quarkus project.

This project can be built and install with "mvn clean install".

Quarkus files

Quarkus files are authored by an IT team and persisted in another git repository.

This Maven project ("com.mainsysgroup.dmn:mainsysgroup-dmn-allocation-score-quarkus") is the project that does not work as expected.

This project contains Quarkus + Kogito extension files only:

• \pom.xml
• \src\main\resources\application.properties

DMN files are not located in folder "\src\main\resources" of this project; as illustrated in https://github.com/kiegroup/kogito-examples.
DMN files are generated at build time from the Maven dependency "com.mainsysgroup.dmn:mainsysgroup-dmn-allocation-score-resources:jar" and reference as a resource directory in the current Maven project.

Excerpt from pom.xml for this purpose...

<properties>
  ...
  <dependency-plugin.version>3.6.1</dependency-plugin.version>
  ...
</properties>

...

<dependencies>
...

<!-- Maven dependency with DMM resources only -->
<dependency>
<groupId>com.mainsysgroup.dmn</groupId>
<artifactId>mainsysgroup-dmn-allocation-score-resources</artifactId>
<version>${project.version}</version>
</dependency>
...
</dependencies>

...
<build>
<resources>
<resource>
<directory>src/main/resources</directory>
</resource>

<!-- DMN generated resources unpacked from com.mainsysgroup.dmn:mainsysgroup-dmn-allocation-score-resources -->
<resource>
<directory>${project.build.directory}/generated-resources/dmn</directory>
</resource>
</resources>
  ...
 <plugins>

<!-- Unpacked DMN resources from com.mainsysgroup.dmn:mainsysgroup-dmn-allocation-score-resources -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>${dependency-plugin.version}</version>
<executions>
<execution>
<id>unpack</id>
<phase>generate-resources</phase>
<goals>
<goal>unpack</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>com.mainsysgroup.dmn</groupId>
<artifactId>mainsysgroup-dmn-allocation-score-resources</artifactId>
<version>${project.version}</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>${project.build.directory}/generated-resources/dmn</outputDirectory>
<includes>**/*.dmn</includes>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

This project can be run with "mvn clean process-resources quarkus:dev".

However the DMN file "AllocationScore.dmn" generated at build time and declared as a Maven resource is not detected by Kogito codegen.
So the DMN file is never exposed as a REST API by the Quarkus service at runtime!

Looking at the Kogito codegen source code I found that the Kogito Maven plugin lookups resources in directories following the pattern "src/main/**" only.

https://github.com/kiegroup/kogito-runtimes/blob/main/kogito-codegen-modules/kogito-codegen-core/src/main/java/org/kie/kogito/codegen/core/utils/ApplicationGeneratorDiscovery.java

Collection<CollectedResource> collectedResources = CollectedResourceProducer.fromPaths(context.ignoreHiddenFiles(), context.getAppPaths().getPaths());

https://github.com/kiegroup/kogito-runtimes/blob/main/kogito-codegen-modules/kogito-codegen-api/src/main/java/org/kie/kogito/codegen/api/context/impl/AbstractKogitoBuildContext.java

protected AppPaths appPaths = AppPaths.fromProjectDir(new File(".").toPath(), Path.of(".", AppPaths.TARGET_DIR));

https://github.com/kiegroup/drools/blob/main-apache/drools-model/drools-codegen-common/src/main/java/org/drools/codegen/common/AppPaths.java

    public static AppPaths fromProjectDir(Path projectDir, Path outputTarget) {
          return new AppPaths(Collections.singleton(projectDir), Collections.emptyList(), false, BuildTool.findBuildTool(), "main", outputTarget);
      }

Shouldn't the Kogito Maven plugin use Maven resources in a standard way instead of looking for them in an hardcoded path within Kogito codegen ? What do you think ?

Maybe could you also provide another solution to solve my use case (DMN and Quarkus files in separated git repositories and Maven artifacts) ?

Thanks for your help,

Bertrand

[Rhett S]

yea our team experienced something similar. we decided to disable the code gen, make the controller have static URLs, and pull the DMNs from a remote repo/jfrog at compile time. 

but that could be 1-4 sprints of work depending upon your NFRs. 

consider a shell-script that pulls the remote DMNs into the resources folder before maven runs. 

...imo kogito wasnt designed to handle all scenarios all teams will face. it is the best *starter*, cloud-native, no-code rule&process engine

imo, Rhett

--
You received this message because you are subscribed to the Google Groups "Kogito development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kogito-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kogito-development/cf79326f-1a65-42f8-b36e-58ba0606a3b8n%40googlegroups.com.

[Bertrand GILLIS]

Hi Rhett,

Thank you for your input.

However adding a shell script that pulls DMN files into the resources folder of a Maven project is nothing more than a similar way as the one I'm already using with Maven dependency plugin and its "unpack" goal.

I could unpack the DMN file directly into "\src\main\resources" folder instead of "${project.build.directory}/generated-resources/dmn" folder.

But, it is a very bad practice to mess up Maven project (re)sources with generated (re)sources.

Moreover, when I will try to release my Maven project  ("com.mainsysgroup.dmn:mainsysgroup-dmn-allocation-score-quarkus"), Maven release plugin will detect uncommitted changes and will refuse to do the release.

So your suggestion will generate other Maven issues because Maven best practices are not followed.

Regarding your first statement, I'm not sure to understand what your are trying to suggest.

      "we decided to disable the code gen, make the controller have static URLs, and pull the DMNs from a remote repo/jfrog at compile time."

If we need to disable the code gen of Quarkus Kogito extension, I guess we will have to develop manually all sources produced by code gen (as a Quarkus extension ?) and also update it at each DMN file update/release ("com.mainsysgroup.dmn:mainsysgroup-dmn-allocation-score-resources").

It is quite cumbersome... do you suggest then that using Kogito codegen should be avoided ???

You also talk about NFRs... Do you mean Not-For-Resale subscriptions ?

We are a Red Hat Partner and we own NFRs indeed.

What should I understand from your second statement ?

      "but that could be 1-4 sprints of work depending upon your NFRs."

Thanks a lot for your support and clarifications,

Bertrand 

[Rhett S]

hey  Bertrand 

- oh I completely agree that shell-script pulling in DMNs from another location is terrible design. :-)

- NFR = non-functional requirements. sorry, :-)

- re: disabling code-gen. the LoE isnt too bad. we mostly keep the same code that reads in all the DMNs from a specified location/directory/URI.  the biggest change is the controller. if you look at the generated kogito java mvc controller, it has an endpoint for every DMN. our modification: have one endpoint (method) that expects a parameter = DMNName, looks up and finds that DMN via a map, and executes that DMN. let me see if I can create a MVP for you this week. R

To view this discussion on the web visit https://groups.google.com/d/msgid/kogito-development/7cdd0a27-4140-4855-8ad8-e3cfaaf9470en%40googlegroups.com.

[Bertrand GILLIS]

Hi Rhett,

Did you manage to create a MVP in order to solve my issue ?

Kind regards,

Bertrand

[Rhett S]

Hi Bertrand, 

here you go. read the readme-from-rhett.md

I'll try to improve it in the future. 

https://github.com/x1c1cle/public-static-kogito-dmn/

To view this discussion on the web visit https://groups.google.com/d/msgid/kogito-development/4abac1c7-593f-48fa-a458-acfd3e3c80dbn%40googlegroups.com.

[Bertrand GILLIS]

Thanks a lot Rhett for taking the time to provide this Springboot example.

As Quarkus is our target, could please also provide an example with Quarkus ? 

Kr,

Bertrand

[Rhett S]

welcome B. 

I wont be able to give you a quarkus app; I dont have quarkus installed rn. if you can try, the steps for quarkus are very similar to sb. an annotation here and there.  i can maybe try in a few weeks, but I have a couple large deliverables for work. 

what we're trying to do here(  allow retrieval of DMNs from locations other than resources folder) is absolutely possible with quarkus however, there is one major caveat with quarkus and drools-dmn. let me explain below, as i understand it.

if you look on line 24 of DecisionModels.java, you'll see the kogito generated code lists out all the DMNs in a static block. 

    static {

        init(/* arguments provided during codegen */
        null, null, readResource(Application.class.getResourceAsStream("/TrafficViolation-v2.dmn"), "UTF-8"), readResource(Application.class.getResourceAsStream("/TrafficViolation-v1.dmn"), "UTF-8"));
    }

Why would kogito choose to list out the DMNs, and not iterate through the resources folder? it would seem that a retrieval of all DMNs in the resource directory would be a better design. less tight coupling to a set of DMNs. here is why, as i understand it.

when the DMNs are loaded into drools, there is drools code that does class introspection and reflection. (for both quarkus and sb)

quarkus does pre-compiling/AOT-complication of certain code to make the native executables start faster and take less memory than JVM. some of the code that quarkus pre-compiles is class introspection and reflection. 

so, when using quarkus the list of DMNs you want to load into your executable..must be known at compile time. 

this is why, the kogito team decided to put the list of DMNs to load in a static block, and listed out the dmn names explicitly. 

so, if you choose to use kogito quarkus, every time you change the inventory of DMNs in your executable/app/container, you'll need to rebuild your container/app/executable.

now, with spring-boot, it does not have the pre-compile step. so you can write code to "read all DMNs from a directory" build your container, run your container .and if your inventory of DMNs changes, all you need to do is stop and start your container. 

this is the first reason my team is using sb kogito/drools, and not quarkus. the second reason, is we've found (through extensive perf tests) that DMNs execute faster in spring boot- and that is a major criterion for us. 

(granted, quarkus apps start up 20x faster than sb- but that wasnt a major criterion for us.)

(if Im incorrect, kogito team, please chime in.)

To view this discussion on the web visit https://groups.google.com/d/msgid/kogito-development/7b7bd973-8ca3-4e7d-9b92-3be1a3c49cffn%40googlegroups.com.

[Gabriele Cardosi]

Hi Rhett,
I started working on a fix some time ago (https://github.com/apache/incubator-kie-issues/issues/847) using the reproducer kindly provided by Bertrand, but having other priorities I could not dedicate full time.
What I can say is that I could not find a quick and easy fix, because it must be something that works 
1 during compilation (codegen)
2 during runtime (execution)
3 for native images.

Having said that, I found your analysys interesting.
About " .and if your inventory of DMNs changes, all you need to do is stop and start your container. ":  this has been one of our internal recurrent discussions, in the past. And, the answer has been exactly what you wrote: by design, the idea is to shut down and restart the container, if the model changes.
As you wrote, this makes sense for native images, but there are a lot of different use-cases, so I think we'll have to modify things a bit to provide a better user-experience for all those different cases.

M2C

Best

Gabriele

To view this discussion on the web visit https://groups.google.com/d/msgid/kogito-development/CAMVsg7ApmNPNwkxmx-f03nVJq2xawkFpmmyfhRcvtM%2BcWoSHPw%40mail.gmail.com.