Deploying Kie servers and Drools Workbench on Kubernetes cluster

[Yogesh Jeswani]

Hi Team,

I have been currently working on setting up the development environment for business analysts to use Drools. 

Here is what i have up and running right now:

1) Drools workbench is up and running on Kubernetes cluster. I have used Docker image and made configuration changes. 

2) Kie server is up and running on kubernetes cluster. used docker image for this as well.

In terms of deployment, I have synched gitlab repo with drools workbench , so that any changes made on drools workbench will be pushed to git repo. CI/CD pipeline will then run which will build, publish the kjar in an artifactory and will be deployed on kie server using kieserver maven plugin.

My Question is, how can i make sure the kjar is available for kie - server in case of pod gets deleted and new pod gets created. How can i automate this and make sure that whenever a new pod is created, it should have all the kie-containers deployed before?

Please do help me out with my query.

Thanks

Yogesh Jeswani

[Karel Suta]

Hi Yogesh,

>How can i automate this and make sure that whenever a new pod is created, it should have all the kie-containers deployed before?

You can achieve it by connecting Kie server to the workbench (so Kie server becomes managed, see https://docs.jboss.org/drools/release/latest/drools-docs/html_single/#_managed_kie_server). For Kie server to become managed you need to configure system properties in Kie server:

org.kie.server.id

org.kie.server.location

org.kie.server.controller.user

org.kie.server.controller.pwd

For more info about these properties see https://docs.jboss.org/drools/release/latest/drools-docs/html_single/#kie-server-system-properties-ref_kie-apis.

Also Drools workbench needs to have these system properties:

org.kie.server.user

org.kie.server.pwd

Both Kie server and Drools workbench need to be able to reach each other.

>how can i make sure the kjar is available for kie - server in case of pod gets deleted and new pod gets created

That can be achieved by referencing artifactory in a custom Maven settings.xml file. Then store the settings.xml in image and set Kie server system property kie.maven.settings.custom to point to this settings.xml. Kie server will replace its default settings.xml with provided one and properly resolve artifacts based on repositories defined there.

Karel Suta

[Yogesh Jeswani]

Hi Karel,

Thanks for the response. I do have all the properties set as you mentioned. 

I have tried running drools workbench and kie server on local on port 8080 and 8180 respectively. I do have those properties set .

For kie server , these are the parameters I am passing while starting ./standalone.sh . I read somewhere that we need websocket if trying to connect to remote kie server. ( Could you please confirm)

As you can see, i do have custom maven setings.xml to point to my artifactory.

-server-config=standalone-full.xml -Dorg.kie.server.id=wildfly-kieserver -Dorg.kie.server.location=http://localhost:8180/kie-server/services/rest/server -Dorg.kie.workbench.controller=ws://localhost:8080/business-central/websocket/controller -Dorg.kie.workbench.controller.user=kieserver -Dorg.kie.workbench.controller.pwd=kieserver1! -Dorg.kie.server.user=kieserver -Dorg.kie.server.pwd=kieserver1! -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true -Dkie.maven.settings.custom=/opt/jboss/.m2/settings.xml

For Drools Workbench, these are the parameters. 

--server-config=standalone-full.xml -Dorg.kie.server.user=kieserver -Dorg.kie.server.pwd=kieserver1! -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true -Dorg.kie.server.location=http://localhost:8180/kie-server/services/rest/server -Dorg.kie.workbench.controller=ws://localhost:8080/business-central/websocket/controller -Dorg.kie.workbench.controller.user=kieserver -Dorg.kie.workbench.controller.pwd=kieserver1! -Dorg.uberfire.nio.git.hooks=$APP_SERVER_HOME/hooks/ -Dkie.maven.settings.custom=/opt/jboss/.m2/settings.xml -Dorg.kie.server.user=kieserver -Dorg.kie.server.pwd=kieserver1!

Both are docker images running on my local and the images have their own wildfly servers. I am not seeing kie servers in Drools workbench after above settings. 

My questions was more for real time. Let say I have few containers up and running on a kie server and all of a sudden, pod gets deleted due to some issues. In that case, new pod comes up instantly but without any containers. Now, we need to deploy the containers again on new pod. how can we achieve this automation that whenever new pod gets created, it should get all the required containers to serve traffic?

Thanks in advance!!

Regards

Yogesh

[Karel Suta]

Hi Yogesh.

> I read somewhere that we need websocket if trying to connect to remote kie server.

No, you can use either HTTP or WebSockets, it is up to you. The advantage of WebSockets is that Workbench doesn't need Kie server URL as it use WebSockets to pass container actions to Kie server. But WebSockets keeps an open connection between Kie server and Workbench.

> I am not seeing kie servers in Drools workbench after above settings.

Thinking whether the Workbench and Kie server can actually reach localhost, as they run in container. Did you apply some specific configuration to achieve it like [1]?

> Now, we need to deploy the containers again on new pod. how can we achieve this automation that whenever new pod gets created, it should get all the required containers to serve traffic?

There are two options.

In case of unmanaged Kie server (which doesn't connect to the Workbench) - you can provide Kie server state file which will contain all the configuration and deployed containers.

For managed Kie server (your case) the Kie server should fetch containers from Workbench. For this to work the Kie server and Workbench need to have an access to each other.

Karel

[1] https://stackoverflow.com/questions/24319662/from-inside-of-a-docker-container-how-do-i-connect-to-the-localhost-of-the-mach

[Yogesh Jeswani]

Hi Karel,

Thanks for answering above questions. I was able to get it work and link the kie servers. Here is where I am struggling right now:

1) Since I am using Docker images in kubernetes cluster, what is the best way to persist the configurations I make in Drools Workbench? For example, If i create new users, import some projects, how can i persist so that new pod will have that info available on top of docker image?

2) What is the best way to create clusters for kie servers. can i just have multiple kie servers with same kie server id? I have been using kie server maven plugin to deploy kjars in our kie servers, I am able to deploy it on single kie server running on kubernetes cluster but how can we setup to have multiple kie servers and use canary deployments? 

Any thoughts will be appreciated.

Thanks

Yogesh Jeswani

[Karel Suta]

Hi Yogesh

>1) Since I am using Docker images in kubernetes cluster, what is the best way to persist the configurations I make in Drools Workbench? For example, If i create new users, import some projects, how can i persist so that new pod will have that info available on top of docker image?

I would say that best option is to mount a persistent volume, then redirecting Workbench to use the volume by specifying path to Workbench Git directory using system property "org.uberfire.nio.git.dir".

>2) What is the best way to create clusters for kie servers. can i just have multiple kie servers with same kie server id? I have been using kie server maven plugin to deploy kjars in our kie servers, I am able to deploy it on single kie server running on kubernetes cluster but how can we setup to have multiple kie servers and use canary deployments?

The expectations are that Kie servers with same Kie server ID have same containers deployed (so they are interchangeable), specially when Kie servers are managed. For such scenario try to deploy Kie server with different ID and handling canary deployments by routing (i.e. Ingress?).

Karel

[Yogesh Jeswani]

Hi Karel,

Thanks for your response.

Is there a way to define the server id dynamically, because when the new pod starts up, since its using the same docker image, it uses the same kie server id. How can I dynamically assign kie server id when new pod starts up and where should i specify configurations to let the pod know which containers to deploy? Sorry if I am asking lot of questions, but I am really stuck at this phase. 

Regards

Yogesh

[Karel Suta]

Hi Yogesh

>Is there a way to define the server id dynamically,

Yes, you can set it by environment variable "KIE_SERVER_ID", environment variable value is applied in case system property "org.kie.server.id" is not set.

> where should i specify configurations to let the pod know which containers to deploy?

In case of managed Kie servers the configuration is taken from Workbench. When Kie server ID match the server template name in Workbench then server template configuration is applied to the Kie server.

In case of unmanaged Kie server you need to provide server state file. This file contains Kie server configuration and deployed containers. The easiest way to obtain it is to take an existing one and modify it according to your needs. It is usually located in bin folder of EAP. It is named "<Kie server ID>.xml". Once you modify it then either create a ConfigMap in Kubernetes using this file and mount it or mount is as persistent volume.

You can specify path to the folder with server state file using system property "org.kie.server.repo"

Karel

[Yogesh Jeswani]

Hi Karel,

I am using wildfly-kieserver as my server id. I deployed one container and used that wildfly-kieserver.xml file. I added a mount for .m2/repository folder to make sure files are present and added the configuration "org.kie.server.repo=/opt/jboss/wildfly/bin" , this is the location where I am storing my wildfly-kieserver.xml file. I tried creating a new container with this setup but it doesn't seems to be working. I used Rest endpoint to get the list of containers, but no results were found. Container was not instantiated. Is there anything missing in the setup. My assumption was at startup, kie server will look into wildfly-kieserver.xml file and try to regain that state. 

Please do let me know.

Thanks

Yogesh

[Karel Suta]

Hi Yogesh,

I suppose you are using unmanaged Kie servers, right?

The "org.kie.server.repo" needs to point to a volume, so it can be persisted and shared. Kie server should look to wildfly-kieserver.xml file in folder defined by "org.kie.server.repo" syst property, but the file needs to be persisted.

Karel

[Yogesh Jeswani]

Thanks a lot , it worked. Also, do we need a kie router to scale or kubernetes cluster can take care of scaling kie servers?

Regards

Yogesh

[Karel Suta]

Kie router has more meaning for jBPM applications. In case of Drools it is enough to rely on Kubernetes clustering.

Karel

[Yogesh Jeswani]

Thanks Karel for the response.

Next, I was trying to persist the Workbench configurations but struggling again. 

Could you please let me know if these configs are correct:

exec ./standalone.sh -b $JBOSS_BIND_ADDRESS --server-config=standalone-full.xml -Dorg.kie.server.user=kieserver -Dorg.kie.server.pwd=kieserver1! -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true  -Dorg.uberfire.nio.git.hooks=$APP_SERVER_HOME/hooks/ -Dkie.maven.settings.custom=/opt/jboss/.m2/settings.xml -Djboss.server.base.dir=/Users/yogeshjeswani/Downloads/standalone -Dorg.guvnor.m2repo.dir=/Users/yogeshjeswani/Downloads/m2repo -Dorg.uberfire.nio.git.dir=/Users/yogeshjeswani/Downloads/git -Dorg.kie.server.repo=/Users/yogeshjeswani/Downloads -Dorg.uberfire.nio.git.ssh.host=0.0.0.0 -Dorg.uberfire.nio.git.ssh.enabled=true

here are the JAVA_OPTS: ENV JAVA_OPTS -Xms256m -Xmx2048m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=512m -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 Djboss.home.dir=/Users/yogeshjeswani/Downloads/ -Djboss.server.config.user.dir=/Users/yogeshjeswani/Downloads/standalone/configuration

while running, I am trying to create a mount on /opt/jboss/wildfly/standalone folder so that the data will be persisted.

basically, the need is i need to persist the configurations and the git data so that it can be used by all the pods in the kubernetes cluster. could you please help me with that.

Thanks

Yogesh

[Karel Suta]

>while running, I am trying to create a mount on /opt/jboss/wildfly/standalone folder so that the data will be persisted.

From the configuration it looks like you are trying to mount path in container /Users/yogeshjeswani/Downloads, is it so?

Can you check mounted location to see if the files are properly stored there? Does the mount volume have proper rights so system can store files there?

Karel

[Yogesh Jeswani]

Hi Karel,

There was some issue with the way I was mounting it. It's working now. Thanks

[Yogesh Jeswani]

Hi Karel,

Can we use persistent volume for scaling workbench or there is a need for vfs clustering as mentioned on the website?

[Yogesh Jeswani]

Hi Karel,

I am trying to mount a persistent volume on the standalone folder to persist all the configurations, but its throwing this error:

"Unable to read the logging configuration from 'file:/opt/jboss/wildfly/standalone/configuration/logging.properties' (java.io.FileNotFoundException: /opt/jboss/wildfly/standalone/configuration/logging.properties (No such file or directory))

java.lang.IllegalStateException: WFLYSRV0121: Configuration directory does not exist: /opt/jboss/wildfly/standalone/configuration

at org.jboss...@11.1.1.Final//org.jboss.as.server.ServerEnvironment.<init>(ServerEnvironment.java:449)

at org.jboss...@11.1.1.Final//org.jboss.as.server.Main.determineEnvironment(Main.java:388)

at org.jboss...@11.1.1.Final//org.jboss.as.server.Main.main(Main.java:96)

at org.jboss.modules.Module.run(Module.java:352)

at org.jboss.modules.Module.run(Module.java:320)

at org.jboss.modules.Main.main(Main.java:593)"

I am using a stateful set and defined a volumeclaimtemplate like this:

here is the snippet of my yaml file:

volumeMounts:
- name: pv-data
mountPath: /opt/jboss/wildfly/standalone
volumeClaimTemplates:
- metadata:
name: pv-data
spec:
accessModes: [ "ReadWriteMany" ]
resources:
requests:
storage: 10Gi

I have been trying to resolve this issue since last 48 hrs but not able to do that. Could you please help.

Thanks

Yogesh 

[Karel Suta]

Hi Yogesh,

I think the problem is that EAP expects logging.properties file to exist in the folder. When you mount the PV then it is empty.

What all do you need to actually persist?

Karel

[Yogesh Jeswani]

Hi Karel,

I was looking to persist the configuration files so that data should be available even if pod goes down. what is the best way to do that? I think i identified my mistake by copying the standalone folder to my PV manually first and then mounted it. is this how we should persist data for workbench? Also, I am trying to restrict users from writing to "master" repo in branch management but that doesn't seems to work. Could you please let me know how we can restrict users to merge their changes to master branch?

Regards

Yogesh

[Karel Suta]

Hi Yogesh,

It should be enough to persist only niogit directory and Maven repo. IMHO the standalone folder itself doesn't need to be shared.

About the master branch, I am not sure about that, maybe somebody else here can help with that.

Karel

[Yogesh Jeswani]

Hi Karel,

If we don't persist standalone folder, how can we persist all the configurations? Configurations is a part of standalone folder?

Also, should we use standalone mode or domain mode in production? i was trying to look for the info, but couldn't find much as to which one to use.

Thanks

Yogesh

[Karel Suta]

Hi Yogesh,

Thinking about what configuration do you want to persist. Do you need to modify the configuration on runtime? If not you can do the configuration changes directly in the image, so every pod will have same configuration out of the box when it starts.

You should use standalone mode, domain mode is AFAIK deprecated.

Karel

[Yogesh Jeswani]

Got it. My use case is I should be able to create new users for business analysts, developers, so for that purpose I need to persist those configs. 

[Karel Suta]

hmm, in such case I see three options

- You can persist whole config folder, but in such case you need to create the folder with expected content on PV first. Also I am not sure if will the WildFly work properly when multiple instances will access the shared config folder.

- You can store user files in ConfigMap/Secret and inject them. But in such case they are immutable, you would have to adjust the ConfigMap/Secret every time you want to add/remove user.

- You can leverage external user management tool (i.e. Keycloak) and integrate Business central with it.

Karel

[Yogesh Jeswani]

Thanks for the response. I am going with the first option. 

I was reading about kogito but not sure how to use it? I have Drools Workbench (business central war file) Dockerfile which I am starting up in kubernetes cluster with a PV. I do have a repo based out of kie jar archetype which can be imported in workbench to make any changes, and then I have a dockerfile for kie server which is up and running on kubernetes cluster. The flow is "user can make a change in Drools workbench in imported project" which then be pushed and published to our artifactory using gitlab CI/CD pipeline. The jar files will be then deployed to kie servers after making change in state file. Could you please let me know if this is ideal setup and where will kogito comes into picture in this setup? 

Regards

Yogesh 

[Rafael Soares]

Kogito uses a completely different runtime/deployment approach.

There are no KJAR or Kie-Server concepts in Kogito land. Kogito Services are kind of self-contained in a sense that your project (src) contains only your models (BPMN for Process Definitions and DMN for Decisions). You can package these models both with Quarkus or SpringBoot runtimes. Kogito will auto-generate the service code and expose your processes/decisions as Restful MicroServices.

The best way to understand Kogito is to follows its getting started guides: https://kogito.kie.org/get-started/

and try its samples: https://github.com/kiegroup/kogito-examples

________________________
Rafael Torres Coelho Soares

--
You received this message because you are subscribed to the Google Groups "Drools Usage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to drools-usage...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/drools-usage/68c16c9e-caec-40d6-9de1-b266b0aba47dn%40googlegroups.com.

[Yogesh Jeswani]

Got it. Thanks for the response. So in my use case, I need to use workbench so that non developers can create rules and make any changes. Could you please confirm if my setup with Workbench, kie servers and kie jar is the best way to go? Basically, the constraint is to have a workbench to create or edit rules.. please do let me know.

Thanks

Yogesh

[Rafael Soares]

Well, reading your deployment workflow

I have Drools Workbench (business central war file) Dockerfile which I am starting up in kubernetes cluster with a PV. I do have a repo based out of kie jar archetype which can be imported in workbench to make any changes, and then I have a dockerfile for kie server which is up and running on kubernetes cluster. The flow is "user can make a change in Drools workbench in imported project" which then be pushed and published to our artifactory using gitlab CI/CD pipeline. The jar files will be then deployed to kie servers after making change in state file. Could you please let me know if this is ideal setup and where will kogito comes into picture in this setup? 

I would only argue this statement:

 The jar files will be then deployed to kie servers after making change in state file

Containers are meant to be immutable (at least from a cloud-native perspective). So It would be more "cloud compliant" to have your kie-server image built at every CI/CD pipeline run. But this is something you have to decide with your team.

________________________
Rafael Torres Coelho Soares

To view this discussion on the web visit https://groups.google.com/d/msgid/drools-usage/e4025b4d-73ff-4ce1-8eae-f6414deeb031n%40googlegroups.com.

[Yogesh Jeswani]

Hi Rafael,

ok, ConfigMap is a part of my kie server repo, which actually builds a kie-server image at every CI/CD run and deploys the containers mentioned in my configmap kieserverid.xml file. 

Can we create our own rest endpoints for the containers deployed on our kie servers or have to use the swagger docs provided by kie server to query/evaluate the decisions? I will be designing a DMN for my use case.. 

Thanks

Yogesh

[Rafael Soares]

Well, if you wanna stick with standard Kie-Server you can extend it by adding your own Rest endpoints. See the following post for this approach: https://blog.kie.org/2015/12/kie-server-extend-existing-server-capability-with-extra-rest-endpoint.html

Another approach would be to use the engine in embedded mode. So you can for instance build your own Service (eg. using a SpringBoot app) and use the Kie libraries to manage your kjars exposing your own endpoints. You can find an example of this approach here: https://reachmnadeem.wordpress.com/2021/05/09/spring-boot-starters-for-kie-and-jbpm-servers/

________________________
Rafael Torres Coelho Soares

To view this discussion on the web visit https://groups.google.com/d/msgid/drools-usage/532698f9-415f-4c7a-b9f6-1e9e675dc3e9n%40googlegroups.com.

[facu lamas]

Hello, everyone, I want to deploy drools and kie with helm, does anyone have any documentation that can guide me?

[Luca Molteni]

> On 31 Jul 2023, at 19:39, facu lamas <facu.carne...@gmail.com> wrote:
>
> Hello, everyone, I want to deploy drools and kie with helm, does anyone have any documentation that can guide me?
> El viernes, 17 de septiembre de 2021 a la(s) 11:20:41 UTC-3, rafa...@gmail.com escribió:
> Well, if you wanna stick with standard Kie-Server you can extend it by adding your own Rest endpoints. See the following post for this approach: https://blog.kie.org/2015/12/kie-server-extend-existing-server-capability-with-extra-rest-endpoint.html
>
> Another approach would be to use the engine in embedded mode. So you can for instance build your own Service (eg. using a SpringBoot app) and use the Kie libraries to manage your kjars exposing your own endpoints. You can find an example of this approach here: https://reachmnadeem.wordpress.com/2021/05/09/spring-boot-starters-for-kie-and-jbpm-servers/

Hi there’s not an helm chart to deploy Drools currently - I’m afraid you will have to create one by yourself

Luca