Serverless Workflow Versioning

[Jose Suero]

My serverless workflow process  have a 0 the version column (Postgres DB), I added property 

kogito.workflow.version-strategy=workflow

but still not adding version, what do I have to add?

[Cristiano Nicolai]

Hi Jose,

There are actually two columns, version is used for optimistic locking, so it will be updated based on changes to the process instance to avoid concurrent modifications. Then process_version is the one that will store the information regarding version strategy. By setting version-strategy=workflow it means it will store the version string that is defined in your Serverless Workflow file.

--
You received this message because you are subscribed to the Google Groups "Kogito development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kogito-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kogito-development/44a5f4c1-24ba-452c-b7f9-5b44f130a1abn%40googlegroups.com.

--

Best regards,

Cristiano Nicolai

[Jose Suero]

Thanks for your reply, I feel silly I didn't see the other column. 

The process version is indeed being saved, but is always taking the maven project version e.g. "1.0.0-SNAPSHOT" 

I tried with removing the property, and adding with workflow, workflows (don't ask) and it always saves project version

Any ideas?

I'm setting: 

kogito.workflow.version-strategy=workflow

[Cristiano Nicolai]

Hi Jose,

That's a bug, reported here https://issues.redhat.com/browse/KOGITO-8847

The fix should be merged soon.

Btw, workflow strategy is the default anyway, you can skip adding it.

To view this discussion on the web visit https://groups.google.com/d/msgid/kogito-development/1df4065d-bbc9-4c6b-9bd8-786c9f49c0dbn%40googlegroups.com.

[Jose Suero]

That makes sense, thanks for reporting it. 

I went ahead and changed my project version and now the app is not finding any of the processes created on the previous version. Is this expected behaviour? 

I'm also trying to see if it's possible to fix my processes to adapt to the new version. For what I understand debugging it loads project data as a byte field from database and tries to unmarshall it to the current process definition, what I'm not fully getting is where the "old" definition is suppose to be stored so it can continue finish the processes when a change in version is made. 

Would you be as kind as to give me a little bit more information into how this works?

thanks in advance

Jose

You received this message because you are subscribed to a topic in the Google Groups "Kogito development mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kogito-development/gKzTu8p8ACs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kogito-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kogito-development/CAOVo6cQaU_yhVLeLO62uvX_8FKcRJPve7z-JtFzarB5%3DBiar8Q%40mail.gmail.com.

[Cristiano Nicolai]

Hi Jose,

Sure, at this stage, versioning support does not support any migration, so there is really no guarantee that if you load a new sw definition using an instance/binary data from another version that it will work.

The version strategy works to isolate the runtimes, so you can run multiple versions in parallel. This allows you to keep the older versions of runtime running while process instances are still in flight. Then redirecting new requests to new versions of the service. So isolation happens on the service level itself.

Another possibility is to, once you deploy a new version, manually migrate instances from one version to another. You can do that by loading the SW data, starting a new instance with that as payload and defining the X-KOGITO-StartFromNode header. That will bypass any node execution in the new instance before that point. 

To view this discussion on the web visit https://groups.google.com/d/msgid/kogito-development/CAJcoKwzjPEHfLN9bEZOv04b5xkphSCNNc%2BA_8r-dBuMZVtPGpA%40mail.gmail.com.

[Jose Suero]

I think a migration will work for me since making a service layer just won't work since my service has many workflows and it would be a nightmare to keep track of them individually, on a side note I think Kogito needs to consider this, for enterprise services not being able to adjust your workflow will be very problematic.

I'm currently sending a cloud event to start my workflow, this has a correlation id, I see the header is used on the post for the auto generated resource endpoint which, as far as I understand doesn't use cloudevents, how would I add my correlation on that request?

[Cristiano Nicolai]

On Wed, Mar 15, 2023 at 7:38 PM Jose Suero <josep...@gmail.com> wrote:

I think a migration will work for me since making a service layer just won't work since my service has many workflows and it would be a nightmare to keep track of them individually, on a side note I think Kogito needs to consider this, for enterprise services not being able to adjust your workflow will be very problematic.

Agree, there are ideas to support more complex use cases.

 

I'm currently sending a cloud event to start my workflow, this has a correlation id, I see the header is used on the post for the auto generated resource endpoint which, as far as I understand doesn't use cloudevents, how would I add my correlation on that request?

I believe you would have to switch to quarkus-http connector, and post the cloudevent as payload in there.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/kogito-development/fbb0b9a0-5096-4ca0-bc1e-ae3938a23351n%40googlegroups.com.

[Jose Suero]

This is what I'm using, but it doesn't seem to take the "X-KOGITO-StartFromNode" header into account

[Tiago Dolphine]

In this case, you need to send the startFromNode as a cloud event extension attribute, the attribute name should be kogitoprocstartfrom.

The X-KOGITO-StartFromNode is only used by the generated domain REST endpoints, if you use the quarkus-http connector, kafka or whatever this info is part of the cloud event as an extension attribute.

Best regards,

Tiago Dolphine

Principal Software Engineer

Red Hat

tdol...@redhat.com   

To view this discussion on the web visit https://groups.google.com/d/msgid/kogito-development/7c5ec933-547a-4b61-b6a1-28084dd76aban%40googlegroups.com.

[Jose Suero]

Pefect! just what I needed, thank you both for your help! 

[Jose Suero]

I found a super weird behaviour, when I add kogitoprocstartfrom it starts the process and populates correlation_instances and pushed data to data-index, so processes and nodes have data, nodes have the correct node data so looks like process is indeed in the right step. But main table (process_instances) has does not have data on that process. When I don't include this extension all  tables data get populated correctly.

So process can not be used, any idea?

any ideas?

On Thursday, March 16, 2023 at 11:37:09 AM UTC+1 tdol...@redhat.com wrote:

[Tiago Dolphine]

Are these processes you are starting with the kogitoprocstartfrom in a wait state after they are started? I mean are they active or completed? because if they are completed they are removed from the process_instances, but if they are active on a wait state they are supposed to be there.

Tiago Dolphine

Principal Software Engineer

Red Hat

tdol...@redhat.com   

To view this discussion on the web visit https://groups.google.com/d/msgid/kogito-development/24859130-c130-4fa0-9378-d051037d255en%40googlegroups.com.

[Jose Suero]

I really shouldn't end, this is a split node waiting for events on the logs is shows:

2023-03-16 15:52:37,261 INFO  [org.kie.kog.eve.imp.ProcessEventDispatcher] (kogito-event-executor-1) Starting new process instance with signal 'new_exception_event_type'
2023-03-16 15:52:45,136 INFO  [org.kie.kog.ser.wor.dev.DevModeServerlessWorkflowLogger] (kogito-event-executor-1) Triggered node 'Wait For Answer' for process 'exceptionfailed' (6e3cd4a4-b751-41f2-af5a-7efc102e795e)
2023-03-16 15:52:45,172 INFO  [org.kie.kog.ser.wor.dev.DevModeServerlessWorkflowLogger] (kogito-event-executor-1) Triggered node 'HandlingUnitCanceledEvent' for process 'exceptionfailed' (6e3cd4a4-b751-41f2-af5a-7efc102e795e)
2023-03-16 15:52:45,206 INFO  [org.kie.kog.ser.wor.dev.DevModeServerlessWorkflowLogger] (kogito-event-executor-1) Triggered node 'ManuallyResolvedProcessEvent' for process 'exceptionfailed' (6e3cd4a4-b751-41f2-af5a-7efc102e795e)
2023-03-16 15:52:45,244 INFO  [org.kie.kog.ser.wor.dev.DevModeServerlessWorkflowLogger] (kogito-event-executor-1) Triggered node 'HandlingUnitReleasedEvent' for process 'exceptionfailed' (6e3cd4a4-b751-41f2-af5a-7efc102e795e)
2023-03-16 15:52:45,287 INFO  [org.kie.kog.ser.wor.dev.DevModeServerlessWorkflowLogger] (kogito-event-executor-1) Triggered node 'TimerNode_71' for process 'exceptionfailed' (6e3cd4a4-b751-41f2-af5a-7efc102e795e)

so there's no mentioning of it ending, 

The actual node looks like:

{

"name": "Wait For Answer",

"type": "switch",

"eventConditions": [

{

"eventRef": "HandlingUnitCanceledEvent",

"transition": "Set status to unit cancelled"

},

{

"eventRef": "HandlingUnitReleasedEvent",

"transition": "Set status to unit released"

},

{

"eventRef": "ManuallyResolvedProcessEvent",

"transition": "Set status to manually resolved"

}

],

"timeouts": {

"eventTimeout": "P61D"

},

"defaultCondition": {

"transition": "Create SupportHub Ticket"

}

},

My node id is 48, I found on the workflow definition:

org.jbpm.ruleflow.core.factory.SplitFactory<?> splitNode48 = factory.splitNode(48);

splitNode48.name("Wait For Answer");

splitNode48.type(4);

splitNode48.metaData("UniqueId", "48");

splitNode48.metaData("EventBased", "true");

splitNode48.done();

And nodes show that all those node have exit null on the nodes DB so it's doesn't appear to be ending the workflow

Any clues?

[Tiago Dolphine]

It would be better if you share a reproducer for this issue with the workflow and steps you executed, a simple project. It seems that the workflow got stuck at some point after the timerNode or it was completed, these are the 2 situations I see that may cause it to not be present in the database.

Tiago Dolphine

Principal Software Engineer

Red Hat

tdol...@redhat.com   

To view this discussion on the web visit https://groups.google.com/d/msgid/kogito-development/26c033a8-bc80-4979-9cc3-386bda2567c2n%40googlegroups.com.

[Jose Suero]

Please take a look at https://github.com/josesuero/serverless-workflow-correlation-quarkus

I'm running:

docker-compose up

quarkus dev

and posting to http://localhost:8080/start

with body:

{

"specversion": "1.0",

"id": "ed6d712e-9b14-47f7-a72f-3654000772cf",

"source": "",

"type": "new_exception_event_type",

"time": "2023-03-16T15:16:12.020944+01:00",

"kogitoprocstartfrom": "48",

"exceptionid": "fd7493aa-da20-47ff-bede-1b2c8e021819",

"data": {

"status": "created"

}

}

Thanks again for looking at this

[Jose Suero]

I started logging the Queries on postgres and I think I know the process inserts into correlation_instances, and tries to update process_instances, which because this is a new process does not exists

LOG:

2023-03-17 16:43:59 2023-03-17 15:43:59.512 UTC [59] LOG:  execute <unnamed>: SELECT correlated_id, correlation FROM correlation_instances WHERE encoded_correlation_id = $1
2023-03-17 16:43:59 2023-03-17 15:43:59.512 UTC [59] DETAIL:  parameters: $1 = '0de60a56109ce3669ee93fdb9ee59e99'
2023-03-17 16:43:59 2023-03-17 15:43:59.566 UTC [59] LOG:  execute <unnamed>: INSERT INTO correlation_instances (id, encoded_correlation_id, correlated_id, correlation) VALUES ($1, $2, $3, $4::json)
2023-03-17 16:43:59 2023-03-17 15:43:59.566 UTC [59] DETAIL:  parameters: $1 = '8c27e78a-2327-4f0b-9ca7-bc7a4f6b3c18', $2 = '0de60a56109ce3669ee93fdb9ee59e99', $3 = '9b51f27c-6199-482e-bd9b-68caaefcb746', $4 = '{"key":"exceptionid","value":[{"key":"exceptionid","value":"fd7493aa-da20-47ff-bede-1b2c8e021819"}]}'
2023-03-17 16:43:59 2023-03-17 15:43:59.664 UTC [59] LOG:  execute <unnamed>: UPDATE process_instances SET payload = $1 WHERE process_id = $2 and id = $3and process_version = $4
2023-03-17 16:43:59 2023-03-17 15:43:59.664 UTC [59] DETAIL:  parameters: $1 = '\x0a025357120871634661696c65641a2439623531663237632d363139392d343832652d626439622d3638636161656663623734363a1a48616e646c65205143204661696c656420457863657074696f6e400148d0f8f781ef3058018a010218009201a6080ab0010a0c776f726b666c6f776461746112346f72672e6b69652e6b6f6769746f2e6a61636b736f6e2e7574696c732e4f626a6563744e6f64654c697374656e657241776172651a6a0a4a747970652e676f6f676c65617069732e636f6d2f6f72672e6b69652e6b6f6769746f2e73657269616c697a6174696f6e2e70726f636573732e70726f746f6275662e4a736f6e4e6f6465121c0a1a7b0a20202273746174757322203a202263726561746564220a7d1293010a2434376363333865632d626534632d346666362d386166352d373734323839316336383662104b1a5c0a5a747970652e676f6f676c65617069732e636f6d2f6f72672e6b69652e6b6f6769746f2e73657269616c697a6174696f6e2e70726f636573732e70726f746f6275662e4576656e744e6f6465496e7374616e6365436f6e74656e74200128daf8f781ef30320218001293010a2438356534383238392d623532612d346230312d383332332d63313930343361613165616410481a5c0a5a747970652e676f6f676c65617069732e636f6d2f6f72672e6b69652e6b6f6769746f2e73657269616c697a6174696f6e2e70726f636573732e70726f746f6275662e4576656e744e6f6465496e7374616e6365436f6e74656e74200128daf8f781ef30320218001293010a2461303736646639312d343737312d346563632d383234652d66376566396630643535636310451a5c0a5a747970652e676f6f676c65617069732e636f6d2f6f72672e6b69652e6b6f6769746f2e73657269616c697a6174696f6e2e70726f636573732e70726f746f6275662e4576656e744e6f6465496e7374616e6365436f6e74656e74200128d5f8f781ef303202180012c4010a2466333463376135622d366233362d343364652d623838322d30383636326136343530323610431a8c010a5a747970652e676f6f676c65617069732e636f6d2f6f72672e6b69652e6b6f6769746f2e73657269616c697a6174696f6e2e70726f636573732e70726f746f6275662e54696d65724e6f6465496e7374616e6365436f6e74656e74122e0a2c54494d45523a303a62373637663764372d653231622d343764372d623237352d353662356335343462363332200128d7f8f781ef30320218001a98010a2466333463376135622d366233362d343364652d623838322d3038363632613634353032360a2438356534383238392d623532612d346230312d383332332d6331393034336161316561640a2434376363333865632d626534632d346666362d386166352d3737343238393163363836620a2461303736646639312d343737312d346563632d383234652d66376566396630643535636322060a023435100122060a023637100122130a0f5f6a62706d2d756e697175652d3534100122130a0f5f6a62706d2d756e697175652d3536100122130a0f5f6a62706d2d756e697175652d35381001', $2 = 'qcFailed', $3 = '9b51f27c-6199-482e-bd9b-68caaefcb746', $4 = '1.0.0-SNAPSHOT'

[Tiago Dolphine]

Nice, thanks for the update probably that's the root cause of the issue, I opened this Jira to track it https://issues.redhat.com/browse/KOGITO-8880

  

Tiago Dolphine

Principal Software Engineer

Red Hat

tdol...@redhat.com   

To view this discussion on the web visit https://groups.google.com/d/msgid/kogito-development/78343a94-27f4-437a-a4d3-c8c4da0df12cn%40googlegroups.com.

[Jose Suero]

any ideas what I can do as a workaround to migrate my processes? multiple steps are ok as long as I can end in the right step with a correlation

[Jose Suero]

Hi, 

any updates on this fix?

[Tiago Dolphine]

Fix was merged https://github.com/kiegroup/kogito-runtimes/pull/2885, it should be available in the next release 1.37.

Tiago Dolphine

Principal Software Engineer

Red Hat

tdol...@redhat.com   

To view this discussion on the web visit https://groups.google.com/d/msgid/kogito-development/5fe41151-f745-4f47-9c60-bf2c5f00b4a2n%40googlegroups.com.

[Jose Suero]

I just tried on 1.37, same result :-(, can you please confirm this was in fact added to 1.37?

is there an easy way to use it from the source? 

[Tiago Dolphine]

Hi Jose,

It should be there since https://github.com/kiegroup/kogito-runtimes/commit/e0a4633f13f45693a2856786bef9eb2e0104d7b8 is on 1.37.x branch.

Are you using the nodeId or the node name, the node id is "auto" generated by the process but it could change over time in case this process is changed. Can you try to set a node name and use it in the extension attribute kogitoprocstartfrom just to check? 

There is an IT for this https://github.com/kiegroup/kogito-runtimes/blob/e0a4633f13f45693a2856786bef9eb2e0104d7b8/quarkus/extensions/kogito-quarkus-serverless-workflow-extension/kogito-quarkus-serverless-workflow-integration-test/src/test/java/org/kie/kogito/quarkus/workflows/CorrelationIT.java#L104-L106.

Best regards,

Tiago Dolphine

Principal Software Engineer

Red Hat

tdol...@redhat.com   

To view this discussion on the web visit https://groups.google.com/d/msgid/kogito-development/202aaa3f-2003-4dfc-b4df-30e0225f125cn%40googlegroups.com.

[Jose Suero]

I apologize, My POM was not updating correctly, I fixed it and just test it and works very well.

I also checked https://issues.redhat.com/browse/KOGITO-8847 and is also included and fixed, 

thanks a lot for all the help!

Jose