New processes with a Message Start do not complete after an engine restarts

[Gareth]

Hi All,

Im wondering if anyone can help / clarify the expected behaviour in the following scenario.

Setup:

Camunda 7.2

Tomcat

Use Case:

As a user,

I want to deploy a process with a message start, that I can start at any time ( including after restarting Camunda ) with a simple message send,

So that I do not have to deploy a new version of my process everytime I want to execute it.

Observed behaviour:

I can deploy my process definition with a message start, trigger the process without any problems any number of times by sending a message.  If I stop and start the Camunda engine, the message send is received and the process is started, however it never completes.  I believe its 'stuck' just after the message start activity.  If I redeploy the same process, I can once again send messages to the process and it behaves as expected.

Any hints of tips on how to debug this ( Cockpit shows the process as running, with process variables, but no incidents, no exceptions in the logs ), or confirmation is this is the expected behaviour would be appreciated.

This is reproducible using a simple Message Start -> Sequence Flow -> End e.g.

<definitions xmlns="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:camunda="http://activiti.org/bpmn" id="process-definition" targetNamespace="http://camunda.org/examples">

 

  <process id="federos-7F0000013345A9FB0EA07467776D4740" isExecutable="true" name="SimpleMessageStart">

    <laneSet id="laneSet_ba49cb4c-bb3e-4f71-aac1-0e37d7f18a51">

      <lane id="lane_ab444428-deef-48f5-b154-b1a618f9e63e" name="admin">

        <flowNodeRef>GUID-7F00000121A768C30098D4407EAC41FD</flowNodeRef>

        <flowNodeRef>GUID-7F00000121A768C3EB022B17D3824786</flowNodeRef>

      </lane>

    </laneSet>

    <endEvent id="GUID-7F00000121A768C30098D4407EAC41FD" name="Silent end">

      <extensionElements>

        <camunda:executionListener event="end" expression="${execution.setVariable(&quot;FDProcessInstance_processFinalState&quot;,&quot;Completed&quot;)}"/>

      </extensionElements>

      <incoming>GUID-7F00000121A768C330019E5B03ED427E</incoming>

    </endEvent>

    <startEvent camunda:asyncBefore="true" id="GUID-7F00000121A768C3EB022B17D3824786" name="Manual start">

      <outgoing>GUID-7F00000121A768C330019E5B03ED427E</outgoing>

      <messageEventDefinition id="GUID-7F0000013345A9FBC1D8F5118F9D4ADD" messageRef="GUID-7F0000013345A9FB74085196DE5449E2"/>

    </startEvent>

    <sequenceFlow id="GUID-7F00000121A768C330019E5B03ED427E" sourceRef="GUID-7F00000121A768C3EB022B17D3824786" targetRef="GUID-7F00000121A768C30098D4407EAC41FD"/>

  </process>

  <collaboration id="collaboration_e76b595a-664f-47ee-bbf0-11c031a3ca4a">

    <participant id="participant_d8bcc5a2-9c56-4af0-bf54-2c56e9b99394" name="admin" processRef="federos-7F0000013345A9FB0EA07467776D4740"/>

  </collaboration>

  <message id="GUID-7F0000013345A9FB74085196DE5449E2" name="GUID-7F00000121A768C3EB022B17D3824786"/>

</definitions>

Kind Regards

Gareth

[thorben....@camunda.com]

Hi Gareth,

Could you please post your process engine configuration (e.g. bpm-platform.xml) and how you deploy the process (by a process application deployment, programmatically via Java API or REST API?).

My guess would be that since the process' start event is async, the job executor does not resume acquiring/executing jobs for a deployment that was made before the server restart (background reading: deployment-aware job executor [1]).

Cheers,
Thorben

[1] http://docs.camunda.org/latest/guides/user-guide/#process-engine-the-job-executor-job-execution-in-heterogeneous-clusters

[Gareth]

Thanks for the quick reply Thorben.

I've included the bpm-platform.xml configuration file inline below.  The process is deployed via the REST API and triggered by the REST API.  I've read the article you linked, but i'm not sure I understood all of it :(.  I have a single node ( for the mean time ), and nothing else, so I don't expect registerDeploymentForJobExecutor to make any difference ? But I'm happy to turn that off if that is whats causing the problem ( The Engine will only be used for a single application so any node will have the required libs installed in the future )

 Is there a setting I can change to "reregister" the deployments ? Or does it need to be done programatically ? ( i'm exclusively using the REST API at the moment and havent seen any sign of that being available ).

<?xml version="1.0" encoding="UTF-8"?>
<bpm-platform xmlns="http://www.camunda.org/schema/1.0/BpmPlatform" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.camunda.org/schema/1.0/BpmPlatform http://www.camunda.org/schema/1.0/BpmPlatform ">


  <job-executor>
    <job-acquisition name="default" />
  </job-executor>


  <process-engine name="default">
    <job-acquisition>default</job-acquisition>
    <configuration>org.camunda.bpm.engine.impl.cfg.StandaloneProcessEngineConfiguration</configuration>
    <datasource>java:jdbc/ProcessEngine</datasource>


    <properties>
      <property name="history">full</property>
      <property name="databaseSchemaUpdate">true</property>
      <property name="authorizationEnabled">true</property>
      <property name="jobExecutorDeploymentAware">true</property>
    </properties>


    <plugins>
      <!-- plugin enabling Process Application event listener support -->
      <plugin>
        <class>org.camunda.bpm.application.impl.event.ProcessApplicationEventListenerPlugin</class>
      </plugin>
     
      <!-- plugin enabling integration of camunda Spin -->
      <plugin>
        <class>org.camunda.spin.plugin.impl.SpinProcessEnginePlugin</class>
      </plugin>


      <!-- plugin enabling connect support -->
      <plugin>
        <class>org.camunda.connect.plugin.impl.ConnectProcessEnginePlugin</class>
      </plugin>
    </plugins>
  </process-engine>
</bpm-platform>

Thanks again,

Gareth

[thorben....@camunda.com]

Hi Gareth,

The process engine has the setting "jobExecutorDeploymentAware" and manages a set of deployments accessible from the management service (see methods ManagementService#registerDeploymentForJobExecutor and ManagementService#unregisterDeploymentForJobExecutor). The basic idea behind this concept is that in a clustered environment, not every cluster node may be aware of every process application. However, process applications may need to be present to ensure proper job execution since they may contain resources required for execution. For example, a process application can contain a class that implements a service task. If that class is not accessible on a cluster node, that node should not attempt to execute that service task (i.e. the job that leads to that service task).
The setting "jobExecutorDeploymentAware" in combination with the set of deployments is a way to prevent such behavior. Setting "jobExecutorDeploymentAware" to true means that the job executor only acquires and executes jobs that belong to deployments registered with the management service. This happens automatically when deploying a process application (and apparently also when deploying via REST). For process applications, the deployments are automatically re-registered when the application server is restarted, because the process application being present indicates that the other classes are also present. This cannot be automatically done with deployments made via REST.

You now have two options:
1) Re-registering your previous deployments with every restart with the management service. This can for example be done in a process engine plugin.
2) Disabling deployment aware job execution by setting "jobExecutorDeploymentAware" to false.

If you don't expect to run in a cluster or don't expect a resource loading issues as outlined above as you have all the required classes on all cluster nodes, 2) is the simpler way to go.

I hope that helps.

Cheers,
Thorben

[Gareth]

Thanks for the explanation.  All of the nodes will have the required service implementations so i'm happy ( and would like ) any node to pick up / re-register deployed process so I will update the config as advised.

Thanks again ( it was rather baffling )

Gareth