Rest API Enhancement Proposal - Single query call returns detailed process instance data

[Clint Manning]

Hi,

We have a requirement for the Rest API to query process instances (Get /process-instance) and return the following information for each instance:

    id
    definitionId
    businessKey
    suspended
    start time
    end time
    duration
    current activity

 This would mean the above Rest API call would return information on active, suspended, ended and deleted processes. I'm curious as to what Camunda's opinion is on this matter? If it is not advisable to change the existing call in Rest, then perhaps an alternative Rest implementation is feasible? One idea we had is to add an additional query to Rest, example url:
 
 @Get
 /process-instance/extended

 Both implementations would warrant another query implementation within the engine, because changing either ProcessInstanceQuery or HistoricProcessInstanceQuery seems too invasive. In the backend, it would involve a SQL select joining the  act_hi_procinst and act_ru_execution tables. Overall, we would require a new interface and implementation that extends both the ProcessInstance and HistoricProcessInstance interfaces, a new query interface and implementation, a new persistence entity, an additional find method added to the HistoricProcessInstanceManager

, an entry in HistoricProcessInstance mapping file and changes to Rest.
 
 At the moment, however, we can obtain all of the above information by using the ProcessInstanceQuery and HistoricProcessInstanceQuery implementations. Unfortunately, this requires two query calls and some work to combine the two lists that are returned. It would be preferable if we can obtain this information using only one query implementation. Would Camunda be interested in accepting either of these changes? I’m also open to other suggestions.
 
Overall proposal

New process interface and implementation (extends ProcessInstance and HistoricProcessInstance):
org.camunda.bpm.engine.history.ExtendedProcessInstance
org.camunda.bpm.engine.impl.ExtendedProcessInstanceImpl

New query and interface:
org.camunda.bpm.engine.history.ExtendedProcessInstanceQuery
org.camunda.bpm.engine.impl.ExtendedProcessInstanceQueryImpl

New persistence entity:
org.camunda.bpm.engine.impl.persistence.entity.ExtendedProcessInstanceEntity
org.camunda.bpm.engine.impl.persistence.entity.ExtendedScopeInstanceEntity

Either add a new query to Rest or extend the existing one. Adding an additional query would require changes to:
org.camunda.bpm.engine.rest.ProcessInstanceRestService
org.camunda.bpm.engine.rest.impl.ProcessInstanceRestService

In the ProcessInstanceRestService interface
  @GET
  @Path("/extended")
  @Produces(MediaType.APPLICATION_JSON)
  List<ExtendedProcessInstanceDto> getExtendedProcessInstances(@Context UriInfo uriInfo,
      @QueryParam("firstResult") Integer firstResult,
      @QueryParam("maxResults") Integer maxResults);

  @POST
  @Path("/extended")
  @Consumes(MediaType.APPLICATION_JSON)
  @Produces(MediaType.APPLICATION_JSON)
  List<ExtendedProcessInstanceDto> queryExtendedProcessInstances(ProcessInstanceQueryDto query,
      @QueryParam("firstResult") Integer firstResult,
      @QueryParam("maxResults") Integer maxResults);


Additional method added to:
Class: org.camunda.bpm.engine.impl.persistence.entity.HistoricProcessInstanceManager
Method: findExtendedProcessInstancesByQueryCriteria
Method Returns: List<ExtendedProcessInstance>

Additional sql query and result map added to the HistoricProcesssInstance mapping file:
src/main/resources/org/camunda/bpm/engine/impl/mapping/entity/HistoricProcessInstance.xml

  <select id="selectExtendedProcessInstancesByQueryCriteria" parameterType="org.camunda.bpm.engine.impl.ExtendedProcessInstanceQueryImpl" resultMap="extendedProcessInstanceResultMap">
          ${limitBefore}
    select RES.* ${limitBetween}
    <include refid="selectExtendedProcessInstancesByQueryCriteriaSql"/>
    ${orderBy}
    ${limitAfter}
  </select>

<sql id="selectExtendedProcessInstancesByQueryCriteriaSql">
    from ${prefix}ACT_HI_PROCINST RES
    inner join ${prefix}ACT_RU_EXECUTION RUE on RES.PROC_INST_ID_ = RUE.ID_
    <where>
        ...
        ...
  </where>
</sql>

  <resultMap id="extendedProcessInstanceResultMap" type="org.camunda.bpm.engine.impl.persistence.entity.ExtendedProcessInstanceEntity">
    <id property="id" column="ID_" jdbcType="VARCHAR" />
    <result property="processInstanceId" column="PROC_INST_ID_" jdbcType="VARCHAR" />
    ...
    ...        
  </resultMap>
        

[Daniel Meyer]

Hi Clint,

First of all: thank you for wanting to contribute this feature and it is also good to discuss the contribution upfront!

We briefly discussed this internally: Generally we think that it is problematic to provide a resource which joins runtime and history databases. Let me briefly explain why:

- History may be written to a different database than runtime data. We will offer the feature to write history to a different datasource. In that case it will not be possible to join runtime and history data.

- History may be written asynchronously. We will also provide the feature to make writing of History asynchronous which means that if a process instance is present in the runtime data does not guarantee that it is also present in History. 

- And there is a general, I would call it "reluctance" to offer a Query in the public API which ties history and runtime this closely together. 

But fortunately I think that for your requirements this is also not needed. 

First: consider that history contains all process instances (unfinished + finished).

Second: You write that you want to retrieve the following data in a single query / request:

    id   => Historic Process Instance
    definitionId  => Historic Process Instance
    businessKey => Historic Process Instance
    suspended => Runtime only (currently)
    start time => Historic Process Instance
    end time => Historic Process Instance
    duration => Historic Process Instance
    current activity => Historic Activity Instance (note that a single instance can have mutiple active activity instances)

So almost all data is present in History (except for the suspended state). 

If you agree on this, then I would propose the following: 

- lets continue this in a direction where we build a set of  History-Only Resources. 

- If the "suspended" flag is really important to you then lets think about how we would model that in History. 

WDYT?

Daniel Meyer

[Clint Manning]

"- lets continue this in a direction where we build a set of  History-Only Resources. " - That's fine. I have something implemented already that I would like to offer. I have exposed the HistoricProcessInstance, HistoricActivityInstance and HistoricVariableInstance querys in the Rest API. The querys are exposed in the same manner as the other querys, ProcessInstanceQuery etc...

At the moment, the urls are in the form of:

@GET
/historic-process-instance/

@GET
/historic-activity-instance/

@GET
/historic-variable-instance/


But someone here raised the option of having the design as

@GET
/process-instance/historic

@GET
/activity-instance/historic

@GET
/variable-instance/historic

WDYT?


"- If the "suspended" flag is really important to you then lets think about how we would model that in History." - Our client would very much like the suspended flag returned with the same data. If possible, modeling it in history would be an option for us.

[Clint Manning]

If Camunda is interested I can do a pull request next week

[Clint Manning]

11:28 AM (less than a minute ago)

Some clarification on the alternative design:

@GET
/process-instance/historic

@GET
/activity-instance/historic

@GET
/variable-instance/historic

This would mean the calls would be moved into the ProcessInstanceRestService,VariableInstanceRestService etc...

[Daniel Meyer]

Hi Clint,

 

Great!

 

what about the following?

 

/histor(y/ic?)/process-instance

/histor(y/ic?)//activity-instance

/histor(y/ic?)//variable-instance

 

This way we won’t have to think about namespace clashes. This could be ambiguous if your process instance id is “historic”.

@GET
/process-instance/historic

Unlikely, but still J

 

And we could have something like a “history root resource” which provides all the history resources and which would allow you to add / remove the history resources to your JAX-RS deployment “en bloc”. Ie. If you do not have history or do not want to expose it, it would be very simple to not deploy it.

 

@Path(HistoryRestService.PATH)

public interface HistoryRestService {

 

  public static final String PATH = "/history(y/ic)?";

 

  @Path(HistoricProcessInstanceRestService.PATH)

  HistoricProcessInstanceRestService getProcessInstanceService();

 

  @Path(HistoricActivityInstanceRestService.PATH)

  HistoricActivityInstanceRestService getActivityInstanceService();

 

...

 

}

 

WDYT?

Cheers,

Daniel Meyer

--
You received this message because you are subscribed to the Google Groups "camunda BPM platform internal" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camunda-bpm-d...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Clint Manning]

Hi,


  I'm happy with what you put forward. I think it should be History and not Historic so:

/history/process-instance

/history/activity-instance

/history/variable-instance

I  will make the adjustments and do a pull request next week.

[Daniel Meyer]

Awesome!

Cheers,

Daniel Meyer

[Clint Manning]

Hi,


   Another question regarding modeling the suspended value in a history table. Would this be added to the act_hi_procinst table? Example (is_active_ boolean)


CREATE TABLE act_hi_procinst
(
  id_ character varying(64) NOT NULL,
  proc_inst_id_ character varying(64) NOT NULL,
  business_key_ character varying(255),
  proc_def_id_ character varying(64) NOT NULL,
  start_time_ timestamp without time zone NOT NULL,
  end_time_ timestamp without time zone,
  duration_ bigint,
  start_user_id_ character varying(255),
  start_act_id_ character varying(255),
  end_act_id_ character varying(255),
  super_process_instance_id_ character varying(64),
  delete_reason_ character varying(4000),
  is_active_ boolean,
  CONSTRAINT act_hi_procinst_pkey PRIMARY KEY (id_),
  CONSTRAINT act_hi_procinst_proc_def_id__business_key__key UNIQUE (proc_def_id_, business_key_),
  CONSTRAINT act_hi_procinst_proc_inst_id__key UNIQUE (proc_inst_id_)
)


That way it could be returned by a HistoricProcesssInstanceQuery?

[Daniel Meyer]

Hi Clint,

 

that could certainly be an option.

 

We are currently also thinking about adding some sort of user interaction audit log, ie. something like

“At 2013-07-25T14:18:20, Jonny has suspended the process instance 4aaebf59-f524-11e2-9cf2-3c970e140ef1”.           

Since from a history perspective, the fact that a process instance is suspended or active is probably most interesting in respect to who has suspended / activated it?, when did it happen, etc …?

 

It could be modeled somewhat like this:

ID_	USER_ID_	OPERATION	RESOURCE_TYPE_	RESOURCE_ID_	TIMESTAMP_
10001	jonny	suspend	process-instance	2320001	2013-07-25T14:18:20
10001	jonny	activate	process-instance	2320001	2013-08-25T14:18:20

 

 

The downside from your usecase perspective would probably be that it would not be “queriable” as easily / efficiently.

 

Cheers,

Daniel Meyer

[Clint Manning]

Hi,

   I don't have an issue with an audit log. It looks like a sound idea.

  Going back to the original proposal, it comes down to whether the rest client gets the information relevant for them in one rest call or multiple calls. If its one call, then the result sets from these querys would need to be combined in some way and then returned to the client. I'm not sure if Camunda would want to move in this direction where a Rest call does this?


Regards, Clint

[Daniel Meyer]

Hi Clint,

 

I think that adding the audit table would be nice but we do not have to do that right away. What we could do is add a logical STATE_ column to the HistoricProcessInstance and provide a proper lifecycle for process instances:

 

(maybe the names are not perfect COMPLETED vs. FINISHED seems to be a little ambiguous)

 

 

And then we could include all these states into the query:

 

createHistoicProcessInstanceQuery().running() // = active + suspended

createHistoicProcessInstanceQuery().suspended()

createHistoicProcessInstanceQuery().active()

 

createHistoicProcessInstanceQuery().finished() // = completed + cancelled

createHistoicProcessInstanceQuery().completed()

createHistoicProcessInstanceQuery().cancelled()

 

And the Historic process instance could return the value:

HistoricProcessInstanceLifecycleState state = historicProcessInstance.getLifecycleState();

HistoricProcessInstanceLifecycleState could be an interface with two properties getState() and getSubstate() and there is an enum which implements the interface and provides the built-in constants.

 

WDYT?

 

Cheers,

Daniel Meyer

[Bernd Rücker (camunda)]

Quick feedback: Like the idea – but wonder what the best names are indeed.

 

Completed vs. finished and running vs. active is a bit confusing for me.

--

[Clint Manning]

 Hi,
    Great!

    As for the naming issue:


    Running vs Active - Initially a little confusing but ok if you think about it :)
    Finished vs Completed - Same here. Completed is definitely correct. Finished can mean ended or completed and canceled can mean ended, so the other candidate that I can see is "Ended"

   I, however, can live with the naming as it is :)

Regards, Clint

[Thorben Lindhauer]

+1

I like "Ended" instead of "Finished", because all it expresses is that the instance is not running. "Completed" is fine, because it expresses success.

----- Ursprüngliche Mail -----
Von: "Clint Manning" <clint....@gmail.com>
An: camunda...@googlegroups.com
CC: "Clint Manning" <clint....@gmail.com>
Gesendet: Freitag, 26. Juli 2013 13:45:02
Betreff: Re: Rest API Enhancement Proposal - Single query call returns detailed process instance data

[Daniel Meyer]

Yes, sounds reasonable.

 

Update:

 

Cheers,

Daniel

 

-----Ursprüngliche Nachricht-----
Von: camunda...@googlegroups.com [mailto:camunda...@googlegroups.com] Im Auftrag von Thorben Lindhauer
Gesendet: Freitag, 26. Juli 2013 13:53
An: Clint Manning
Cc: camunda...@googlegroups.com

Betreff: Re: Rest API Enhancement Proposal - Single query call returns detailed process instance data