Python (non-JVM languages)

[Brady McCary]

camunda-bpm-users,

I have a collection of image processing tasks implemented in python.
At certain, detectable points human interaction may be necessary to
finish processing an image. These python/human tasks are logically
organized into a workflow. I am currently using Amazon SWF to
coordinate the execution of the python/human tasks. I would like to
move away from Amazon SWF to something open source such as Camunda.
After reading the Camunda docs and blog postings for about an hour, it
seems that using non-JVM languages is not supported at present. These
image processing tasks are implemented in CPython, and it is not
possible for me to use Jython instead.

Would someone please clarify this issue of non-JVM languages with Camunda?

Brady

[Daniel Meyer]

Hi Brady,

welcome to our community :)

Your usecase is something we are currently discussing intensly and it is very interesting to get feedback from non-JVM-developers like yourself.

Let me first try to answer your question and then take the opportunity for doing some shameless market research :)

Human Tasks aka User Tasks: The user task is performed "outside" of the process engine. You have two options: 

1. you can use the built-in task list in which case you can develop the task forms entirely in HTML + Javascript. 
2. you can build you own task list with custom forms. In that case you would probably choose Python and interact with the process engine using the REST Api.

External Non-JVM Service Tasks: You have multiple options, none of which would work completely "out of the box": 

1. Expose your python tasks as REST / Webservice and have the process engine invoke them,
2. Write a Java Delegate implementation which executes a shell command and in this way execute the python script,
3. Competing consumers pattern: Implement a POLL / LOCK* / SIGNAL Scheme similar to what you are probably familiar with in Amazon SWF. In your process model you use receive tasks instead of service tasks. This way the process engine does not try to "call you" bit will just suspend execution and wait for an external trigger. You could then poll the Execution REST service for process instances arriving at a certain activity (the activity your python script is in charge of executing.) You would then perform the python script and send a signal once you are done. The main problem here is that we do not have a locking mechanism. This means that if you have multiple consumers on a single activity, you will have contention.

So at the moment you could probably achieve what you are looking for but it still involves a fair amount of customization. 

I mentioned in the beginning that we are currently discussing making it easier for non JVM developers to use the process engine technology. I think that in order to make camunda attractive for non JVM developers, we would need to make sure that 

a) non JVM developers are not exposed to any Java developer tooling like Eclipse or Maven

b) non JVM developers have a language native SDK (similar to amazon) which provides a comfortable API for interacting with the process engine. We have experimented with this in PHP.

c) external service tasks can be implemented using the competing consumers pattern but the related boilerplate code is handled by the SDK.

What I would be interested in is the following: given that you can do all the heavy lifting (in your case the image processing) in native CPython, would you run simple python scripts which are maybe concerned with variable mapping or xml manipulation or incrementing loop counters :) on the JVM (in Jython)? 

It true, what kind of developer experience (Tooling, testing, ...) would you expect in that case?

Regards,

Daniel

[Hari]

On Sunday, May 18, 2014 at 1:27:20 AM UTC-7, Daniel Meyer wrote:
> Hi Brady,
>
>
> welcome to our community :)
>
>
> Your usecase is something we are currently discussing intensly and it is very interesting to get feedback from non-JVM-developers like yourself.
> Let me first try to answer your question and then take the opportunity for doing some shameless market research :)
>
>
>
> Human Tasks aka User Tasks: The user task is performed "outside" of the process engine. You have two options: 
> you can use the built-in task list in which case you can develop the task forms entirely in HTML + Javascript. you can build you own task list with custom forms. In that case you would probably choose Python and interact with the process engine using the REST Api.
>
>
> External Non-JVM Service Tasks: You have multiple options, none of which would work completely "out of the box": 

> Expose your python tasks as REST / Webservice and have the process engine invoke them,Write a Java Delegate implementation which executes a shell command and in this way execute the python script,Competing consumers pattern: Implement a POLL / LOCK* / SIGNAL Scheme similar to what you are probably familiar with in Amazon SWF. In your process model you use receive tasks instead of service tasks. This way the process engine does not try to "call you" bit will just suspend execution and wait for an external trigger. You could then poll the Execution REST service for process instances arriving at a certain activity (the activity your python script is in charge of executing.) You would then perform the python script and send a signal once you are done. The main problem here is that we do not have a locking mechanism. This means that if you have multiple consumers on a single activity, you will have contention.

> So at the moment you could probably achieve what you are looking for but it still involves a fair amount of customization. 
>
>
> I mentioned in the beginning that we are currently discussing making it easier for non JVM developers to use the process engine technology. I think that in order to make camunda attractive for non JVM developers, we would need to make sure that 
> a) non JVM developers are not exposed to any Java developer tooling like Eclipse or Maven
> b) non JVM developers have a language native SDK (similar to amazon) which provides a comfortable API for interacting with the process engine. We have experimented with this in PHP.
> c) external service tasks can be implemented using the competing consumers pattern but the related boilerplate code is handled by the SDK.
>
>
> What I would be interested in is the following: given that you can do all the heavy lifting (in your case the image processing) in native CPython, would you run simple python scripts which are maybe concerned with variable mapping or xml manipulation or incrementing loop counters :) on the JVM (in Jython)? 
> It true, what kind of developer experience (Tooling, testing, ...) would you expect in that case?
>
>
> Regards,
> Daniel

Hello,

Interesting to find the use-case! My application is a bunch of image-processing python scripts with a possible Human Task, and am trying to use Camunda to log process' states into an AWS RDS postgre instance. Is there a way for the python scripts to interact with the RDS instances.