stack overflow if there is a loop in a process

[galen...@gmail.com]

Hi,

I've tested several BPMN diagrams that have a loop, and they result in java.lang.StackOverflowErrors.

For example, this

http://camunda.org/share/#/process/e0611a8c-9177-4075-9c30-9f5d9006efc7

causes the below stack trace. If left to run, the loop should exit after 500 times, but it never gets that far due to the stack overflow.. I have an execution listener, and I think this may have something to do with that..

Any ideas?

Thanks,
Galen

====================

java.lang.StackOverflowError
at java.security.AccessController.doPrivileged(Native Method)
at sun.org.mozilla.javascript.internal.ScriptRuntime$DefaultMessageProvider.getMessage(ScriptRuntime.java:3719)
at sun.org.mozilla.javascript.internal.ScriptRuntime.getMessage(ScriptRuntime.java:3698)
at sun.org.mozilla.javascript.internal.ScriptRuntime.getMessage0(ScriptRuntime.java:3646)
at sun.org.mozilla.javascript.internal.Parser.lookupMessage(Parser.java:240)
at sun.org.mozilla.javascript.internal.Parser.lookupMessage(Parser.java:236)
at sun.org.mozilla.javascript.internal.Parser.parse(Parser.java:586)
at sun.org.mozilla.javascript.internal.Parser.parse(Parser.java:507)
at sun.org.mozilla.javascript.internal.Context.compileImpl(Context.java:2436)
at sun.org.mozilla.javascript.internal.Context.compileString(Context.java:1394)
at sun.org.mozilla.javascript.internal.Context.compileString(Context.java:1383)
at sun.org.mozilla.javascript.internal.Context.evaluateString(Context.java:1135)
at com.sun.script.javascript.RhinoScriptEngine.getRuntimeScope(RhinoScriptEngine.java:364)
at com.sun.script.javascript.RhinoScriptEngine.eval(RhinoScriptEngine.java:210)
at com.sun.script.javascript.RhinoScriptEngine.eval(RhinoScriptEngine.java:240)
at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
at org.camunda.bpm.engine.impl.scripting.ScriptingEngines.evaluate(ScriptingEngines.java:69)
at org.camunda.bpm.engine.impl.bpmn.behavior.ScriptTaskActivityBehavior.execute(ScriptTaskActivityBehavior.java:51)
at org.camunda.bpm.engine.impl.pvm.runtime.AtomicOperationActivityExecute.execute(AtomicOperationActivityExecute.java:44)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:719)
at org.camunda.bpm.engine.impl.pvm.runtime.AtomicOperationTransitionNotifyListenerStart.eventNotificationsCompleted(AtomicOperationTransitionNotifyListenerStart.java:63)
at org.camunda.bpm.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:63)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:56)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:56)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:719)
at org.camunda.bpm.engine.impl.pvm.runtime.AtomicOperationTransitionCreateScope.execute(AtomicOperationTransitionCreateScope.java:54)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:719)
at org.camunda.bpm.engine.impl.pvm.runtime.AtomicOperationTransitionNotifyListenerTake.execute(AtomicOperationTransitionNotifyListenerTake.java:73)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:719)
at org.camunda.bpm.engine.impl.pvm.runtime.AtomicOperationTransitionNotifyListenerTake.execute(AtomicOperationTransitionNotifyListenerTake.java:58)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:719)
at org.camunda.bpm.engine.impl.pvm.runtime.AtomicOperationTransitionDestroyScope.execute(AtomicOperationTransitionDestroyScope.java:117)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:719)
at org.camunda.bpm.engine.impl.pvm.runtime.AtomicOperationTransitionNotifyListenerEnd.eventNotificationsCompleted(AtomicOperationTransitionNotifyListenerEnd.java:36)
at org.camunda.bpm.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:63)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:56)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:56)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:719)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.take(ExecutionEntity.java:525)
at org.camunda.bpm.engine.impl.bpmn.behavior.ExclusiveGatewayActivityBehavior.leave(ExclusiveGatewayActivityBehavior.java:80)
at org.camunda.bpm.engine.impl.bpmn.behavior.FlowNodeActivityBehavior.execute(FlowNodeActivityBehavior.java:36)
at org.camunda.bpm.engine.impl.pvm.runtime.AtomicOperationActivityExecute.execute(AtomicOperationActivityExecute.java:44)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:719)
at org.camunda.bpm.engine.impl.pvm.runtime.AtomicOperationTransitionNotifyListenerStart.eventNotificationsCompleted(AtomicOperationTransitionNotifyListenerStart.java:63)
at org.camunda.bpm.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:63)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:56)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:56)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:719)
at org.camunda.bpm.engine.impl.pvm.runtime.AtomicOperationTransitionCreateScope.execute(AtomicOperationTransitionCreateScope.java:54)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:719)
at org.camunda.bpm.engine.impl.pvm.runtime.AtomicOperationTransitionNotifyListenerTake.execute(AtomicOperationTransitionNotifyListenerTake.java:73)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:719)
at org.camunda.bpm.engine.impl.pvm.runtime.AtomicOperationTransitionNotifyListenerTake.execute(AtomicOperationTransitionNotifyListenerTake.java:58)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:719)
at org.camunda.bpm.engine.impl.pvm.runtime.AtomicOperationTransitionDestroyScope.execute(AtomicOperationTransitionDestroyScope.java:117)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:719)
at org.camunda.bpm.engine.impl.pvm.runtime.AtomicOperationTransitionNotifyListenerEnd.eventNotificationsCompleted(AtomicOperationTransitionNotifyListenerEnd.java:36)
at org.camunda.bpm.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:63)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:56)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:56)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:93)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:728)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:719)
at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.take(ExecutionEntity.java:525)
at org.camunda.bpm.engine.impl.bpmn.behavior.BpmnActivityBehavior.performOutgoingBehavior(BpmnActivityBehavior.java:102)
at org.camunda.bpm.engine.impl.bpmn.behavior.BpmnActivityBehavior.performDefaultOutgoingBehavior(BpmnActivityBehavior.java:51)

...
KEEPS GOING A LONG TIME -- TRUNCATED
KEEPS GOING A LONG TIME -- TRUNCATED
...
...

[Daniel Meyer]

Hi Galen,

this is a long standing, known limitation of the process engine. For quite some time I had lost hope that we would ever solve it. 

With some persistence refactorings we made 2 month back I regained hope that we can solve it again. Actually I gave it a quick try a couple weeks back and I could almost solve it.

It would be interesting to know what kind of process you are attempting to implement. Why do you need this kind of loop?

Regards,

Daniel

[thorben....@camunda.com]

Hi Galen,

The engine's internal logic of executing a process fragment between two transaction boundaries is based on recursion. That is why your stack trace has many fragments that occur repeatedly. So for 500 iterations, the stack grows to roughly the size of 500*n, where n is some kind constant.
There are two basic solutions two avoid stack overflows:
1. Increase the JVM's stack size
2. Introduce transaction boundaries by making the script task in your process asynchronous. Then, the stack "collapses" every time the asynchronous task is reached and a job is persisted to the database.

I hope this explains why you see the stack overflow and it's not your execution listener that causes problem (unless the listener itself instantiates another process that adds even more frames onto the stack).

As for refactoring the recursion, see Daniel's answer :)

Cheers,
Thorben

[galen...@gmail.com]

Thanks for the responses,

My example that I posted above was just an example that had no content inside the loop, but more typical use cases for loops might be:

1) Some sort of simulation (e.g. scientific, financial, etc..) that coordinates the interactions of different components for N iterations.
N can be quite high, and performance matters (i.e. total time to complete N iterations).

2) A long running process that does something repeatedly (e.g. run some logic once a day).

++++++++++++++++

For #2, I'm not so concerned, since a whole other process can just be kicked every day, or asynchronous workaround (mentioned by Thorben) can be used.

However for #1, time is a consideration, and either kicking off another process and/or introducing asynchronous flags would:
-- slow it down considerably
-- cause a lot of DB traffic/contention (imagine it N = 1 million)

Perhaps Camunda is not the best fit tool for the "very fast loop with a lot of iterations" use case. We are just exploring the possibility of using it for that case, among others..

+++++++++++++++++++++++++++++

One alternate workaround I just thought of would be to have an asynchronous task run every Nth iteration. For example every tenth iteration of the loop, the stack could be "collapsed". This would probably be more efficient than doing it every iteration.

Thanks,
Galen

[webcyberrob]

Hi Galen,

Have you considered changing the default stacksize (-Xss) in the JVM? Does not eliminate the problem, but it may give you a useable workaround. Last time I looked, the default stack size was in the KB rather than MB range...

regards

Rob

[galen...@gmail.com]

Hi,

I settled on the approach of "collapsing" the stack every Nth iteration. I also increased the stacksize. Both of these options seem to help with performance. While increasing the stacksize allows for less frequent collapsing, the collapsing is still necessary.

The only downside to these approaches is that:

1) The process developer needs to be aware of this limitation
2) Extra flow/elements need to be introduced into the BPMN diagram, complicating things

Hopefully you guys can come up with a good approach in the future that can scale regardless of how many iterations are in a (synchronous) loop.

Thanks for the help,
Galen

[pek...@gmail.com]

Hi. Could you please explain in more details the approach of collapsing the stack?

[galen...@gmail.com]

Basically I have a loop in my process that only executes synchronous tasks. However, at each iteration of the loop, it increments a counter, and if the counter mod N == 0, then I redirect the flow to an asynchronous task, before continuing the loop.
The async task causes the stack to collapse, as it flushes state to the database.

Thanks,
Galen