Evaluating Copper workflow engine

242 views
Skip to first unread message

Kay

unread,
Sep 18, 2017, 2:36:57 AM9/18/17
to COPPER Engine
Hi,

We are very interested in your Copper workflow engine and would like to use it. I have played around the copper-starter sample which is very impressive. But going beyond the sample is something challenging due to lack of documentation.

It will be great if you can help me out on the 2 questions below.

1) How can I add Audit Trail to a workflow, say ResetMailbox?

I added the following code in ResetMailbox.java

 

    private transient AuditTrail auditTrail;

 

…

 

    @AutoWire

    public void setAuditTrail(AuditTrail auditTrail) {

        this.auditTrail = auditTrail;

    }

 

…

    public void main() throws Interrupt {

 

        BatchingAuditTrail batchingAuditTrail = new BatchingAuditTrail();

       

        try {

            batchingAuditTrail.startup();

        } catch (Exception e) {

            logger.error("batchingAuditTrail request failed", e);

        }

 

    private boolean resetMailbox() throws Interrupt {

       

        for (int i = 0; ; i++) {

            final String correlationId = networkServiceAdapter.resetMailbox(getData().getMsisdn());

 

            auditTrail.asynchLog(0, new Date(), "resetMailbox", "-", this.getId(), correlationId, null, "finished", "TEXT");

 

The batchingAuditTrail.startup() failed with the following error:

 

2017.09.18 12:02:46,462 ERROR [P#DEFAULT#0] org.copperengine.examples.orchestration.wf.ResetMailbox [790b5c63-9dbb-4850-8e60-9047c93a5801] - batchingAuditTrail request failed

java.lang.NullPointerException

            at org.copperengine.core.audit.BatchingAuditTrail.startup(BatchingAuditTrail.java:152)

            at org.copperengine.examples.orchestration.wf.ResetMailbox.main(ResetMailbox.java:76)

            at org.copperengine.core.persistent.PersistentProcessor$1.run(PersistentProcessor.java:57)

            at org.copperengine.core.persistent.PersistentProcessor$1.run(PersistentProcessor.java:49)

            at org.copperengine.core.persistent.txn.CopperTransactionController.run(CopperTransactionController.java:63)

            at org.copperengine.core.persistent.PersistentProcessor.process(PersistentProcessor.java:49)

            at org.copperengine.core.common.Processor.run(Processor.java:79)

 

Is this because that the dataSource is not set to the batchingAuditTrail? How can I set the right dataSource so that the audit data can go to the derby DB - ./build/copperExampleDB?

 

If I remove the batchingAuditTrail related lines of code and keep only the auditTrail.asynchLog call, it gives me a different error on the auditTrail.asynchLog line of code.

 

2017.09.18 12:21:55,255 ERROR [P#DEFAULT#0] org.copperengine.core.common.Processor [fd63da71-76d7-45cf-b5e7-1467d4595786] - execution of workflow instance failed

java.lang.NullPointerException

            at org.copperengine.core.audit.BatchInsertIntoAutoTrail$Command.<init>(BatchInsertIntoAutoTrail.java:50)

            at org.copperengine.core.audit.BatchingAuditTrail.createBatchCommand(BatchingAuditTrail.java:265)

            at org.copperengine.core.audit.BatchingAuditTrail.doLog(BatchingAuditTrail.java:257)

            at org.copperengine.core.audit.BatchingAuditTrail.doLog(BatchingAuditTrail.java:250)

            at org.copperengine.core.audit.BatchingAuditTrail.asynchLog(BatchingAuditTrail.java:235)

            at org.copperengine.core.audit.BatchingAuditTrail.asynchLog(BatchingAuditTrail.java:225)

            at org.copperengine.examples.orchestration.wf.ResetMailbox.resetMailbox(ResetMailbox.java:121)

            at org.copperengine.examples.orchestration.wf.ResetMailbox.main(ResetMailbox.java:86)

            at org.copperengine.core.persistent.PersistentProcessor$1.run(PersistentProcessor.java:57)

            at org.copperengine.core.persistent.PersistentProcessor$1.run(PersistentProcessor.java:49)

            at org.copperengine.core.persistent.txn.CopperTransactionController.run(CopperTransactionController.java:63)

            at org.copperengine.core.persistent.PersistentProcessor.process(PersistentProcessor.java:49)

            at org.copperengine.core.common.Processor.run(Processor.java:79)

2017.09.18 12:21:55,255 ERROR [P#DEFAULT#0] org.copperengine.core.common.Processor [fd63da71-76d7-45cf-b5e7-1467d4595786] - Storing error information for workflow instance...


2) How can I add a new workflow (say HelloWorldWorkFlowV2) into the Orchestration Engine instead of resetMailBox?


The OrchestrationServiceTestClient.java sample has the following code:


    private static final QName SERVICE_NAME = new QName("http://orchestration.copperengine.org/", "OrchestrationService");


    public static void main(String args[]) throws java.lang.Exception {

        URL wsdlURL = new URL(args[0]);


        OrchestrationService_Service ss = new OrchestrationService_Service(wsdlURL, SERVICE_NAME);

        OrchestrationService port = ss.getOrchestrationServicePort();

        ....

        port.resetMailbox(resetMailbox_msisdn, resetMailbox_secret);

    }


So resetMailbox is recognized by OrchestrationService because resetMailBox is a 'method' of OrchestrationService defined in OrchestrationService.java. But OrchestrationService.java  is generated. There must be somewhere else having this resetMailBox and OrchestrationService linkage defined. Not quite sure where it is.

Thanks for help.

Regards,

Kay

Michael Austermann

unread,
Sep 18, 2017, 4:15:57 AM9/18/17
to copper...@googlegroups.com
Hi!

1)
Don't create BatchingAuditTrail in den main method of the workflow.
By creating the setAuditTrail method together with the @AutoWire Annotation, COPPER will inject it into the workflow instance.
You may want to check the "dependency injection" design pattern to better understand what is happending there - e.g. see here: https://en.wikipedia.org/wiki/Dependency_injection

So in short: Just remove all new code that you mentioned in your eMail from the main method  - it should work then

2)
See org.copperengine.examples.orchestration.adapter.OrchestrationInputAdapter

/Michael


Von: "Kay" <mao...@gmail.com>
An: "COPPER Engine" <copper...@googlegroups.com>
Gesendet: Montag, 18. September 2017 08:36:57
Betreff: [COPPER Users] Evaluating Copper workflow engine

--
You received this message because you are subscribed to the Google Groups "COPPER Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to copper-engin...@googlegroups.com.
To post to this group, send email to copper...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/copper-engine/ae0add11-6de8-4972-b8a0-b9baaa9828a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Dirk Möbius

unread,
Sep 18, 2017, 4:18:17 AM9/18/17
to copper-engine
Hello Kay,

thanks for your interest in COPPER and your kind feedback.

Yes, you're right, you must set the DataSource in BatchingAuditTrail. You can do it in your main method like this:

BatchingAuditTrail batchingAuditTrail = new BatchingAuditTrail();
DataSource dataSource = ... // get your datasource from somewhere, e.g. use C3P0PooledDataSource
batchingAuditTrail.setDataSource(dataSource);

Hope this helps,
Dirk
--
SCOOP Software GmbH - Gut Maarhausen - Eiler Straße 3 P - D-51107 Köln
Dirk Möbius

T +49 221 801916-167 - F +49 221 801916-17 - M +49 175 5930071
dirk.m...@scoop-software.de - www.scoop-software.de
Sitz der Gesellschaft: Köln, Handelsregister: Köln,
Handelsregisternummer: HRB 36625
Geschäftsführung: Dr. Oleg Balovnev, Frank Heinen,
Dr. Wolfgang Reddig, Roland Scheel

Kay

unread,
Sep 18, 2017, 9:54:56 PM9/18/17
to COPPER Engine
Thanks Michael and Dirk for your kind replies.

1) I did try removing the new code in the main method. But the following line failed with NullPointerException.

            auditTrail.synchLog(new AuditTrailEvent(1, new Date(), cid, "beforeFoo", getId(), cid, cid, "beforeFoo", "String", null));

2017.09.19 10:58:35,173 ERROR [P#DEFAULT#1] org.copperengine.core.common.Processor [ae6428b2-4a84-497a-a08a-d6d6fc459b27] - execution of workflow instance failed
java.lang.NullPointerException
at org.copperengine.core.audit.BatchInsertIntoAutoTrail$Command.<init>(BatchInsertIntoAutoTrail.java:50)
at org.copperengine.core.audit.BatchingAuditTrail.createBatchCommand(BatchingAuditTrail.java:265)
at org.copperengine.core.audit.BatchingAuditTrail.doLog(BatchingAuditTrail.java:257)
at org.copperengine.core.audit.BatchingAuditTrail.doLog(BatchingAuditTrail.java:250)
at org.copperengine.core.audit.BatchingAuditTrail.synchLog(BatchingAuditTrail.java:287)
at org.copperengine.examples.orchestration.wf.ResetMailbox.resetMailbox(ResetMailbox.java:124)
at org.copperengine.examples.orchestration.wf.ResetMailbox.main(ResetMailbox.java:87)
at org.copperengine.core.persistent.PersistentProcessor$1.run(PersistentProcessor.java:57)
at org.copperengine.core.persistent.PersistentProcessor$1.run(PersistentProcessor.java:49)
at org.copperengine.core.persistent.txn.CopperTransactionController.run(CopperTransactionController.java:63)
at org.copperengine.core.persistent.PersistentProcessor.process(PersistentProcessor.java:49)
at org.copperengine.core.common.Processor.run(Processor.java:79)

The database is already connected by the Engine to store the workflow instance data into the derby DB - ./build/copperExampleDB defined in the copperContext.xml file. The AuditTrail somehow needs to know storing the audit data into this derby DB. As Dirk pointed out that I need to get the DataSource, I need to know the details how the to use the existing copperExampleDB as the dataSource in the following line.

DataSource dataSource = ... // get your datasource from somewhere, e.g. use C3P0PooledDataSource

2) I added HelloWorldWorkflow into OrchestrationInputAdapter. But it seems that I also need to add HelloWorldWorkflow in the OrchestrationEngine.wsdl file because I got the following error:

2017.09.19 11:45:25,063 INFO  [main] org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean [] - Creating Service {http://orchestration.copperengine.org/}OrchestrationService from WSDL: http://localhost:9090/services/orchestration?wsdl
Exception in thread "main" javax.xml.ws.WebServiceException: Could not find wsdl:binding operation info for web method HelloWorldWorkFlow.
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:127)
at com.sun.proxy.$Proxy30.HelloWorldWorkFlow(Unknown Source)
at org.copperengine.examples.orchestration.simulators.clients.OrchestrationServiceTestCGI.main(OrchestrationServiceTestCGI.java:45)

Does this mean that any workflow has to implement Web Service calls in order to work with OrchestrationEngine?

I apologize if my questions don't make sense. I am learning :-)

Regards,

Kay

Michael Austermann

unread,
Sep 19, 2017, 12:59:39 AM9/19/17
to copper...@googlegroups.com
Hi Kay,

1) is a bug in the copper-starter spring context file.

I also added an example call to auditTrail in the main method.
Just "git pull" the current master of the copper starter project.

2)
Please provide the source code of what you added.

/Michael


Von: "Kay" <mao...@gmail.com>
An: "COPPER Engine" <copper...@googlegroups.com>
Gesendet: Dienstag, 19. September 2017 03:54:56
Betreff: [COPPER Users] Re: Evaluating Copper workflow engine

--
You received this message because you are subscribed to the Google Groups "COPPER Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to copper-engin...@googlegroups.com.
To post to this group, send email to copper...@googlegroups.com.

Kay

unread,
Sep 19, 2017, 9:54:36 PM9/19/17
to COPPER Engine
Thanks Michael for the quick response and fix :-)

After I changed the copperContext.xml file, AuditTrail does work. Great!

        auditTrail.synchLog(0, new Date(), "conversationId", "context", this.getId(), null, null, "workflow instance started", "String");

But I notice that the LONG_MESSAGE is not what I write in the code "workflow instance started". 

SEQ_ID OCCURRENCE CONVERSATION_ID LOGLEVEL CONTEXT INSTANCE_ID CORRELATION_ID TRANSACTION_ID LONG_MESSAGE MESSAGE_TYPE
301 2017-09-20 11:33:22 conversationId 0 context bc7efc9f-5c3d-477b-a378-f6de98e4e013 UrO0ABXQAGXdvcmtmbG93IGluc3RhbmNlIHN0YXJ0ZWQ= String

Do you know the reason?

Regards,

Kay

Michael Austermann

unread,
Sep 20, 2017, 1:48:57 AM9/20/17
to copper...@googlegroups.com
In copperContext.xml the BatchingAuditTrail is configured to use a message postprocessor for compression (see CompressedBase64PostProcessor).
Well, from my point of view this is probably confusing in a beginners example, so I removed it. See

Now you should see your message clear text.

/Michael


Von: "Kay" <mao...@gmail.com>
An: "COPPER Engine" <copper...@googlegroups.com>
Gesendet: Mittwoch, 20. September 2017 03:54:36

Betreff: [COPPER Users] Re: Evaluating Copper workflow engine
--
You received this message because you are subscribed to the Google Groups "COPPER Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to copper-engin...@googlegroups.com.
To post to this group, send email to copper...@googlegroups.com.

Kay

unread,
Sep 20, 2017, 9:41:51 PM9/20/17
to COPPER Engine
Thanks Michael,

Now I do see the clear text.

Back to the second question - 2) How can I add a new workflow (say HelloWorldWorkFlowV2) into the Orchestration Engine instead of resetMailBox?

Here is my use case:

1) The workflow engine needs to be run as a standalone Java application on one server.
2) Oracle database is on another DB server.
3) The client application, which receives requests from other systems, will call Copper workflow engine API to fire up the request related workflow (similar to HelloWorldTestApplication.java). Another option is that the client application sends JMS messages from another server to the workflow engine server. Then the workflow engine picks up the messages and fire up the corresponding workflows.
4) The workflows will send JMS messages to several external systems for querying and updating and process the response JMS messages accordingly.
5) Workflow instances need to stay in the database even after the workflow instances complete successfully for audit and support purpose. The completed workflow instances can be deleted later on (e.g. after 1 year).
6) Workflows set timer for something to happen in future. For example, a required change will be applied to the external system if no objection received in 2 weeks.
7) Workflows interact each other. When an objection is received, another workflow is initiated. This new workflow needs to notify the existing request workflow which is in waiting status (waiting for 2 weeks) to cancel the required change.

Is this something achievable by Copper? Does OrchestrationEngine fit in this use case? 

Regards,

Kay

Michael Austermann

unread,
Sep 25, 2017, 2:30:01 AM9/25/17
to copper...@googlegroups.com
To add a new workflow to the copper starter project do the following:
1. extend OrchestrationEngine.wsdl, e.g. by adding a new service "foo"
2. Start gradle build to create new jaxb and cxf binding
3. Create your new workflow, e.g. FooWorkflow in src/workflow/java
4. implement the new service "foo" in OrchestrationInputAdapter, e.g. by calling "engine.run("org.copperengine.examples.orchestration.wf.FooWorkflow", ...)

And regarding your use case:
from my point of view, copper fits your requirements pretty well.

But I wouldn't leave the workflow instances in the database after completion. The main copper database tables, e.g. COP_WORKFLOW_INSTANCE, are very volatile and for the reason of performance, I wouldn't mess them up with historic data.
If you need audit data for a year or so, then you should keep that in the Audit-Table but not in the others.

/Michael


Von: "Kay" <mao...@gmail.com>
An: "COPPER Engine" <copper...@googlegroups.com>
Gesendet: Donnerstag, 21. September 2017 03:41:51

Betreff: [COPPER Users] Re: Evaluating Copper workflow engine
--
You received this message because you are subscribed to the Google Groups "COPPER Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to copper-engin...@googlegroups.com.
To post to this group, send email to copper...@googlegroups.com.

Kay

unread,
Sep 25, 2017, 7:18:11 PM9/25/17
to COPPER Engine
Excellent! Thanks Michael for the confirmation.

I will give it a go.

Regards,

Kay

Kay

unread,
Sep 26, 2017, 1:13:07 AM9/26/17
to COPPER Engine
Hi Michael,

I have a few more questions if you don't mind:
1) We could have tens of thousands workflow instances running for several months. Of course they are all persisted in the database table. But what about the memory usage? Are these instances in the memory as well? In other words, if a workflow instance is in 'wait' (say for 2 weeks), is the instance still hold in memory or is removed from the memory?
2) In the ResetMailbox workflow sample, there is a retry loop for 5 times. If 5 times are all failed, the workflow gives up. This is OK if the external system is not accessible for a short period (say less than one hour). But this workflow internal retry will not work if the external system has days of outage. The workflow is totally dead. Is there a way we can change the data in the database so that the dead workflow instance can be reprocessed from where it stopped? For example, if I change the STATE field value in the 
COP_WORKFLOW_INSTANCE table from 5 to 2, will it cause the dead workflow instance being processed NORMALLY again?

Regards,
Kay

Michael Austermann

unread,
Sep 26, 2017, 1:42:22 AM9/26/17
to copper...@googlegroups.com
Hi Kay,

1)
If a workflow instance is not running, then it is not in memory but only in the database.
So you could have millions of wf instances unless your database is able to handle them.

2)
If a workflow instance throws an uncaught exception (an exception that is not handled by itself but by the copper engine arround), then processing of this workflow instance is aborted and its state is changed to ERROR (5).
You can wake up a workflow instance in this state by using the jmx interface org.copperengine.management.PersistentProcessingEngineMXBean.restart(String)
The workflow instance will then resume at its last successful checkpoint.

/Michael



Von: "Kay" <mao...@gmail.com>
An: "COPPER Engine" <copper...@googlegroups.com>
Gesendet: Dienstag, 26. September 2017 07:13:06

Betreff: [COPPER Users] Re: Evaluating Copper workflow engine
--
You received this message because you are subscribed to the Google Groups "COPPER Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to copper-engin...@googlegroups.com.
To post to this group, send email to copper...@googlegroups.com.

Kay

unread,
Sep 26, 2017, 10:34:29 PM9/26/17
to COPPER Engine
Hi Michael,

Thanks again for your quick response. Good to know our concerns are all addressed in Copper  :-).

Regards,

Kay
Reply all
Reply to author
Forward
0 new messages