Stuck in QUEUE?

[Howie T]

Hi all,

I wonder if anyone has experienced this strange behavior -- after starting the workflow, it only inserted itself in COP_QUEUE and stuck there for as long as it takes. Only after restarting the JVM running engine the workflow instance then suddenly resumes from the workflow Workflow.wait, with with response == null. 

This happens randomly, making debugging and tracing very hard. So, if anyone have any ideas or directions please share with me.

Thank you very much!

Howie

[Theo Diefenthal]

Hi Howie,

 

Can you provide a few more details like which COPPER Version you are currently using with which database? Cluster mode or one engine – one db?

 

Best regards

Theo

 

SCOOP Software GmbH - Gut Maarhausen - Eiler Straße 3 P - D-51107 Köln
Theo Diefenthal

T +49 221 801916-196 - F +49 221 801916-17 - M +49 160 90506575
theo.di...@scoop-software.de - www.scoop-software.de
Sitz der Gesellschaft: Köln, Handelsregister: Köln,
Handelsregisternummer: HRB 36625
Geschäftsführung: Dr. Oleg Balovnev, Frank Heinen,
Martin Müller-Rohde, Dr. Wolfgang Reddig, Roland Scheel

--
You received this message because you are subscribed to the Google Groups "COPPER Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to copper-engin...@googlegroups.com.
To post to this group, send email to copper...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/copper-engine/2328fdaa-bce6-44ae-8b47-d5183852d70d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Howie T]

Hi Theo,

Thank you for the fast reply.

I'm using 4.2.0 release, MariaDB database(MysqlDialect),  one engine, one db and one JVM. 

Also some stats: around 5000 workflow instances, 10000 wait events with only the stuck instances in COP_QUEUE. 

I think with increasing number of workflows, this happens more frequently.

Thank you! 

Howie

[Howie T]

So.. yeah, if we have many workflows constantly waking up(like having 100 workflows in a loop sleep(1000)) this seems to happen more frequently.

What's the dequeue  strategy? Did we sort by ascending creation date?

[Theo Diefenthal]

Hi Howie,

 

I just looked over the COPPER code in “insert” and “dequeue” behavior and don’t see anything coming up into my eyes what could go wrong only from time to time. (Which doesn’t mean that there isn’t anything. Nasty bugs are always nasty to find 😊) But I also know that this COPPER version is used in some production environments with Oracle DB having no observed problems thus far with tons of workflows.

 

To limit the search area of this problem:

• Are you running with or without the batcher being enabled?
• Could you switch to an alternate database like PostgreSQL and check whether this issue still occurs?

To view this discussion on the web visit https://groups.google.com/d/msgid/copper-engine/4e30021f-87fe-486b-9d56-8d9a368c71b4%40googlegroups.com.

[Theo Diefenthal]

Hi Howie,

 

One more question: What happens if you run the perftest from COPPER?

I just run the perftest from COPPER 4.2.0 like so:

 

1. Setup a MariaDB docker image (So to be compatible with you: MariaDB with MySQL dialect)

version: '3'

services:

  mariadb:

    image: mariadb:latest

    ports:

     - "3306:3306"

    environment:

     - MYSQL_ROOT_PASSWORD=copper4711

     - MYSQL_DATABASE=copper

     - MYSQL_USER=copper

     - MYSQL_PASSWORD=copper4711

    volumes:

     - mysqlcopperdata:/var/lib/mysql

volumes:

  mysqlcopperdata:

 

2. Setup regtest properties (Create File projects/copper-performance-test/src/main/resources/performancetest.YOURUSERNAME.properties (Note: YOURUSERNAME should be your system username, it will override the default properties then). Following content in my case:

ds.jdbcURL=jdbc:mysql://localhost:3306/copper?zeroDateTimeBehavior=convertToNull
ds.user=copper
ds.password=copper4711

throughput.numberOfWfI=5000
throughput.dataSize=8000
throughput.numberOfInsertThreads=4
throughput.batchSize=100
throughput.numberOfExtraProcPools=0

compression=false
mockAdapter.numberOfThreads=8
procPool.numberOfThreads=8

 

 

3. Run main method projects/copper-performance-test/src/main/java/org/copperengine/performancetest/main/Main.java with program argument “throughput”.

 

• Works on my machine. I see that the program terminates fine (Even though fairly slow on my windows environment with mysql in the docker container) and all workflows were executed.

 

Btw: In your first mail, you mentioned that the workflow gets stuck in COP_QUEUE and after restart, the workflow resumes with response == null. This is actually the desired standard behavior. If a workflow is set to be run, it is not executed immediately but put into the SQL database. It immediately is put into the COP_QUEUE table (Thus showing that this workflow is ready to be executed) and usually immediately afterwards fetched for execution (by dequeing). In the very first time that a workflow is inserted, it is not waiting for any response and it is not into a timeout, thus response == null and timeout is false. It’s just strange that your workflow is not fetched by the dequeue… next time you observe this issue, can you copy an example of such a “broken row”? What is written in the engine column?

 

Best regards

Theo

 

SCOOP Software GmbH - Gut Maarhausen - Eiler Straße 3 P - D-51107 Köln
Theo Diefenthal

T +49 221 801916-196 - F +49 221 801916-17 - M +49 160 90506575
theo.di...@scoop-software.de - www.scoop-software.de
Sitz der Gesellschaft: Köln, Handelsregister: Köln,
Handelsregisternummer: HRB 36625
Geschäftsführung: Dr. Oleg Balovnev, Frank Heinen,
Martin Müller-Rohde, Dr. Wolfgang Reddig, Roland Scheel

From: copper...@googlegroups.com <copper...@googlegroups.com> On Behalf Of Howie T

Sent: Freitag, 10. August 2018 05:19

To view this discussion on the web visit https://groups.google.com/d/msgid/copper-engine/4e30021f-87fe-486b-9d56-8d9a368c71b4%40googlegroups.com.