nflow performance tests

[Julian Purse]

Hi, 

I am looking for any articles on nflow performance settings and throughput/ horisontal scale setup. are there some links that can be shared ?

Regards.

Julian

[nflow-users]

Hi and sorry for late reply!

We have not published any articles about nflow performance. But we have fixed issues with scaling and performance issues as they have been noticed.

What kind of performance you are aiming for? Usually the bottleneck has been some external system, not nflow or nflow’s database. Something like 10 actions per second should be easily achievable even on a smallish database. There is one nflow user that has used nflow for years and has processed more than 2 billion (2 000 000 000) actions with averaging over 10 actions per second.

Br, Juha

[Julian Purse]

Hi, 

Some context, based in banking and financial industry. we are using nflow for a few of the complex multi system workflows, ie go into the different systems and perform the separate actions in sequence, and its doing that really well. single worker/db embedded in spring. this is running in the orders of a few workflows per minute.

the real question being if we started loading smaller workflows but lots of them, ie this may have bursts up to 50 workflows per second. if that is the case. 

* at what point do you run multiple workers, 

* are there indications of ie "pending workflows" that give indications of when you need to start modifying configs.

* i have seen the folder in github https://github.com/NitorCreations/nflow/tree/master/nflow-perf-test

    is this for running benchmarks against a DB/worker setup to get some baselines ? 

* any idea on the hardware/worker configs that the large company you mentioned has used ? 

happy to help in creating some benchmark guidelines.

-Julian 

[Mikko Tiihonen]

I've seen bursts of 10k workflows, which can cause large delays in processing. To combat those we have added few features:

• priority for workflows (you can mark the bursts as lower priority)
• batch workflow support where you link the jobs under one batch workflow that then executes the child workflows in smaller batches and thus does not steal the executors from other concurrent work

The other scalability parameters are:

• number of executor threads in each executor
• number of executors (usually one per JVM)

The number of threads/executors and scalability depends very much on what the workflows do. Usually the workflows run remote calls to other systems and are not CPU intensive so it is safe to add quite large amount of threads for each executor.The database will at some point also become bottleneck. Each step requires updates/inserts to 2-3 tables. As long as the maintenance is enabled that cleans the finished workflows from the tables (or moves to archive tables). The database does not usually slow things down since the sql statements are tuned well.

The system does log a warning when the wanted execution time of a workflow is much delayed from the actual execution time, which indicates that either something is stuck (all executors waiting for a service that is not responding) or the systems is not fast enough, or the bursts were to big to handle quickly.

You can also utilize autoscaling since it is safe to add/remove executors on the fly (they do their best do do graceful shutdown). But when running more than one executor do remember that it is recommended to first update the workflow code of all executors before starting to schedule workflows that need them.

-Mikko