Architecture and decisions for a high-performance, heavy workload, and dynamic multi tenant app

[cmastr...@gmail.com]

Hello everyone, I'm in charge of building a complex system for my company, and after some research decided that Camunda fits most of my requirements. But some of my requirements are not common, and after reading the user guide I realized there are many ways of doing the same thing, so I hope this question will clarify my thoughts and also will serve as a base questión for everyone else looking for building something similar.

First of all, I'm planning to build a specific App on top of Camunda BPM. It will use workflow and BPM, but not necessarily all the stuff BPM/Camunda provides. This means it is not in my plans to use mostly of the web apps that came bundled with Camunda (tasks, modeler...), at least not for end users. And to make things more complicated it must support multiple tenants... dynamically.

So, I will try to specify all of my requirements and then hopefully someone with more experience than me could explain which is the best architecture/solution to make this work.

Here we go:
- Single App built on top of Camunda BPM
- High-performance
- Workload (10k new process instances/day after few months).
- Users (starting with 1k, expected to be ~ 50k).
- Multiple tenants (starting with 10, expected to be ~ 1k)
- Tenants dynamically managed (creation, deploy of process definitions)
- It will be deployed on cluster
- PostgreSQL
- WildFly 8.1 preferably

After some research, this are my thoughts
- One Process Application
- One Process Engine per tenant
- Multi tenancy data isolation: schema or table level.
- Clustering (2 nodes) at first for high availability, and adding more nodes when amount of tenants and workload start to rise.

Doubts
- Should I let camunda manage my users/groups, or better manage this on my app? In this case, can I say to Camunda “User X completed Task Y”, even if camunda does not know about the existence of user X?
- What about dynamic multi tenancy? Is it possible to create tenants on the fly and make those tenants persist over time even after restarting the application server? What about re-deployment of processes after restarting?
- After which point should I think on partitioning of engines on nodes? It’s hard to figure out how I’m going to do this with dynamic multi tenancy, but moreover... Is this the right way to deal with high workload and growing number of tenants?
- With my setup of just one process application, should I take care of something else in a cluster environment?

I'm not ruling out using only one tenant, one process engine and handle everything related to tenants logically within my app, but I understand that this can be very (VERY!) cumbersome.

All answers are welcome, hopefully we'll achieve a good approach to this problem.

Regards,
Cristian.

[thorben....@camunda.com]

Hi Cristian,

I'll try to answer the questions. Just for reference, I assume you have already read this portion of the user guide that deals with multi-tenancy and the one-engine-per-tenant approach: [1].

On question 1:
Task assignees/candidates etc. need not exist in the engine's identity tables, so you can entirely decouple them. You can also exchange the engine's identity server or configure an LDAP. If you're interested in that, a starting point to read could be [2].

On question 2:
Creating a tenant using this approach requires two things: First, creating the necessary database tables/schemas. While Camunda provides the scripts for this, you'll have to find a solution yourself to perform this step dynamically.
Second, creating a process engine and registering it with the platform. This is easily doable in Java code, cf [3]. You can also programmatically register a process engine as a shared engine. With re-start of the application server, these registrations will be lost. Camunda does not persist process engine configurations, so you'll have to provide such an infrastructure yourself, e.g. in the most simple form a table that maps tenants to their database schema and some code that bootstraps each tenant's engine on server startup. Process re-deployment is not necessary, since deployments are persisted to the database.
Deploying a single process application to a dynamic set of engines might also require some custom code, since this is not supported at the moment in the processes.xml deployment descriptor, cf [4].

On question 3:
It's hard to give good advice on partitioning in general. However, with the engine-per-tenant approach you are quite flexible in this regard. You can partition engines across nodes as you like and you can configure each engine against different physical databases as you like. Factors influencing the need to partition would be the load of requests served by a node and the job executor workload of asynchronous tasks.

On question 4:
With one process application, you'll have to make sure to to deploy it to every cluster node and you'll have to look into the issue of deploying one application to a dynamic number of engines (as mentioned in my answer on question 2).

Best regards,
Thorben

[1] http://docs.camunda.org/latest/guides/user-guide/#process-engine-multi-tenancy
[2] http://docs.camunda.org/latest/guides/user-guide/#process-engine-identity-service
[3] http://docs.camunda.org/latest/guides/user-guide/#process-engine-process-engine-bootstrapping
[4] https://stackoverflow.com/questions/28489074/camunda-multi-tenancy-processes-xml

[webcyberrob]

Hi,

Ive not tried this myself, but conceptually it could work. If you embraced Amazon Web Service cloud, you could create an App/Engine/RDS configuration and thus have a dedicated instance per tenant. The AWS operational tools my help you commoditize this. However, this architecture comes with consequence and that is your operational overheads grow linearly with number of tenants, eg 'n' tenant instances, 'n' databases to administer etc.

For what its worth, on a basic AWS 2cpu tomcat node using RDS (Mysql 4 cpu) I was easily able to drive the Tomcat node to create 110 new process instances per second. In addition, with 50 threads in the job executor, I could make 50 remote procedure calls per second. Thats 4 million service calls (eg service task executions) in 24 hours. Hence a little node can get you a very long way without trying hard at all...

regards

Rob

[cmastr...@gmail.com]

Thorben, thank you for your time and all your answers!

As I see, one application (one single war) dealing with multiple tenants dynamically can be somewhat difficult to handle. Both the dynamic creation of databases as bootstrapping each tenant's engine on server startup (potentially 1k engines) may become a headache. This, coupled with the problem of deploying a single process application to a dynamic set of engines, make me think in this solution as somewhat hard to implement and maintain.

Even the partitioning could require a lot of care, because since I'll have one application and multiple engines distributed across different nodes (each one against different schemas), then I should take care of balancing the users of every tenant to the node which has that tenant's engine.

So, I'm wondering what if I have one Process Application, and just one Process Engine for all my "tenants". Since all of them will use the same app across all nodes, maybe it's better to manage all the multi tenancy related stuff internally because it's more like "logical multitenancy" rather than "physical multitenancy". Of course I'll have to use the Tenant Marker per process instance approach and deal with the tenancy separation internally.

So, the question this time would be if you consider with this mentioned setup Camunda will be able to handle all this workload properly.

- One Process Application
- One Process Engine, replicated across different nodes.
- Almost 10k process definitions in the worst scenario
- Up to 10k process instances per day

One thing that I think makes some noise in this solution is from the performance point of view, would be better to have multiple engines to manage that load? Or do you think one engine (clustered) can perfectly manage all of it?

Againg, thanks for your time :)
Regards,
Cristian.

[cmastr...@gmail.com]

Hi Rob!

Those numbers are pretty impressive, thanks for sharing. With that on mind, I can manage to attempt sizing my solution in terms of nodes needed per cluster to handle my expected workload.

Again, thinking on "tenants" as logical and internal concept, this could work very well.

Regards,
Cristian

[thorben....@camunda.com]

Hi Cristian,

On concrete numbers I can't give you a good answer. Other users like Rob who have setup such a scenario can help you better in that regard. In general, there are users (like Rob) who have numbers in that order of magnitude and higher and have no problems.

As for your last question, one engine can be configured against one database. If your database can handle all the tenants' load, you're perfectly fine. If your database can't handle it, you'll need an additional engine that points to another database. It's as simple as that :) I can't tell you though when that threshold is reached.

Cheers,
Thorben