Hazelcast support for reactive streaming and sync API

[Gaurav Gupta]

We are evaluating frameworks to build an orchestrator with reactive streaming capability and sync API capability. 

 

On a high level the requirements of our orchestrator are to support:

1. Sync and Async calls for customer request with guaranteed SLA to serve the client request
2. Streaming and batch request from our clients
3. Dynamic DAG (workflow) support to serve the client usecases (onboarded as declarative spec on our multi-tenant system) – i.e. clients define the workflow in form of configuration (can vary across clients’ business usecase), the workflow we define in the orchestrator should be executable as a state machine (I.e conditionally call functions based on output of previous function).
4. High throughput and processing at scale (scale to 80K-100K requests per second).
5. Support long running workflow execution (~20 minutes per request) as well as low latency ones (~20ms) for different client needs.
6. Lower deployment time (~1-2 minutes) per host.
7. Ability to recover from failed state and resume through checkpoints.

 

We read about Hazelcast and see a good match of certain capabilities with Hazelcast Jet offering. However, there is a few things we need further clarity on, can you help us with answers to below:

1. Can Hazelcast Jet support sync calls with reactive stream processing.
2. Can hazelcast support batch calls? I see mention of streaming support but nothing about batch processing.
3. Can it support dynamic DAG based workflow execution (based in outputs from predeceesor processor In the workflow).
4. Support for long running transactions (executions) ~10’s of minutes.
5. While Jet can achieve 99.99^th percentile in milisecond of latency, can we implement APIs that offer strict SLA to clients at each step in workflow. Will there be any loss in time in event processing (i.e. when event moves from one processing node/operator to other in streaming application), that the application layer will not have control on (consider a DAG workflow with 250 operators) – due to event streaming across nodes in Hazelcast Jet.
6. How much time does an application deployment take in Hazelcast (considering the application jar is somewhere around 200MB).  Can we have instances warmed up to start serving traffic quickly?

 

Looking forward for response to above queries. TIA.

[Gaurav Gupta]

Any updates on this please? Can we sync on this?