Job priority

[vladc...@gmail.com]

Hi,

Is there any chance to configure job priority?

Problem: We have several Hazelcast jobs and need to achieve that under heavy load the job A won't affect job B. In other words, CPU capacity should be preferably allocated to messages processed by job B.

Martin

[Viliam Durina]

We don't support setting thread priority. It's even not possible to do so for cooperative processors because different jobs run on the same thread.

An indirect way to achieve that effect is to use higher parallelism for the prioritized job.

Viliam

[marko.t...@gmail.com]

I would add to this that it's an even better idea to constrain the local parallelism to 1 on all stages in the low-priority pipeline.

[Can Gencer]

Hi Martin,

I'm also curious if this is something you've tested (i.e. under heavy load you notice that both jobs slow down) or just something you are thinking of implementing as a precaution?

--
You received this message because you are subscribed to the Google Groups "hazelcast-jet" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast-je...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast-jet/b71c57e3-528c-41b1-a326-f9f8a4868d6d%40googlegroups.com.

This message contains confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version. Thank you, 

[vladc...@gmail.com]

Hi.

Well, we have about ten ongoing stream processing jobs and we observe that one overloaded job can screw total performance of the application, so just thinking about possibilities.

I noticed that Edge has a "priority" attribute. Do you think it would help to merge the jobs into one complex DAG and play with the edge priority?

Martin

Dne úterý 28. ledna 2020 13:41:41 UTC+1 Can Gencer napsal(a):

Hi Martin,

I'm also curious if this is something you've tested (i.e. under heavy load you notice that both jobs slow down) or just something you are thinking of implementing as a precaution?

On Tue, Jan 28, 2020 at 12:20 PM <vladc...@gmail.com> wrote:

Hi,

Is there any chance to configure job priority?

Problem: We have several Hazelcast jobs and need to achieve that under heavy load the job A won't affect job B. In other words, CPU capacity should be preferably allocated to messages processed by job B.

Martin

--
You received this message because you are subscribed to the Google Groups "hazelcast-jet" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hazelc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast-jet/b71c57e3-528c-41b1-a326-f9f8a4868d6d%40googlegroups.com.

[marko.t...@gmail.com]

Is it maybe due to some blocking code in the processor logic? All the processors execute round-robin, and a processor gets all the data that was enqueued while it wasn't running, so this seems like a self-balancing scheme without starvation.

You can't use edge priority for anything performance-related. A stream on a higher-priority edge will be consumed in full before any lower-priority edges are even considered. This feature is used for example in a hash join, which needs all the data from the side inputs before it starts processing the main stream.

[Vladimir Schreiner]

Hi Martin,

I agree with Marko. Start with profiling the jobs to find the bottleneck. 

If the bottleneck turns to be the CPU, consider isolating resources on an OS level. E.g. deploy multiple Jet clusters and use a resource manager (Docker, virtual environment, …) to assign CPU cores to respective Jet JVM processes. You can operate high and low priority clusters. Jet is fairly lightweight so the overhead of multiple clusters running on one physical host shouldn’t be dramatic.

Vladimir 

28. 1. 2020 v 14:25, marko.t...@gmail.com:

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast-je...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast-jet/62521aba-ce88-4a5d-8f70-4891d5c56662%40googlegroups.com.

[Viliam Durina]

And more to Marko's point: you can also use Edge queue length. A processor gets all the data that was enqueued while it wasn't running, but the queue size is limited. If you use a small queues for your lower-priority jobs, they will get smaller batches each time and will have to yield more often and leave more time for other jobs. Unlike tuning the local parallelism, this works only for cooperative processors. However this can have quite complex interactions because a full queue also prevents the upstream processor from pushing more data, up to the source.