Slow operation within ReadWithPartitionIteratorP

[vladc...@gmail.com]

Hi,

We occasionally experience very slow batch jobs and the only problem I noticed in Hazelcast diagnostics is a slow operation when doing

ReadWithPartitionIteratorP.java:127 (which seems to use MapFetchWithQueryOperation)

Full stack trace attached as an image. Also, I see that Hz InvocationMonitor sometimes complains about backup-timeouts.

Any idea what could be slowing down the iterator? Note that we're using Hazelcast Jet 0.7.2

Thanks,

Martin

[Can Gencer]

Is your cluster stable during these events (nodes being added/removed?)

Are you using predicates?

[vladc...@gmail.com]

Using Predicates.alwaysTrue(). The cluster looks more or less stable, but the specific node manifest backup operation timeouts. 

M

Dne pondělí 6. dubna 2020 13:36:19 UTC+2 Can Gencer napsal(a):

[Can Gencer]

Hi,

Is this the same problem you mentioned on the hazelcast group, or a different one specific to Jet?

--
You received this message because you are subscribed to the Google Groups "hazelcast-jet" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast-je...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast-jet/923b740e-fca8-4b85-939e-81c3e354e40a%40googlegroups.com.

This message contains confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version. -Hazelcast

[vladc...@gmail.com]

Possibly, but not sure. That's why I am trying to find out here (from the Jet perspective) what it means when ReadWithPartitionIteratorP is working slowly.

Martin

Dne úterý 7. dubna 2020 15:05:32 UTC+2 Can Gencer napsal(a):

Hi,

Is this the same problem you mentioned on the hazelcast group, or a different one specific to Jet?

On Mon, Apr 6, 2020 at 4:06 PM <vladc...@gmail.com> wrote:

Using Predicates.alwaysTrue(). The cluster looks more or less stable, but the specific node manifest backup operation timeouts. 

M

Dne pondělí 6. dubna 2020 13:36:19 UTC+2 Can Gencer napsal(a):

Is your cluster stable during these events (nodes being added/removed?)

Are you using predicates?

On Sunday, April 5, 2020 at 8:43:17 AM UTC+2, vladc...@gmail.com wrote:

Hi,

We occasionally experience very slow batch jobs and the only problem I noticed in Hazelcast diagnostics is a slow operation when doing

ReadWithPartitionIteratorP.java:127 (which seems to use MapFetchWithQueryOperation)

Full stack trace attached as an image. Also, I see that Hz InvocationMonitor sometimes complains about backup-timeouts.

Any idea what could be slowing down the iterator? Note that we're using Hazelcast Jet 0.7.2

Thanks,

Martin

--
You received this message because you are subscribed to the Google Groups "hazelcast-jet" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hazelc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast-jet/923b740e-fca8-4b85-939e-81c3e354e40a%40googlegroups.com.