Cannot sustain large number of requests and concurrency

[Cheng Li]

Hi all,

Currently, when running the shipped CounterServiceDemo with three replicas, I identified the following two problems:

a) If I issued 10000 requests continuously, then leader replica would hang and output:

changed role to Follower

I don't why the workload will trigger the leadership change.

b) If I issued 10000 requests at two replicas, then the service hung and no plausible responses were produced. An exception might be triggered as follows:

java.lang.RuntimeException: java.util.concurrent.TimeoutException

Can anybody help me figure out what happened?

Thanks,

    Cheng

[Bela Ban]

(b) is a bug that I was able to reproduce quickly: [1]. It seems to be
related to client redirection (REDIRECT), but I need to investigate
further. But this won't happen until some time next week as I'm on PTO
starting today.

(a) I was not able to reproduce this. Perhaps FD_ALL suspected a member
because of the load. You can try and increase the timeout in FD_ALL,
e.g. <FD_ALL timeout="60000".../>.
Cheers,

[1] https://github.com/belaban/jgroups-raft/issues/39

> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-raft" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-raft...@googlegroups.com
> <mailto:jgroups-raft...@googlegroups.com>.
> To post to this group, send email to jgroup...@googlegroups.com
> <mailto:jgroup...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jgroups-raft/fddec1e6-b1b1-47b1-9e9d-6c4fd7f10d5f%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-raft/fddec1e6-b1b1-47b1-9e9d-6c4fd7f10d5f%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

--
Bela Ban, JGroups lead (http://www.jgroups.org)

[Cheng Li]

Hi Bela,

I think for b) you can even reproduce it when you use two threads in the same CounterService server to issue concurrent requests. I don't think that is connected to REDIRECT. It should be the problem in a single CounterServer instance.

Thanks,

    Cheng

> <mailto:jgroups-raft+unsub...@googlegroups.com>.

[Bela Ban]

Can you create a unit test and attach it to issue #39 (see the link below) ?

> > an email to jgroups-raft...@googlegroups.com <javascript:>
> > <mailto:jgroups-raft...@googlegroups.com <javascript:>>.

> > To post to this group, send email to jgroup...@googlegroups.com

> <javascript:>
> > <mailto:jgroup...@googlegroups.com <javascript:>>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/jgroups-raft/fddec1e6-b1b1-47b1-9e9d-6c4fd7f10d5f%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-raft/fddec1e6-b1b1-47b1-9e9d-6c4fd7f10d5f%40googlegroups.com>
>
> >

> <https://groups.google.com/d/msgid/jgroups-raft/fddec1e6-b1b1-47b1-9e9d-6c4fd7f10d5f%40googlegroups.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/jgroups-raft/fddec1e6-b1b1-47b1-9e9d-6c4fd7f10d5f%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> Bela Ban, JGroups lead (http://www.jgroups.org)
>

> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-raft" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-raft...@googlegroups.com

> <mailto:jgroups-raft...@googlegroups.com>.

> To post to this group, send email to jgroup...@googlegroups.com
> <mailto:jgroup...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jgroups-raft/74c41688-d572-4b74-be4a-3ccb109932d1%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-raft/74c41688-d572-4b74-be4a-3ccb109932d1%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Bela Ban]

OK, so the culprit seems to be RAFT.resend_interval which by default is 1000. If you lower it to (e.g.) 100, performance increases. I'm investigating why this is the case. Subscribe to [1] to get updates.

[1] https://github.com/belaban/jgroups-raft/issues/39

>      > <mailto:jgroups-raft+unsub...@googlegroups.com <javascript:>>.

>      > To post to this group, send email to jgroup...@googlegroups.com
>     <javascript:>
>      > <mailto:jgroup...@googlegroups.com <javascript:>>.
>      > To view this discussion on the web visit
>      >
>     https://groups.google.com/d/msgid/jgroups-raft/fddec1e6-b1b1-47b1-9e9d-6c4fd7f10d5f%40googlegroups.com
>     <https://groups.google.com/d/msgid/jgroups-raft/fddec1e6-b1b1-47b1-9e9d-6c4fd7f10d5f%40googlegroups.com>
>
>      >
>     <https://groups.google.com/d/msgid/jgroups-raft/fddec1e6-b1b1-47b1-9e9d-6c4fd7f10d5f%40googlegroups.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/jgroups-raft/fddec1e6-b1b1-47b1-9e9d-6c4fd7f10d5f%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>      > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
>     --
>     Bela Ban, JGroups lead (http://www.jgroups.org)
>
> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-raft" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to jgroups-raft+unsubscribe@googlegroups.com
> <mailto:jgroups-raft+unsub...@googlegroups.com>.

> To post to this group, send email to jgroup...@googlegroups.com

> <mailto:jgroups-raft@googlegroups.com>.

[Bela Ban]

Fixed in https://github.com/belaban/jgroups-raft/issues/39