Backend workers restart

[Jiří Sedláček]

Hi,

I have 8 backend workers that ZeroGW forwards to. The workers use the ZMQ_REP socket type. I need to restart the workers periodically but so far was unable to do so.

The problem is lost requests. Workers restart after rand(N) requests. I close the socket after responding to the last request and then exit.

By "lost requests" I mean that GET from outside would not return.

I tried setting high watermark in zerogw config to 1. Didn't help.

Any ideas?

Thanks
Jiri

[Paul Colomiets]

Hi Jiri,

Unfortunately non loosing messages on disconnect is impossible in
zeromq by design. If you use ZMQ_REP, then I believe you are talking
about HTTP, not websockets, right? Then it's possible to setup timeout
for requests:

zmq-forward:
timeout: 5 # seconds
retry: !RetryLast 3 # three times

The !RetryLast means that if request is retried, the request id is
changed, so if late reply is received, it would be discared. In
!RetryFirst mode, we retry request but not change the request id, so
that first response arrived is sent back to a client, no matter which
attempt it was (good of idempotent requests).

--
Paul

[Jiří Sedláček]

Hi Paul,

I tried that and it works but still can't use that because of (not idempotent) POST.

My colleague came with this - a proxy between ZeroGW and worker. The proxy would fork the worker and restart it after every N requests.

My motivation for restarting the backend HTTP worker is a memory leak.

Jiri

[Jiří Sedláček]

Just notifying that we used the retry solution after all. POSTs are (in our case) almost always idempotent. When not then nothing really bad happens so we can live with it.

Thanks
Jiri

[Paul Colomiets]

Hi Jiri,

On Fri, Nov 21, 2014 at 2:46 PM, Jiří Sedláček
<yirie.se...@gmail.com> wrote:
> Just notifying that we used the retry solution after all. POSTs are (in our
> case) almost always idempotent. When not then nothing really bad happens so
> we can live with it.
>

Cool. Thanks for sharing it!

Depending on the task, it's often also possible to make non-idempotent
queries idempotent, by first creating a "cookie" that is put in redis
or memcache, and checking it before doing expensive operation.
"cookie" expire time should be equal to total time request can be
repeated, so usually kinda couple of tens of seconds. The downside is
more complex handing of the operation on client, and working with
redis/memcache. The upside however is that it handles not only retries
between zerogw and workers, but also network failures between client
(browser) and zerogw.

--
Paul