how does nats streaming server know subscriber is dead

[yang sheng]

Hi

If a subscriber is dead for some reason, how does nats server knows about that and dispatch the msg the dead worker was working on to another worker?  

[Ivan Kozlovic]

The server periodically sends heartbeats to the client. If the client fails to respond a certain number of times, the client is removed from the server.

Look for `hb_interval` in the table of this section: https://github.com/nats-io/nats-streaming-server#configuration-file

[yang sheng]

Thanks!

Though it is removed from server, will the msg get re-delivered immediately or have to wait till acktimeout?

[Ivan Kozlovic]

If you are referring to a member of a queue group, then message will be redelivered right away when the server closes the connection due to timeout.

[Ivan Kozlovic]

At least, this is behavior you would get from master branch. There have been some issues fixed around that behavior since last release.

[yang sheng]

I see.

I am using it for a long running job. It could take hours or even days. If the "member of queue group" is lost for some reason, we want to have another member pickup the job. We are kind of achieving this by setting a relatively small `ackTimeout` (i am not sure why we are doing this....). So basically all msgs are re-delivered and checked if this is a redelivery msg and status of the job (stored in DB).

this is a misuse of acktimeout. Trying to figure out the correct way.

BTW we are using 0.9.6... yes this is an ancient version.

[Ivan Kozlovic]

Like I said, you can configure the server to reduce the amount of time it would take to detect a connection as dead. In that case, the redelivery will be done as soon as the server removes the connection (and its subscription). There is no NATS Streaming 0.9.6 version, but yeah, like I said, there are bugs that have been fixed in master so you would need to upgrade.

[yang sheng]

nats-streaming-server version 0.6.0, nats-server version 0.9.6

Ok will try the master first.

So we don't even need to overwrite the acktimeout I guess? Also whats the use case of acktimeout then?

[Ivan Kozlovic]

Ok will try the master first.

Since you are using very old version of NATS, please take note of the backward compatibility section from the latest release notes: https://github.com/nats-io/nats-streaming-server/releases/tag/v0.15.1

But you should use master to benefit from latests fixes.

 

So we don't even need to overwrite the acktimeout I guess? Also whats the use case of acktimeout then?

The subscription's AckWait indicates how long the server should wait to redeliver a message after sending it if it did not receive an ack from the subscription. Streaming server and clients are not directly connected through TCP, so there is no way for server to know that if a message was sent it was properly received. The ack is used for that, along with simple way for application to tell the server that they are done with a message and should move to next, etc..