max-msg-timeout

[Wes Carr]

The description for this flag is a little unclear. Am I correct in assuming that this is the maximum amount of time a topic can spend in an un-paused channel before being dropped and marked as timed out? On a related note, do these messages correspond to the "Timed Out" count when viewing the channel details in the nsqadmin web UI?

Thanks,
Wes

[Matt Reiferson]

Hi Wes,

There are a couple of concepts here…

First an important point, timeouts only apply to messages that are in-flight (have been sent to a consumer), not queued.

When nsqd delivers a message to a consumer it starts a timeout while waiting for a response.  There are 3 possible responses.  FIN "finish" will tell nsqd that the consumer is done and to discard the message, REQ "requeue" will tell nsqd that something went wrong and it should put the message back on the queue (with an optional defer duration), and TOUCH indicates that the consumer needs more time to process the message and nsqd should extend the timeout.

On the nsqd side there are a few configurable options that affect the above behavior.

msg-timeout - The initial timeout for a message after being sent to a consumer.  If the timeout expires without a response the message will be automatically re-queued and delivered to a ready consumer.

max-msg-timeout - A consumer can TOUCH a message and extend the default timeout up to this value.  Effectively, this is the maximum amount of time a msg can remain in-flight.

In nsqadmin (or just /stats output of nsqd) "timed out" counts the # of messages that were re-queued due to unresponsive clients according to above.

Hope this helps.  Let me know if you have any other questions.

Regards,

Matt

[Wes Carr]

Hey Matt,

Thanks for the thorough response, that all makes perfect sense. I just wanted to make sure that nsqd didn't have a notion of giving up on messages. I've been using the python client and it does appear to have a max number of retries on messages before giving up but I see it's configurable to our taste.

Cheers,

Wes

[Sergey Ivanov]

Hi Matt,

previously you was talking about requeue "with an optional defer duration", could you please explain it, how i could set the delay to zero, for example? i mean, i need rejected message to be put first (or near that) in the queue.

[Matt Reiferson]

go-nsq, pynsq, etc. (full featured client libraries) support a parameter to the Requeue() function… if you specify 0 for that param, it will disable the defer.

[Sergey Ivanov]

yes, as i see, in the go-nsq it is Config/DefaultRequeueDelay.
thank you very much. 

[Sergey Ivanov]

Well, i found a strange behaviour of requeue despite these parameters. Probably I dont understood some of requeue principles.
I set config.DefaultRequeueDelay =0 in my consumer. Then I start Producer and generate 20 messages marked with "time_sent" to queue. And after that, I start Consumer and it receive messages strictly as the were put in. It's ok.
Then, with an empty queue, I repeat generating, and with Consumer sent first 5 msgs to requeue (return error from message handler) and stop consumer.
After restarting consumer - I didnt receive requeued messages (as expected), first I received last part of the queue and only then -  the requeued messages.
Could you please explain it, where I am wrong? 

[Matt Reiferson]

The “deferred requeue” does not specify order, just the timeout until a requeued messages returns to the queue (in the back).