How long NSQ messages are persisted for?

[kant kodali]

Hi,

How long NSQ Messages are persisted for ? can a consumer come back after one week or one month and read all the message from the begining ?

Thanks!

[Pierce Lopez]

On Thursday, April 6, 2017 at 3:58:31 AM UTC-4, kant kodali wrote:

How long NSQ Messages are persisted for ? can a consumer come back after one week or one month and read all the message from the begining ?

A message queued on a channel will stay queued on that channel until it is received by a consumer and the consumer sends a "finish" command for that message to nsqd, to indicate it is done processing it. After the "finish" command, nsqd deletes the message from the queue. That could theoretically be months after the message was published, but usually it is seconds or minutes after.

If nsqd is stopped and restarted gracefully (with SIGINT or SIGTERM) then it saves all messages which are queued in memory to disk, and reloads them from disk, so they are not lost. But if nsqd (or the whole server) crashes, then messages in the memory queues on that server are lost. (There are also disk-queues, used when the memory queues exceed a limit.)

For long-term archiving of messages, we typically use the "nsq_to_file" utility to consume from an extra channel and create a log file of messages, in hour- or day- long parts. It is then possible to "replay" those messages into the same or a new topic using a separate utility.

NSQ isn't Kafka, it works differently in some respects ... it may help to read http://nsq.io/overview/design.html a couple of times.

[kant kodali]

On Thu, Apr 6, 2017 at 10:48 AM, Pierce Lopez <pierce...@gmail.com> wrote:

On Thursday, April 6, 2017 at 3:58:31 AM UTC-4, kant kodali wrote:

How long NSQ Messages are persisted for ? can a consumer come back after one week or one month and read all the message from the begining ?

A message queued on a channel will stay queued on that channel until it is received by a consumer and the consumer sends a "finish" command for that message to nsqd, to indicate it is done processing it. After the "finish" command, nsqd deletes the message from the queue. That could theoretically be months after the message was published, but usually it is seconds or minutes after.

   Ok say the messages are persisted and a consumer comes back after 2 weeks. Can that consumer now read from where it left off for same topic and channel?  If yes, does it require nsqd to be up? what happens if nsqd goes down during that 2 week period and comes back up?

[Pierce Lopez]

 Yes, Yes, and Yes if nsqd exits gracefully each time it "goes down" (e.g. for nsqd upgrade or server restart). nsqd keeps track of the progress on a topic/channel. Assuming nsqd does not lose data for some reason, it keeps the messages until a consumer gets in contact, receives them, and finishes them.

[kant...@gmail.com]

Thanks!

Sent from my iPhone