go-nsq in flight message consumption issue

[Nitin Thakur]

This is the code to register consumers

AddConcurrentHanlders(handler, 1)

So the concurrency factor is of 1.

But on nsqadmin is see these consumers have ready count of 3 and also in flight count of 3 when messages are coming.

What my understanding is that this count of 3 means we can concurrently process 3 messages.But for that i will have to add a value of 3 not 1. However my doubts are  

1> Internally does each consumer will create 3 go routines to process these messages 

2> Will topic wait for confirmation of all 3 messages from consumer and then send a fresh batch.

3> What is one of the message sends requeue or timeout while other succeed . Will topic send a new batch of 2 new messages and 1 existing messages? 

4> though each handler only have one nsq message as argument. Does in this case it means it will register 3 hanlders?

Thank you in advance

[Pierce Lopez]

The number of messages "in-flight" for a particular consumer, is not necessarily linked to the number of concurrent handlers in that consumer. Having a higher "in-flight" can hide the round-trip latency between the nsqd and the consumer, by having some messages sent, and taking some milliseconds to arrive, while other messages are being processed.

Messages are not associated in or with batches. There might be multiple sent at pretty much the same time, but they can be finished or requeued or timed-out independently. If the consumer's max-in-flight is 3, and 3 messages are sent to it by nsqd, and soon after the consumer finishes 1 message, then nsqd will see that only 2 are still "in-flight" and will be able to send 1 more message to bring in-flight back up to 3 (... if that was the max).

Anyway, with the go-nsq client library, you can separately choose the Config.MaxInFlight, and how many concurrent handlers to run, and it often makes sense for max-in-flight to be higher.

I hope that helps,

- Pierce

[Nitin Thakur]

Hi Pierce, 

Thank you so much for this information. I was going through one more config which is 

MaxAttempts uint16 `opt:"max_attempts" min:"0" max:"65535" default:"5"`

this defines the max attempts after which the messages will be marked finished (dropped) in case of retries. I have a use case where i need to retry messages again and again as we are sending data to third party with some API rate limit. Is there any way i can have this to be of max value or infinite so that messages is not dropped. Or any suggestion you would give.

(I read a thread where is was mentioned to have requeue with backoff for such case as it will disable consumer for that time). But still would want to know your points 

Thanks again 

Nitin