How to detect slow consumer in nats-streaming ?

[Kishore Karunakaran]

How to detect slow consumer with nats-streaming using client API ?

Is there a way to create subscribers dynamically in nats ?

[iv...@synadia.com]

Kishore,

Could you clarify what you mean by slow consumer in the context of NATS Streaming?

Are you concerned that old messages are removed from the message log before the 

consumers had a chance to consume them? 

You could have a look at those links:

https://github.com/nats-io/nats-streaming-server/issues/631

Also, here is a recent discussion on Slack:

----------------

Zack Bartel [5:29 PM]

if my consumer is unable to keep up with the incoming messages, can i tell when undelivered messages "fall off" the end of the stream (reach the default 1M limit)

Ivan Kozlovic [5:37 PM]

This is a default config and can obviously set to what you want. You can even configure per-channel limits. Note: the old messages are removed, not the new. That being said, no, there is no notification that the old messages are dropped. Your consumer may suddenly get gaps in message sequence. This is not an indication that this happened, since for instance it always happens with queue groups as long as you have more than 1 member.

Zack Bartel [5:38 PM]

i see, if i have multiple consumers and the average gap between sequence numbers jumps up, would that be a reasonable indication that im not fast enough?

Ivan Kozlovic [5:40 PM]

if you are not using queue subs, maybe, but let me ask you: what would you do with that information?

Zack Bartel [5:41 PM]

confirm my suspicion that that is where data loss is occurring

Ivan Kozlovic [5:41 PM]

if you use `-m <port` for monitoring, then you can check state of channel and its subs

Zack Bartel [5:42 PM]

"loss" isnt the right word, but confirm my suspicion that because i am not able to keep up with the incoming data and cannot process the stream in time the messages are aging out

Ivan Kozlovic [5:42 PM]

check this: https://github.com/nats-io/nats-streaming-server#monitoring

GitHub

nats-io/nats-streaming-server

nats-streaming-server - NATS Streaming System Server

Zack Bartel [5:42 PM]

got that in front of me now

Zack Bartel [5:42 PM]

added this Plain Text snippet: Untitled 

{

"max_channels": 100,

"max_msgs": 1000000,

"max_bytes": 1024000000,

"max_age": 0,

Ivan Kozlovic [5:43 PM]

do that and replace `foo` with your channel you want to investigate: `http://localhost:8222/streaming/channelsz?channel=foo&subs=1`

(well change host:port obviously :wink: )

Zack Bartel [5:44 PM]

aha

would first_ and last_ seq diverge?

Ivan Kozlovic [5:47 PM]

these are for the channel, indicating the first and last sequence number in that channel. When first_seq is not 1, it means that some messages have been dropped, either because of count, size or age

Zack Bartel [5:52 PM]

first_seq": 23134540,

"last_seq": 24134539,

well thats no good

Ivan Kozlovic [5:53 PM]

you sent 24M messages to a channel capped to 1M, so yeah.. maybe you should revisit your limits :wink:

Zack Bartel [5:53 PM]

heh

Ivan Kozlovic [5:54 PM]

but more seriously, is all that data needed persistence?

Zack Bartel [5:54 PM]

in theory if my consumer can keep up, and i dont care about storing them for very long, thats OK?

no it is not

maybe just a day or so

Ivan Kozlovic [5:55 PM]

so you should size how many messages/size 2 days worth of publish data is and then set limits for this channel.

As for your consumers, if you want to make sure that they consume all messages being produced, I guess you would have to measure how long it takes for a consumer to consume a certain amount.

but if you can pause or throttle the publisher, that won’t really help

Zack Bartel [5:58 PM]

if my consumer is backlogged more than 1M would the last_sent be about 1M lower than the last_seq?

Ivan Kozlovic [6:02 PM]

last_sent is the last msg seq the server sent to this consumer. So unless the server has sent all messages to the consumer, last_sent will be lower than last_seq. Say consumer started before any messages are published and is very slow, and publisher publishes very fast, the last_sent will be equal to maxinflight, consumer will be stalled, and last_seq will climb. After a while, you may even observe that last_sent is smaller than first_seq, but as soon as the consumer sends an ack back, the server resumes delivery, and will “fill the gap” and send the first avail message. So the consumer’s lastSent will jump ahead.

Zack Bartel [6:08 PM]

gotcha, thank you sir

---------------------

Ivan.

[Kishore Karunakaran]

Thanks for Pointing out the link the discussion, 

My Publisher is faster than the consumer(Subscriber) and We basically use durable queue subscribers. 

We want to scale our subscribers.