NodeJs subscriber sees long (+2 secs) delays

[Peter Frøkjær Strand]

@google-cloud/pubsub 2.15.1

Single topic with single subscriber in pull mode.

Setup is that I publish around 500 messages in one second.

Subscriber quickly picks up, but after 100 messages, there is a 1-2 second delay until the next 100 messages are received. Handling the 500 messages takes around 7-8 seconds in total, so in a real world scenario the client would quickly fall behind.

• nodejs process is not stalled during delay. A timer that logs on console every .5 sec verifies this
• client side processing is not causing delays. Only processing is sending back ack
• I tried to modify all relevant settings in subscriber setups, including 
  • flowControl : { maxMessages : 1000 },
  • batching: { maxMessages:  1000, [...] '

If I set the flowcontrol maxMessages to 10, I see the same pattern as described above, but with batches of 10 messages being received. Setting it above 100 always keeps the message amount received at 100.

Question - is there anything setup-wise that I have overlooked? It seems like a fairly trivial setup, and the system seems more than capable to handle this kind of load. 

thanks,

Peter

[Peter Frøkjær Strand]

OK - figured this out. It turns out I used an ordering key, which - in this test setup - was the same for each message. If I remove ordering key I get the expected throughput (~1000 messages / sec ).

Still seems like an issue that an ordering key would put this limit on performance - is this expected?

thanks

Peter

[Samar Amara]

Hi,

There should not be such a performance difference between using an ordering key  and not using it. If you are using the Go client library it may be related to this known issue[1].

Furthermore  what do mean by “in this test setup - was the same for each message. ``?

Do you see the same issue when using an ordering key but not the same for all messages?

/Samar

[1]: https://github.com/googleapis/google-cloud-go/issues/3884

[Samar Amara]

Nevermind the question about the client library I see you are using Node JS.

/Samar

[abou...@google.com]

Delays if one used the same ordering key across all messages would not be unexpected. The server and client coordinate carefully to ensure that messages are in fact sent in order. What is probably happening is that you are getting a single batch of messages, acking them all, and then another batch of messages is coming once it is determined that the first 100 messages were acked. Ordering keys is really designed for cases where one has many different keys and messages can be delivered in parallel for different keys. However, to determine if the level of delay you are seeing is expected or not, you'd really have to enter a support case and provide your project ID and subscription name so that it could be investigated in more depth.

Kamal

On Monday, July 12, 2021 at 12:38:08 PM UTC-4 aaar...@gmail.com wrote: