Replaying parked messages and knowing when all messages were processed to set proper $tb

[Marcin B]

Hi,

I have multiple consumers subscribing to one stream (6 consumers to 3 subscription groups).

All the data needs to be processed.

When we cannot process message we park it and a scheduled job triggers replay of parked messages (this is done by each consumer).

Another scheduled job sets stream retention using the following logic

- if there are any parked messaged -> no update to $tb

- If there are any messages in the parked buffer -> no update to $tb

- If two above conditions are not met set $tb to lowest last processed event number from all subscription groups

Recently I have noticed that if there are replayed messages from the parked stream and can be processed immediately (consumers start processing them) then $tb is being updated - as no parked messages are present in the parked stream (replay was called) and retryBufferSize is 0 - as there are no parked messages awaiting their turn to be replayed.

Is there any way to read the amount of replayed parked messages that are in-flight? Any way to read if "replay parked messages" is ongoing?

What is outstandingMessagesCount in the response returned from /subscriptions/{streamName}/{groupName}/info?

Thank you,

Marcin

[Greg Young]

So you can look at how many messages are outstanding on the group but there is not I believe to see if they are in flight. The in flight messages are *all* messages.

What you are looking for is essentially another metric "in flight messages from parked" and/or a flag of "parked replay in progress"?

Is this correct?

--
You received this message because you are subscribed to the Google Groups "Event Store" group.
To unsubscribe from this group and stop receiving emails from it, send an email to event-store...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/event-store/8b18c794-b502-4f99-8069-8eb8f04f8305%40googlegroups.com.

--

Studying for the Turing test

[Marcin B]

Basically - yes.

Maybe there is also another way that I can know when I can safely change the $tb pointer?

W dniu wtorek, 28 kwietnia 2020 17:39:04 UTC+2 użytkownik Greg Young napisał:

So you can look at how many messages are outstanding on the group but there is not I believe to see if they are in flight. The in flight messages are *all* messages.

What you are looking for is essentially another metric "in flight messages from parked" and/or a flag of "parked replay in progress"?

Is this correct?

On Tue, Apr 28, 2020 at 11:29 AM Marcin B <marba...@gmail.com> wrote:

Hi,

I have multiple consumers subscribing to one stream (6 consumers to 3 subscription groups).

All the data needs to be processed.

When we cannot process message we park it and a scheduled job triggers replay of parked messages (this is done by each consumer).

Another scheduled job sets stream retention using the following logic

- if there are any parked messaged -> no update to $tb

- If there are any messages in the parked buffer -> no update to $tb

- If two above conditions are not met set $tb to lowest last processed event number from all subscription groups

Recently I have noticed that if there are replayed messages from the parked stream and can be processed immediately (consumers start processing them) then $tb is being updated - as no parked messages are present in the parked stream (replay was called) and retryBufferSize is 0 - as there are no parked messages awaiting their turn to be replayed.

Is there any way to read the amount of replayed parked messages that are in-flight? Any way to read if "replay parked messages" is ongoing?

What is outstandingMessagesCount in the response returned from /subscriptions/{streamName}/{groupName}/info?

Thank you,

Marcin

--
You received this message because you are subscribed to the Google Groups "Event Store" group.

To unsubscribe from this group and stop receiving emails from it, send an email to event...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/event-store/8b18c794-b502-4f99-8069-8eb8f04f8305%40googlegroups.com.