NATS streaming reconnecting to a subscription

[Thorsten von Eicken]

I'm not managing to get clear on ordering guarantees provided by NATS in the context of resubscribing after a disconnection. The problem I'm trying to solve is for a client to reconnect after some communication failure without missing any messages.

A MsgProto contains a timestamp and a sequence number. Are the sequence numbers guaranteed to be monotonically increasing? Are the timestamps guaranteed to be monotonically increasing? Can I resubscribe using either as a starting point? (I guess since the subscription takes a StartTimeDelta which mixes the client and the server's notion of time the timestamp is already ruled out?) Are there gotchas WRT retransmission and just keeping track of the highest sequence number received on the client side?

[Ivan Kozlovic]

Thank you for your interest in NATS Streaming.

Yes, sequence numbers on a given channel are monotonically increasing. However, there are cases due to message limits (count/size/age) that old messages get discarded. So it could be possible for a subscriber to get some messages from the beginning of the log, subscriber go down, some of the messages at the beginning of the log being removed due to limits, and the subscriber on restart would then see a gap.

The timestamp is assigned by the server, and as for sequence numbers, on a given channel the timestamp will be also monotonically increasing. For the starting point, you can use sequence and/or time, with, as you said, the caveat that server and clients may not have the exact same view of time.

When a durable subscription restarts, or if the server restarts while subscribers (durable or not) were still running, a redelivery of non-acknowledged messages occurs. If a subscriber is stalled (server sent up to MaxInflight messages without receiving an ack), the redelivery is postponed up to 3 times the AckWait interval. This may cause a subscriber to start receiving "new" messages and then a batch of old unacknowledged messages (see https://github.com/nats-io/nats-streaming-server/issues/187). I plan to change that to make redeliveries always ignore the fact that MaxInflight messages were already sent.

That being said, with message redeliveries, there will always be out-of-order delivery. Suppose you receive messages 1, 2, 3, 4 and 5, and don't ack message 1 and 2 right away. New messages come in, say 6, 7 and 8. The the redelivery timer kicks in and messages 1 and 2 are redelivered. From an application perspective it may seem that they are out-of-order, but they are simply redelivered.

Hope this helps.

Ivan.

[Thorsten von Eicken]

Thanks for the thorough answer! I think I still need to re-read a couple of sections a few times :-)

I'm a little puzzled by the whole retransmission stuff in the context of non-durable subscriptions. The lifetime of the subscription is the TCP connection. The recipient receives the bytes and thus the messages in-order. I understand the use of ACKs for the purpose of flow-control, but I fail to see the point of retransmitting. Is there a way for the subscriber to turn off retransmissions?

WRT timestamps, what was the reasoning behind using a StartTimeDelta as opposed to an absolute StartTime in the subscription request?

[Ivan Kozlovic]

Thorsten,

 

I'm a little puzzled by the whole retransmission stuff in the context of non-durable subscriptions. The lifetime of the subscription is the TCP connection. The recipient receives the bytes and thus the messages in-order.

With NATS Streaming, there is no direct TCP connection between streaming clients and streaming server. The picture would be like this:

Streaming server <-> NATS Server <-> Streaming client

 

I understand the use of ACKs for the purpose of flow-control, but I fail to see the point of retransmitting.

Since there is no direct connection, the streaming server does not know for sure if messages were received or not. Subscriber's ACK are therefore used both for flow control and ensure that client receive messages.

 

Is there a way for the subscriber to turn off retransmissions?

Technically no, but you could use a very high AckWait() duration.

 

WRT timestamps, what was the reasoning behind using a StartTimeDelta as opposed to an absolute StartTime in the subscription request?

The idea is that a subscriber can receive messages that have been sent in the last X seconds (or any unit). By the way, there is a StartAtTime() which use absolute time and not delta.

Hope this helps.

Ivan.

[Thorsten von Eicken]

On Thursday, November 17, 2016 at 6:36:36 AM UTC-8, Ivan Kozlovic wrote:

Thorsten,

 

I'm a little puzzled by the whole retransmission stuff in the context of non-durable subscriptions. The lifetime of the subscription is the TCP connection. The recipient receives the bytes and thus the messages in-order.

With NATS Streaming, there is no direct TCP connection between streaming clients and streaming server. The picture would be like this:

Streaming server <-> NATS Server <-> Streaming client

Whooops! I clearly don't understand the architecture :-). Makes sense but it's unexpected. Where can I find docs on the NATS streaming architecture? In the end, I'm a bit puzzled. Having used Kafka I really think that for reliable stream consumption the model where messages are strictly ordered and the position is defined by the offset into the stream is the right one. Is this implementable client-side using NATS streaming or is it just a different model?

Overall, I'm having a hard time figuring out what properties NATS streaming is providing. Is there a page I should read? For example, are sequence numbers consecutive? If not, how could a client tell that it missed some messages? You write "on a given channel the timestamp will be also monotonically increasing" but I have not found that stated & explained anywhere in the docs (probably my fault for not searching well enough) and I'm wondering how that works when running in a clustered configuration.

Thanks much for the responses!

[Ivan Kozlovic]

Whooops! I clearly don't understand the architecture :-). Makes sense but it's unexpected. Where can I find docs on the NATS streaming architecture?

Check doc on nats.io: http://nats.io/documentation/streaming/nats-streaming-intro/

Also, the README.md of both server and go-nats-streaming client may help.

 

In the end, I'm a bit puzzled. Having used Kafka I really think that for reliable stream consumption the model where messages are strictly ordered and the position is defined by the offset into the stream is the right one. Is this implementable client-side using NATS streaming or is it just a different model?

Like I said, on a channel, sequences are monotonically increasing. If you start a subscriber at any give position, it will receive messages in order. The out-of-order situation may come in case of message redelivery as discussed previously.

 

Overall, I'm having a hard time figuring out what properties NATS streaming is providing. Is there a page I should read? For example, are sequence numbers consecutive? I

Yes they are. But as explained, if you set some store limits, it is possible that messages get discarded, which then may appear as gaps in the subscriber. In NATS Streaming, a channel store messages as a ring buffer if you will. If you set limits on that channel (max number, size or age), the server will discard older messages to make room for the new ones. But suppose you don't set any limit, then messages will be sequentially ordered from 1 to infinity (and beyond ;-)).

 

f not, how could a client tell that it missed some messages?

See above. 

 

You write "on a given channel the timestamp will be also monotonically increasing" but I have not found that stated & explained anywhere in the docs (probably my fault for not searching well enough)

The timestamp is assigned by the server when receiving the message from the publisher and before storing it. So timestamp, like sequence number is increasing. Technically two consecutive timestamps could be equal depending on the speed of incoming messages and if time resolution is not precise enough.

 

and I'm wondering how that works when running in a clustered configuration.

That's a good point but right now there is no NATS Streaming cluster support. 

Ivan.