Deleting unused subjects/channels automatically on a long running nats-streaming broker

[Aki Yoshida]

I have a few questions regarding the above topic and would like to ask how I should proceed. 

I have a nats-streaming use case where some of the subjects are no longer used and can be closed or deleted. In particular, when using the file persistence mode, not having the possibility of closing the unused subjects will lead to the situation of running out of the available file descriptors. 

To solve this problem, I have experimented with an idea of adding an automatic channel deletion at the message expiration handling code. The idea is to optionally trigger an automatic channel deletion if the message expiration code deletes the last message from the channel. This approach requires only a small change in the code but has some limitation (e.g., when a channel is created by a subscriber and no message is ever stored, the channel cannot be deleted as no message expiration code is executed). But if we schedule the message expiration timer at the channel creation time, I think we could handle this case as well.

There are more elaborate and flexible approaches possible but I don't know if something in this are is concretely planned. I would appreciate for your feedback on the above approach.

By the way, another related thing that I am experimenting with is to bound the number of active subjects when using the file store mode in order to avoid hitting the limit for the opened file descriptors by temporarily closing the files for the less used subjects and reopening them on demand). If there are many subjects and some subjects are less used but yet still in use, this approach can reduce the file descriptor usage for the broker.

regards, aki

[Ivan Kozlovic]

Aki,

Thanks for your interest in NATS Streaming.

We definitively need a way to delete channels, but not sure yet if this is the way to go. What would happen to existing subscribers on that channel? I am not sure that the action of deleting a channel can be done at the level of the store itself. This would have implications to the server. Server may still holds subscriptions on that channel. If later a new message is sent on that now deleted channel, server would not find it, ask the store to create it. But since it has running subscribers, if messages are delivered to them, that would cause problem when server tries to update the subscription state.

Again, I think we need a way, but not sure yet how safely implements this.

[Ivan Kozlovic]

And on your last comment, we may have to do that indeed. I actually already do that for the file "slices" on a given channel. They are opened/closed when looking up messages that are not in the current slices. However, I am planning to change the way file slices work, so may have to add that to the list.

Thanks!

On Monday, November 14, 2016 at 4:14:18 AM UTC-7, Aki Yoshida wrote:

[Colin Sullivan]

Another approach toward scalability here would be to create another store type that does not require the use of as many file descriptors, perhaps backed by a database.  I've opened a PR for this and assigned the issue to myself, but it may be a little while before I can get to it. 

If you'd like to experiment, certainly feel free to do so - contributions are welcome!

Colin

[Aki Yoshida]

Hi Ivan,

thanks for your comment. Initially I thought about doing the deletion over some admin type of mechanism (e.g., running over REST or as part of the nats wire protocol). But it seemed that different approaches have advantages and disadvantages in different use case and it would be practical to have one admin type of mechanism and an automatic mechanism.

Regarding the subscription behavior of the automatic channel deletion, the old subscribers won't get the new messages when a channel is deleted and later recreated because the new channel has a different subject prefix from the one used by the old subscribers. In other words, the old subscribers have to be restarted to get the new prefix so that they subscribe to the newly created channel. 

I think this behavior isn't wrong in principle. And one of the use cases for this automatic deletion is a scenario where some publisher and subscriber components are dynamically started and later shutdown. In this case, the subscribers are managed for the duration of the scenario's activity and there are no stale subscribers. 

In other use cases where the old subscribers need to be automatically terminated or reassigned to the new channel, we could send some control or error message to the old subscribers and let the subscribers react to that message. But this could be done at some later time as an improvement to the above channel deletion mechanism, no?

If you think this approach is worth considering, I am happy to share my code.

Thanks.

regards, aki

[Ivan Kozlovic]

Hi Aki,

Regarding the subscription behavior of the automatic channel deletion, the old subscribers won't get the new messages when a channel is deleted and later recreated because the new channel has a different subject prefix from the one used by the old subscribers.

Either I missed something (very possible), or this is very specific to your use case. That cannot be generalized. Say message is published in channel "foo" which has a message age limit. A subscriber starts on "foo" and gets the message. The message finally expires and with your change the channel store is deleted. Some time later, another (or same) publisher publishes on "foo". Sever will create this store channel but has still a live subscriber on that channel. Not sure if this is good.

You also have the edge scenario where say that message is sent on "foo" and there is a 1 minute age limit (just for demonstration). A subscriber starts at 59 second, get that message, but processing takes a bit of time and so will acknowledge in 2 seconds. The store expires the message and deletes the channel store. Subscriber acknowledges the message: oops.

 

In other words, the old subscribers have to be restarted to get the new prefix so that they subscribe to the newly created channel. 

Again, that may be very specific to your use case but is not the general use case.

 

I think this behavior isn't wrong in principle. And one of the use cases for this automatic deletion is a scenario where some publisher and subscriber components are dynamically started and later shutdown. In this case, the subscribers are managed for the duration of the scenario's activity and there are no stale subscribers. 

So would that automatic channel deletion on last expired message be for all channels or based on some configuration? If for all, again, not sure that is acceptable for all users.

 

 

If you think this approach is worth considering, I am happy to share my code.

Sure, if you have a fork/branch that has this change I would be happy to look at it.

Thanks for your feedback and contribution!

Ivan.

[Aki Yoshida]

Hi Ivan,

On Tuesday, 15 November 2016 17:43:06 UTC+1, Ivan Kozlovic wrote:

Hi Aki,

Regarding the subscription behavior of the automatic channel deletion, the old subscribers won't get the new messages when a channel is deleted and later recreated because the new channel has a different subject prefix from the one used by the old subscribers.

Either I missed something (very possible), or this is very specific to your use case. That cannot be generalized. Say message is published in channel "foo" which has a message age limit. A subscriber starts on "foo" and gets the message. The message finally expires and with your change the channel store is deleted. Some time later, another (or same) publisher publishes on "foo". Sever will create this store channel but has still a live subscriber on that channel. Not sure if this is good.

You also have the edge scenario where say that message is sent on "foo" and there is a 1 minute age limit (just for demonstration). A subscriber starts at 59 second, get that message, but processing takes a bit of time and so will acknowledge in 2 seconds. The store expires the message and deletes the channel store. Subscriber acknowledges the message: oops.

Yes. There are some edge cases. There is no perfect guard against some edge cases unless we synchronize everything which we don't want. So, I think what we could do is while  leaving some edge cases but providing a practical option to run into such situations. One possibility would be to introduce another timer which gets triggered after the last message expires and let this second timer handle the actual deletion. This will require another property to set this time duration, but it may be shown to be practical.

Regarding the subscribers, I offered several delete options using an integer parameter (see below).

 

 

In other words, the old subscribers have to be restarted to get the new prefix so that they subscribe to the newly created channel. 

Again, that may be very specific to your use case but is not the general use case.

Maybe, I am doing something different. In any case, you can take a look at my change and observe the behavior. 

It is in my forked repo's work-autodelete branch

https://github.com/elakito/nats-streaming-server/tree/work-autodelete

and the commit ID link is

https://github.com/elakito/nats-streaming-server/commit/e21baac7419dc8bebe844807d36338e000f96500

What the current code does is:

- this auto deletion mode is configurable via --store_auto_delete

- the deletion is triggered during the message expiration and not using a another timer (I tend to think we should use another timer to handle the case where no message is ever stored and also to offer another buffer time until the actual deletion. So I would like to hear what you think)

 

I think this behavior isn't wrong in principle. And one of the use cases for this automatic deletion is a scenario where some publisher and subscriber components are dynamically started and later shutdown. In this case, the subscribers are managed for the duration of the scenario's activity and there are no stale subscribers. 

So would that automatic channel deletion on last expired message be for all channels or based on some configuration? If for all, again, not sure that is acceptable for all users.

 

As mentioned, the behavior is controlled by property --store-auto-deletion (0 by default for no deletion, 1 for delete if no subscribers present, 2 for delete always).

 

If you think this approach is worth considering, I am happy to share my code.

Sure, if you have a fork/branch that has this change I would be happy to look at it.

If you could look at the change and give me some further feedback, that would be appreciated,.

Thanks.

Regards, aki

[Ivan Kozlovic]

Hi Aki,

Sorry for the delay. I had a look and here are some of my comments.

- It is good that you introduce an option to: disable, only if no subscriber or force. This gives more flexibility. Unfortunately, having the process of deleting the channel done at the store level, again, introduces risk in the server. There is for instance a risk of "panic" if at the store level a channel is deleted while the server was trying to flush the store. We can't take that risk.

- I don't think that this option should be defined in store channel limit. To me, it is an option, not a limit on a channel. I understand the convenience of it, but does not seem right.

- With the same idea, I don't think that the DeleteChannel's Store API should have the "force" boolean. Based on the option, the caller should know if it can or not call DeleteChannel.

- DeleteChannel is a store API and yet only called internally. That reinforces the idea that this API should be called by the server, not the store implementation.

Again, I think we need to find a way for the channel to be deleted safely by the server. We may still need the option to delete a channel regardless of the number of subscribers, but it should be done by the server in places where it knows that no concurrent access to a store can happen or have an inconsistent state.

I hope you understand that we want this feature to be introduced in the mainline, we cannot assume that producers/consumers will always be stopped before the last message expire, etc.. we can't afford to have the server panic due to such feature.

I am thinking that server could have a timer that periodically checks the store's MsgState() for a given channel and take the decision to delete the channel (say if there is no subscriber). Still this timer would then need to be scheduled to the main protocol processing to ensure that if the action is not happening at the same time that server is trying to flush a given store. I am making some changes to the server that may facilitate that (https://github.com/nats-io/nats-streaming-server/issues/195).

Thanks!

Ivan.

[Aki Yoshida]

Hi I van,

On Thursday, 17 November 2016 15:49:38 UTC+1, Ivan Kozlovic wrote:

Hi Aki,

Sorry for the delay. I had a look and here are some of my comments.

No problem. Thank you for your detailed comments.

 

- It is good that you introduce an option to: disable, only if no subscriber or force. This gives more flexibility. Unfortunately, having the process of deleting the channel done at the store level, again, introduces risk in the server. There is for instance a risk of "panic" if at the store level a channel is deleted while the server was trying to flush the store. We can't take that risk.

OK. I understand that we need to make sure that there will be no panic resulting from some concurrent channel deletion and data writing. But this could be handled in the store level and not in the server level, as the server always accesses the channel over its store interface anyway, no? 

- I don't think that this option should be defined in store channel limit. To me, it is an option, not a limit on a channel. I understand the convenience of it, but does not seem right.

Yes. I saw this semantic mismatch but for this prototype I took took the advantage of using this convenient approach. We could move the property to the appropriate group.

- With the same idea, I don't think that the DeleteChannel's Store API should have the "force" boolean. Based on the option, the caller should know if it can or not call DeleteChannel.

The reason for adding this force option was that this DeleteChannel method may be in the future invoked by some administrative service to support both cases conveniently, I passed this option to the method. Are you saying you would prefer to make this method always delete the channel and the caller should determine whether there are subscribers and decide to invoke this delete method (i.e., equivalent to force being true)or not (equivalent to force being false)?

- DeleteChannel is a store API and yet only called internally. That reinforces the idea that this API should be called by the server, not the store implementation.

As mentioned above, I was hoping to get both CreateChannel and DeleteChannel etc to be accessible from some admin service. As symmetric to CreateChannel to the store API, I thought DeleteChannel would fit in there.

Again, I think we need to find a way for the channel to be deleted safely by the server. We may still need the option to delete a channel regardless of the number of subscribers, but it should be done by the server in places where it knows that no concurrent access to a store can happen or have an inconsistent state.

I have to look into this stuff. I thought we could use the lock within the store to guard against such cases but I am probably missing something.

 

I hope you understand that we want this feature to be introduced in the mainline, we cannot assume that producers/consumers will always be stopped before the last message expire, etc.. we can't afford to have the server panic due to such feature.

Yes. I agree. 

 

I am thinking that server could have a timer that periodically checks the store's MsgState() for a given channel and take the decision to delete the channel (say if there is no subscriber). Still this timer would then need to be scheduled to the main protocol processing to ensure that if the action is not happening at the same time that server is trying to flush a given store. I am making some changes to the server that may facilitate that (https://github.com/nats-io/nats-streaming-server/issues/195).

I also think using a specific timer for deleting a channel is preferable than using the message expiration timer which cannot handle the case where there is no message ever stored. When using another timer, we need to ensure its action doesn't interfere with the server's store flush operation and also the store's message expiration action.

I'll spend some time looking into these points.

regards, aki

 

Thanks!

Ivan.

[Ivan Kozlovic]

Hi Aki.

OK. I understand that we need to make sure that there will be no panic resulting from some concurrent channel deletion and data writing. But this could be handled in the store level and not in the server level, as the server always accesses the channel over its store interface anyway, no? 

It does, but it is a matter of ordering of actions. Say the store decides that it can delete a channel because all messages have expired. You start the go routine to delete the channel (at least the way you are doing it in your branch). In the meantime, messages arrive in that channel about to be deleted. The server invokes the store interface to store those messages. The go routine doing the channel delete then execute, and then, the server call Store.Msgs.Flush() => panic.

 

- I don't think that this option should be defined in store channel limit. To me, it is an option, not a limit on a channel. I understand the convenience of it, but does not seem right.

Yes. I saw this semantic mismatch but for this prototype I took took the advantage of using this convenient approach. We could move the property to the appropriate group.

Good.

 

The reason for adding this force option was that this DeleteChannel method may be in the future invoked by some administrative service to support both cases conveniently, I passed this option to the method. Are you saying you would prefer to make this method always delete the channel and the caller should determine whether there are subscribers and decide to invoke this delete method (i.e., equivalent to force being true)or not (equivalent to force being false)?

Yes. I would think that DeleteChannel() should not have the force flag. The caller should decide if it is safe to call this or not.

 

As mentioned above, I was hoping to get both CreateChannel and DeleteChannel etc to be accessible from some admin service. As symmetric to CreateChannel to the store API, I thought DeleteChannel would fit in there.

I am not arguing that DeleteChannel should be in the store interface. I think it should, but to me it makes a stronger case that this should be called by the server itself, not internally by the store.

 

I have to look into this stuff. I thought we could use the lock within the store to guard against such cases but I am probably missing something.

Store has its own lock, but as I have described above, this is not a problem of concurrent access per se, but a problem of order between events. In your approach, by the time the store decides to delete the channel, messages and or subscribers may be added to that channel. I understand why you did start the delete through a go routine (locking and such), but this is leaving room for races.

 

I also think using a specific timer for deleting a channel is preferable than using the message expiration timer which cannot handle the case where there is no message ever stored. When using another timer, we need to ensure its action doesn't interfere with the server's store flush operation and also the store's message expiration action.

I'll spend some time looking into these points.

When https://github.com/nats-io/nats-streaming-server/pull/196 is merged, it will be possible to send to the ioChannel control messages that will ensure that things are processed in order. The server could have a timer that simply put a control message in the channel and that control message could then be processed in the ioLoop.

You are welcome to experiment, and I may also get to this at one point. So many things to do ;-)

Thanks,

Ivan.

[Aki Yoshida]

Hi Ivan,

On Friday, 18 November 2016 16:02:23 UTC+1, Ivan Kozlovic wrote:

Hi Aki.

OK. I understand that we need to make sure that there will be no panic resulting from some concurrent channel deletion and data writing. But this could be handled in the store level and not in the server level, as the server always accesses the channel over its store interface anyway, no? 

It does, but it is a matter of ordering of actions. Say the store decides that it can delete a channel because all messages have expired. You start the go routine to delete the channel (at least the way you are doing it in your branch). In the meantime, messages arrive in that channel about to be deleted. The server invokes the store interface to store those messages. The go routine doing the channel delete then execute, and then, the server call Store.Msgs.Flush() => panic.

I see your point which you described in details at the end of your previous reply. But then the code will need to be synchronously executed by the same busy IOLoop routine. This approach itself is clean and leaves no room for conflicts, but I am not sure if we may consider an alternative approach that works for 98% of the case and handles 2% error cases gracefully than the current panic() approach.

 

- I don't think that this option should be defined in store channel limit. To me, it is an option, not a limit on a channel. I understand the convenience of it, but does not seem right.

Yes. I saw this semantic mismatch but for this prototype I took took the advantage of using this convenient approach. We could move the property to the appropriate group.

Good.

As I looked into the possible alternatives, there seems to be no simple alternative. This property is for the store and not for the server itself, so it should find its place somewhere in the store configuration option. However, we currently do not have a bag for store options but only store limits. So, I don't know whether you prefer to create a new bag or put it into Server's options or eventually to plan to rename the current StoreLimits to StoreOptions?

 

The reason for adding this force option was that this DeleteChannel method may be in the future invoked by some administrative service to support both cases conveniently, I passed this option to the method. Are you saying you would prefer to make this method always delete the channel and the caller should determine whether there are subscribers and decide to invoke this delete method (i.e., equivalent to force being true)or not (equivalent to force being false)?

Yes. I would think that DeleteChannel() should not have the force flag. The caller should decide if it is safe to call this or not.

 

OK

 

As mentioned above, I was hoping to get both CreateChannel and DeleteChannel etc to be accessible from some admin service. As symmetric to CreateChannel to the store API, I thought DeleteChannel would fit in there.

I am not arguing that DeleteChannel should be in the store interface. I think it should, but to me it makes a stronger case that this should be called by the server itself, not internally by the store.

 

I have to look into this stuff. I thought we could use the lock within the store to guard against such cases but I am probably missing something.

Store has its own lock, but as I have described above, this is not a problem of concurrent access per se, but a problem of order between events. In your approach, by the time the store decides to delete the channel, messages and or subscribers may be added to that channel. I understand why you did start the delete through a go routine (locking and such), but this is leaving room for races.

I used a goroutine because because the expireMsgs already owns the MsgStore lock but there is no unlocked version of MsgStore.Close. If we create an unlocked version of Close, the delete operation can be synchronously executed from that spot. But in either way, this doesn't eliminate the race condition. 

As mentioned above, I was thinking of minimizing the race conditions and for some edge cases, handle them in a recoverable way. Concretely, I was thinking of excluding those channels with subscriptions and unflushed messages from deletion and adding an extra waiting time before the deletion to minimize the race condition.

 

I also think using a specific timer for deleting a channel is preferable than using the message expiration timer which cannot handle the case where there is no message ever stored. When using another timer, we need to ensure its action doesn't interfere with the server's store flush operation and also the store's message expiration action.

I'll spend some time looking into these points.

When https://github.com/nats-io/nats-streaming-server/pull/196 is merged, it will be possible to send to the ioChannel control messages that will ensure that things are processed in order. The server could have a timer that simply put a control message in the channel and that control message could then be processed in the ioLoop.

You are welcome to experiment, and I may also get to this at one point. So many things to do ;-)

When using this approach, we either block the processing of IOLoop during the entire delete processing or mark the channel as being deleted and dispatch the delete operation and reject any store calls going into the channel. The former approach will likely cost some performance. If we go for the latter approach, we could have an equivalent behavior without introducing a control message by setting some special status to the channel to let the IOLoop know the channel is being deleted and hence reject any store requests to the channel.

What do you think?

Thanks.

regards, aki

Thanks,

Ivan.

[Ivan Kozlovic]

Ari,

 

I used a goroutine because because the expireMsgs already owns the MsgStore lock but there is no unlocked version of MsgStore.Close. If we create an unlocked version of Close, the delete operation can be synchronously executed from that spot.

Adding a MsgStore.close() version (without locking) is easily done, but that's not enough since the DeleteChannel is a method of FileStore, not FileMsgStore. There can be FileStore.lock -> FileMsgStore.lock but not the opposite. You would end-up with deadlocks due to lock inversion.

When using this approach, we either block the processing of IOLoop during the entire delete processing or mark the channel as being deleted and dispatch the delete operation and reject any store calls going into the channel.

The question I have for you is this: is the occurrence of a channel delete something that will happen often? If not, I don't think that it would costly to "interrupt" the ioLoop with a delete channel request. And as you said, the delete channel itself could be cheap in that removing synchronously from the FileStore's map, but have the heavy lifting of closing (and deleting files) be started from DeleteChannel in a go routine, not blocking the rest (there may still be a race there if the same channel is re-created quickly while removal of the directory is in progress...).

If deleting a channel occurs often, then it makes me think that it would almost be as if you were doing some request/reply with very ephemeral channels, in which case, streaming is not best suited and core NATS request/reply then fits the bill.

 

The former approach will likely cost some performance. If we go for the latter approach, we could have an equivalent behavior without introducing a control message by setting some special status to the channel to let the IOLoop know the channel is being deleted and hence reject any store requests to the channel.

Yes, it may be a mix of the two approaches. I still think that the server should decide if a channel need to be deleted. As I said, the first step of deleting the channel could be cheap (simply removing from the map). The rest, closing and deleting files may be more expensive and require the server to reject creation of such channel until the operation completes in the background. Things to think about.

Thanks Ari!

Ivan.

[Christian Cassar]

Hello Ivan, Aki,

Is the functionality to delete/garbage collect channels in place or is there a roadmap to get  the functionality?

Thanks

Christian

[Ivan Kozlovic]

Christian,

No, this is not in the official product (not sure if Aki did implement that in his fork). It is not currently in the roadmap but we would be interested in your use case and thoughts on how you would use such feature? For instance, if it would be a delete request, how would you want that request to be given to the server (note that there is no admin CLI - and no plan so far)? Or would it be just based on the channel being empty? (which can only happen if you use MaxAge otherwise, there would always be at least 1 message with any other channel limits).

Thanks!

Ivan.

[Christian Cassar]

Ivan,

thanks for the prompt reply.

Our use case would be from the latter category; i.e. where, for an
empty channel, the server side would take care of garbage collecting
resources e.g. the channel directory. The reason we are keen on this
happening is that we expect to have lots of relatively short lived
channels. The channels would be instantiated as part of
request/response pattern overlaid on top of NATS streaming pub/sub,
with each request to a well known topic eliciting one or more
responses from one or more parties on a channel associated with the
request.

Thanks
Christian

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "nats" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/natsio/f_8KRoVZi0Q/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> natsio+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ivan Kozlovic]

Christian,

This is probably why channel delete is not at the top of our list. We believe that request/reply and streaming don't go together (this is why there is no NATS Streaming API to perform request/reply like there is in NATS).

We believe that request/reply does not need persistence since the requestor knows if the replier got the message. If it does not get a reply (say it times out or gets an error), the requestor can send the request again, or decide to report failure. Using persistence for both request and replies doesn't make too much sense to us.

For instance you say that requests would be persisted to a given channel, and applications would then consume from this channel and send replies to another channel. Does it make sense for applications to re-process requests that have been already processed? For the replies, how does the requestor gets the reply intended for a given request? Again, does that make sense to re-process the reply?

Can't this be solved by using core NATS request/reply?

Thanks,

Ivan.

[Christian Cassar]

Ivan,

> We believe that request/reply does not need persistence since the requestor knows if the replier got the message.

In this request/response overlay, more than one entity could be
replying - so it is not quite request/reply between two parties.

Christian

[Ivan Kozlovic]

You can do that in NATS. The Request() API is just a convenience function that creates a subscription on an inbox and auto-unsubscribe with 1 reply, but you could create your own subscription, on any subject you want, set (or not) the auto-unsubscribe to the max number of response you want, and then simply publish a message with a reply subject. Repliers will send their response back to the subject like regular Request/Reply. Your requestor can then process the replies (either synchronously calling NextMsg() if you created the subscription synchronously, or through the message callback).

Ivan.

[Christian Cassar]

Right, much as we did with NATS Streaming, but without the reliable option.

C