Deleting events from NATS

[Richard Vogler]

Hi guys,

I'm currently evaluating to use NATS in one of our projects. We are having a Microservice Architecture and would like to use CQRS. As the central event store we are thinking of using NATS. Due to regulatory requirements, e.g. GDPR, there is the chance that we will sooner or later need to delete an event completely. Did somebody have experience with this topic or know how to do that?

Thanks in advance!

[R.I.Pienaar]

You cannot delete events from NATS other than at the "tail", ie. you can set rules for how long messages can be in the stream, how many messages can be in the stream or how big the stream can be. When any of those thresholds hit the oldest messages are removed.

[Vasiliy Tolstov]

чт, 24 янв. 2019 г. в 15:53, R.I.Pienaar <r...@devco.net>:

I think you can delete also events, but you must have event store that
supports deletion (like sql or write own) and remove only acked
messages in sequential order before first un-acked message arrives. so
as i understand this is mimics how stan works when mesages count above
limit

--
Vasiliy Tolstov,
e-mail: v.to...@selfip.ru

[R.I.Pienaar]

There isnt really "un-acked" messages as such, a message have an infinite number of ack states - every unique consumer/group has their own ack state.

This is very much in the Not User Serviceable side of things imo and digging into the raw data is looking for trouble.

If you need something like GDPR compliance do not use NATS as your CQRS store - I think its a bad store for that purpose anyway - use it as a feed/buffer/input into an appropriate store and then aging out messages is perfectly fine for GDQR, no-one says you have to be able to delete it instantly, windows of time is acceptable.

[Robert Engels]

And if the architecture is moving the “data” to another persistent store, the TTL on the messages in NATS can be very short.

IMO a lot of people misunderstand the proper usage of tools like Kafka/NATS. They are to be used as a persistent buffer/queue, that enables the easy use of parallel and/or partitioned consumers. If your architecture needs neither of those things, you probably shouldn’t be using it. It would s trivial to replace these with using the database directly but the performance and user experience will probably suffer.

> --
> You received this message because you are subscribed to the Google Groups "nats" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to natsio+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[iv...@synadia.com]

I could not agree more with what R.I. Pienaar wrote.

Even if you use SQL backing store, do not manually remove messages,

or if you do, don't be surprise if the system stops working.

As R.I. mentioned, there isn't ack'ed messages. This is not a message queue

as in JMS sense. Messages are in a log for the duration of the channel limits.

Even then, note that depending on the store implementation, messages may

still be present (in the file store for instance) even if they are not going to be

delivered. The FileStore implementation for instance split the message log

in a number of files (slices) and only when all messages in a slice have been

logically removed that the slice itself can be removed. So that would not

satisfy the GDPR requirements.

Ivan.

[Vasiliy Tolstov]

Does it possible to add limit for disk space for specific queue and for all queues?

For me realy hard to understand balance between item count,ttl and needed storage size.

For example i want to limit storage for 5gb. Filestore saves messages with base64. And payload size of nessage may be vary.

чт, 24 янв. 2019 г., 18:51 iv...@synadia.com:

--

[iv...@synadia.com]

Does it possible to add limit for disk space for specific queue and for all queues?

For me realy hard to understand balance between item count,ttl and needed storage size.

For example i want to limit storage for 5gb. Filestore saves messages with base64. And payload size of nessage may be vary.

I try to cover this here.

Since you can set limits for all channels or per channel, and you can limit the number of channels,

you can in some sense set a limit for storage as a whole. But some store implementations may have 

some differences that does not match exactly the store limits. For instance, FileStore, due to the use

of file slices may consume more space on disk than the channel limit.

 

[Chris]

You could also use the "lost key" encryption strategy, where you don't actually delete the events.

Instead you could encrypt the sensitive data (the whole the event of part of the data), but instead of deleting the event, you delete/loose the key, making it "unable" to decrypt the data.