Deletion of aggregates/snapshots marked as deleted

[Eugene Tenkaev]

Hello,

I use MongoEventStorageEngine, so as back I use Mongo for storing Aggregates, Sagas, Snapshots.

The questions is:

1. When I mark aggregate as deleted, where that information is stored? In Mongo or it just marked deleted again and again during replaying of events in hydration process of aggregate?

2. When Mongo event storage implementation decide to completely remove all events/snapshots of deleted Aggregate?

3. How handle bloating of database because of deleted users/accounts etc?

[Allard Buijze]

Hi Eugene,

1- the deleted mark of an aggreate is not stored in the event store. When reconstructing an aggregate from its events, and an event handler marks the aggregate as deleted, the repository will throw an exception (AggregateDeletedException, which extends AggregateNotFoundException).

2- The event store will never delete events of an aggregate. The fact that an aggregate was deleted, doesn't mean the things didn't happen anymore. Snapshots aren't deleted either, but they could...

3- You can choose to delete events after deleting an aggregate. But if you do so, do wonder about why you're using Event Sourcing in the first place. The power of Event Sourcing is in the fact that you know what happened in the past. Even the fact that a certain account existed in the past may contain value.

Cheers,

Allard

Op wo 25 okt. 2017 om 11:06 schreef Eugene Tenkaev <hro...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "Axon Framework Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to axonframewor...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Allard Buijze

CTO

E: allard...@axoniq.io

T: +31 6 34 73 99 89

AxonIQ Conference - 29th September - in Amsterdam

[Evgeniy Kochnev]

Hi Allard,

Almost 6 years ago I participated in some application which collects date (fuel level, gps coordinates ant etc) from fuel tanker trucks thru wireless network.
We never heard about CQRS at this time but architecture was very similar, we collect chain of event for each truck (honestly we don't save difference between events but actual values).
However we provide only 6 months data life cycle support to customers (capacity/performance issue at this time). Actually aggregate almost will be never closed (deleted) but data does.
So, just want to share some real business case where we still need actually to delete aggregates or their data.

Thanks,
Evgeny Kochnev

[Steven van Beelen]

Hi Evgeniy, others,

There are other ways to retain your event store and still disregard old events for your query side without the literal removal of any events.
Taking the very recent GDPR example, which states that somebody has the right to be forgotten data-wise, there is still a much cleaner approach imaginable then just roughly removing your data.

Like Allard says, it's not wrong per say to remove your historical events, but you do lose a lot of the nice features from Event Sourcing by doing so.

Cheers,

Steven

--

[Null Title]

Hi all,

I have a question, what happen when i mark deleted an aggregate then i recreate an aggregate with same identifier ?

Someone explain it for me, pls. Thanks so much.

[Eugene Tenkaev]

Hi Allard,

Thanks for the answers. Axon framework also provide great capabilities to build applications in DDD way and scale.

For example we didn't build ecommerce system so we wont store history about transactions and if admin deletes user in our case it delete forever and wont store all related events, because user can has jobs/subscriptions so all events for this aggregates will be stored even if they don't needed anymore. Yes it will be great to have rollback capabilities for aggregate but in our case only while it exists.

So in this case we don't whant to bloat database by history events for deleted user and all related staff that he done, so Is there any possibility to make even store completely delete aggregate and snapshots(like some flag in config)? 

It will be great if Axon provide such capabilities to make possible for developer to decide what to do with deleted aggregates, this will make Axon usable not only for ecommerce apps.

Please provide information about your plans to provide such option this very important for us to make decision.

[Steven van Beelen]

Hi Eugene,

I don't have the final say in this approach you're suggesting, that's something we'd have to discuss with the team here.
But to give you my personal point of view: I don't think if we'd add an issue for that, that we'd put it high on the backlog. 
Nonetheless you're free to suggest such an implementation by opening up an issue on our Github page.

Or even better, provide a solution by sending in a Pull Request.

To 'not bloat' your event store you could also think of implementing your own off loading strategy.
I'm saying off loading on purpose, as I still don't think removal is the best approach to take if you want to do Event Sourcing.

Hope this give you some insights and I guess Allard will share his opinion as well once he's got the time for it.

Ow and 'Null Title', I suggest against reusing the aggregate identifier of an aggregate you've marked deleted. 

Can't say I've tried this stuff, but I can only imagine the repositories in Axon wont like that. 
When Event Sourcing an Aggregate, the EventStore queries all the events for that aggregate based on the aggregate identifier.
And as pointed out, it's 'marked deleted', not actually deleted.

Hence if you'd introduce another aggregate with the same aggregate identifier, you'd have more events when Event Sourcing then you'd bargained for.
I even think Axon will complain, telling you that you cannot introduce a second aggregate with the given aggregate identifier, since it already exists.

Cheers,

Steven

[Eugene Tenkaev]

Hi,

Sorry for long term delay, I have create issue on GitHub for this https://github.com/AxonFramework/AxonFramework/issues/438

I understand that this goes out from concept of CQRS based systems(or maybe not), but look from my point of view: we create research application and it really easy to build and test it on Axon Framework... but managers ask simple question: why we need to store events for deleted user?

I want to say that is reasonable question, I don't have much time to take a look on how this can be implemented in code, so I'm here pushing you=). 

And I think this questions flying around not only for our project, many projects have such requirements.

So you have great opportunities to power-up CQRS=)

[Steven van Beelen]

Hi Eugene,

Starting a conversation about this is obviously a good thing; gets us to think what we can improve.

Hence thanks for adding an 'issue' on the GitHub page, I'll make sure we'll discuss it over here! :-)

Cheers,

Steven

[Eugene Tenkaev]

For those who interested in results I have wrote post in my medium blog https://medium.com/@Hronom/axon-gods-496525cfd2e

And create repository https://github.com/Hronom/axon-gods

It contains example of how you can use generic repositories(without event sourcing) with MongoDB, see axon-gods-without-event-sourcing module.

Aggregates in this repositories stored as regular entity, so they deleted after you mark as deleted. Also with this approach you can reuse id's.

Thanks Axon Framework guys for pointing me in the right direction=)