Subscriber System for Multi-Project App

[Sebastian Stehle]

Hi,

I am working on a application where we have a project system. The user creates a project, invites other people and makes changes to this project.

At the moment I am working with one subscriber per read model category, e.g. there is one subscriber for all things related to assets, like updating the read model, collecting statistics, cache invalidation and so on. Each subscriber stores the position of the last event and I have a UI where I can see status and reset an subscriber. 

Then the read models get cleared and repopulated. I need it when I change something in the read model or when I introduced a bug. In case the read model fails to handle an event I stop it automatically and show the exception in the UI. It only happened so far when there is a bug. I want to avoid inconsistencies. We also have plans to inform the user. Should be easy to implement. 

As event database use MongoDB or EventStore because some of my users have preferences.

It works fine and I like that I can replay the subscribers.

But it leads to some problems:

1. The subscribers have dependencies, e.g. the asset system needs information from the project data. Usually not a problem, but you have to know it when you repopulate the read models.

2. Subscribers affect each other. When one project does some heavy operations the event processing for other projects is slowed down too.

I want to introduce new features like export/import or cloning a project and then point #2 might become a problem.

Therefore I am looking for a new system, but I am afraid of creating thousands of subscribers. They also generate a lot of load when thousands of subscribers poll the event store regularly and I need a system to create them dynamically. I can also use partitioning but then I need internal queues, which could overflow.

I am looking for some advice. I know that my question is not very specific, sorry.

[@yreynhout]

If things don't have to be "that" real time, embrace the immutability and offload appended events to somewhere else - a bit like the eternal cache-ability of EventStore's atom feeds (I'm sure there are a few caveats in practice).