Recovering Events After Failure

[Nam Nguyen]

Hi all,

My team is creating a web service that we're calling the Event Dispatcher and it:

1. reads from the $all stream 
   
2. filters the events based on metadata and other information
   
3. sends it off to subscribers listening to specific event types 
1. These event types are not specific to just one stream, but from multiple streams
2. For example, $test_stream-123 and $test_stream-321 both contain TestEventTypeOne
   the subscriber would receive TestEventTypeOne from both $test_stream-123 and $test_stream-321.

Without going into too much detail, we've been told that for our situation it would be better if we did not create any projections and read directly from the $all stream. We want this service to have some mechanisms to catch-up/recover events that subscribers need/missed. For example, a subscriber wants a single/a range of TestEventTypeOne from a month ago. To do this, we would have to read a months worth of events to get potentially one single event. Is there a way around? Ideally, we would want to read from a category stream to get just the events for that specific event type, but we did not enable any projections.

[Steven Blair]

Nam,

So you need to connect to a stream and start from a specific number?

We have had a simialr issue in the past, and opted for this approach:

1. Create a projection to create some new streams (based on date)

2. When our customer needs the events from "January 2020" we simply start up a persistent subscription and deliver these events.

3. Since this uses a persistent susbcription, any events which fail to deliver will automatically be parked.

If you want to continue using $all, then I suspect you would have to have a persistent subscription for that stream, then somehow (maybe via  query?) the event number you need to start from.

[Bryan Reynolds]

I think you can create a one time projection use the API off of the $all stream.

My question is, can you create a projection starting from a specific date,  or a specific index number on the all stream? 

Can you create a projection that starts off the all stream that does not start from the beginning of the all stream?

[Yves Lorphelin]

How many events do you generate per unit of time ?
=> you could read from $all and filter quite rapidelly on EventType 

in the .net Client you can event filter server side:
(not sure if the HTTP api has this feature) 

https://eventstore.com/docs/dotnet-api/code/EventStore.ClientAPI.Filter.html


https://eventstore.com/docs/dotnet-api/code/EventStore.ClientAPI.IEventStoreConnection.html#EventStore_ClientAPI_IEventStoreConnection_FilteredReadAllEventsForwardAsync_EventStore_ClientAPI_Position_System_Int32_System_Boolean_EventStore_ClientAPI_Filter_EventStore_ClientAPI_SystemData_UserCredentials_