Recovering events in v1

[machina3k]

Hi guys,

A while back you mentioned work was being done on building in logic for reconnecting and recovering missed events (queuing new events in the meantime until the recovered events have caught up)...did that make it into v1? I've had a look around the API but can't see anything has changed around that area.

Cheers

[Greg Young]

With atompub it is built into the atom pub protocol.

For top I believe that is included in the bits with common domain integration.

Cheers,

Greg

[James Nugent]

Hi,

We haven't build the subscribe+catchup logic into the API directly (there are too many variations on what people might want to make it feasible to easily pick one and satisfy everyone). However, there are a number of new things which make implementing it relatively straightfoward.

The important thing if you want "at-least-once" or "exactly-once" guarantees for rebuilding persistent view models is that you store your last processed position transactionally with whatever work you're doing as a result of the event. You subscribe using something like this:

connection.SubscribeToAll(false, EventAppeared, SubscriptionDropped);

The subscription method returns an acknolwedgement containing a position from which you are guaranteed to see events over the subscription (note it doesn't guarantee you WON'T see earlier ones, but this is a straightforward position comparison to mitigate). EventAppeared is an Action<ResolvedEvent>. ResolvedEvent is a new type introduced during our client API changes recently. As you process events appearing over the subscription, you can find the logical position in the transaction file in the OriginalPosition field. A position is a tuple of prepare and commit position, both of which are longs, represented by the Position type.

When your subscriber restarts, you can do the following:

1. Subscribe to all, enqueing events as they appear. Note the position contained in acknowledgement

2. Read from the last seen position until you reach the end of the stream OR the position contained in the subscription ACK, processing each event as necessary.

3. Start processing from the queue of events appearing over the subscription, discarding any which are earlier than you want.

The same basic process applies if you want to process events from an individual stream (using SubscribeToStream) instead of the all stream.

We'll post some sample code for this shortly, but hopefully this description helps?

Thanks,

James

[machina3k]

Hi James,

Thanks for that, a lot of help. I already built something along these lines a while back but it does look like it's going to be a bit easier now. One thing I did notice, which prompted me to ask about a pre-canned version of this in the first place, was that I started to get overflows if I requested the entire event history (to rebuild a view). Now, there weren't a huge number of events in the store, probably ~1000, so it seemed a bit odd  (not large events either). Haven't had chance to reproduce but I assume the recovering missed events part of the reconnection logic needs to do them in sensible batches (part 2 in your description). 1000 seems a bit small though, what do you expect a 'good' size to be for a batch without it overflowing?

Cheers

[machina3k]

Haha, I've just seen in your gist the ReadPageSize is 500...so I guess I've got my answer