Switching subscriptions between cold and hot streams

[David Dawson]

Before I start, I might be doing something ill conceived. This is always possible and if so, please tell me.

We're building a prototype event store using reactor.  

it accepts data in on one stream.

The data is persisted in a tape like format, or in memory as an array for demo.

Many subscribers can subscribe to the input stream, which by definition is hot and unbounded.

We can do replay of the history too, this has cold stream semantics, as it is bounded.

Both work, and back pressure works very nicely.  

The next bit is the rub.  

I'd quite like to accept a subscription that contains a query. This will then perform replay of the history (the cold bit), then switch the subscription to the input stream (the hot bit). This should happen by buffering up the hot data while the replay occurs, then pushing it in.

And, of course, we could do with preserving the back pressure signals the whole time to avoid saturating the client as it processes the stream.

I've had trouble figuring out the best approach to this using Reactor, and I'd love to know I'm not insane in trying this!

What do you think?

David

[Stephane Maldini]

I'm exactly writing more about that part (http://projectreactor.io/docs/reference/#streams-backpressure), but the answer is not going to be as straight.

What you details is what happens when overflow operations are used, or with the RingBufferProcessor from core. In the latter case already sent operations will be directly consumed and only a reduced demand will propagate upstream. In the former case these are operations specific to alter the demand, ignore or buffer overflow. The buffer overflow acts like a cache and in effect can be combined with our other Chronicle support (more on that later when I get proper doc up for that).

I'm more than happy to see you if you can swing by the office I'd love to hear more about your use case and help :)

--
You received this message because you are subscribed to the Google Groups "reactor-framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reactor-framew...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Stephane Maldini | Solutions Architect, CSO EMEA | London | Pivotal

W: pivotal.io | T: @smaldini