Streaming query results with Slick?

[api...@gmail.com]

Support for the Reactive Streams API for streaming results from the database is claimed as a major new feature of Slick 3.0, however, documentation on Streaming seems to indicate that only single subscriber can be attached to the stream. So here you have ReactiveStreams, right? Back-pressure since subscriber has to be attached to the stream and signal completion and it is all async.

If this is correct I am wondering what is the point if you cannot act on ResultsSet rows async by more than one subscriber?

Here's from docs:

"Streaming

Collection-valued queries also support streaming results. In this case, the actual collection type is ignored and elements are streamed directly from the result set through a Reactive Streams Publisher, which can be processed and consumed by Akka Streams.

Execution of the Action does not start until a Subscriber is attached to the stream. Only a single Subscriber is supported, and any further attempts to subscribe again will fail. Stream elements are signaled as soon as they become available in the streaming part of the Action. The end of the stream is signaled only after the entire Action has completed. For example, when streaming inside a transaction and all elements have been delivered successfully, the stream can still fail afterwards if the transaction cannot be committed."

Thank you,

Andre Piwoni

[Stefan Zeiger]

On 2015-03-04 20:25, api...@gmail.com wrote:

Support for the Reactive Streams API for streaming results from the database is claimed as a major new feature of Slick 3.0, however, documentation on Streaming seems to indicate that only single subscriber can be attached to the stream. So here you have ReactiveStreams, right? Back-pressure since subscriber has to be attached to the stream and signal completion and it is all async.

If this is correct I am wondering what is the point if you cannot act on ResultsSet rows async by more than one subscriber?

Attaching the first Subscriber executes the DBIO action, so it can have side-effects. It does not make much sense to have different semantics for the first Subscriber (driving the execution) and for subsequent ones (receiving the same data). If you want to distribute a stream to multiple subscribers with a suitable back-pressure strategy (e.g. buffering or dropping), you can use Akka Streams.

It would make sense to execute the DBIO action again (independently) for a new Subscriber but the Reactive Streams spec does not currently allow it. This may change before Reactive Streams 1.0 / Slick 3.0 final releases: https://github.com/reactive-streams/reactive-streams-jvm/issues/224

--
Stefan Zeiger
Slick Tech Lead
Typesafe - Build Reactive Apps!
Twitter: @StefanZeiger

[api...@gmail.com]

Thank you for reply, Stefan.

It would be nice to distribute different elements of a stream like one based on ResultSet to many subscribers but I understand that it would be against Reactive Streams specs and like you said I could use DatabasePublisher with something like ActorSubscriber in the example here

Andre