akka-stream: How to map aggregation (fold) and periodic flush to stream ("scanWithin")

[Elmar Weber]

Hello,

I'm currently testing to migrate a larger and complex actor system, one pattern we have is an aggregate and flush kind of operation. An actor receives a set of updates that is applied incrementally to a state. Every n seconds or x updates that state is flushed to the database.

The updates are easy: just a scan operation that aggregates the changes.

For the flush I have two (one not feasible) ideas so far:
- a groupedWithin that just takes the last element from the list (not possible when there are thousand of updates per second)
- a custom actor (seems like a lot of overhead)

Are there any easier / better solutions?

Thanks,
Elmar

[Endre Varga]

Hi Elmar,

You can try the "conflate" operator, and then use a ZipWith junction in a graph to wire it together with a tick source. You should read this section of the documentation for more (and important) details: http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-RC3/scala/stream-rate.html

-Endre

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Roland Kuhn]

Hi Endre,

one issue here is that conflate does not allow you to keep state for the “next round”, so scanWithin is indeed a different kind of operation; or did you mean to .scan(...).conflate(...).zip(TickSource)?

Regards,

Roland

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[Endre Varga]

On Wed, Jun 17, 2015 at 3:58 PM, Roland Kuhn <goo...@rkuhn.info> wrote:

Hi Endre,

one issue here is that conflate does not allow you to keep state for the “next round”,

True, but then a DetachedState is probably the best option. There is one example in the cookbook that just repeats the last element seen, it can be adapted to this case.

http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-RC3/scala/stream-cookbook.html

-Endre