One more comment on the streams API. It is really cool that you've thought about using mapConcat instead of flatMap to enable optimised merge operations. I just wanted to draw your attention to a clojure project that does super-fast merging of immutable Tree/Trie structures: https://github.com/JulesGosnell/seqspert
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
Bjorn asked if I felt any examples were missing, and sadly myoriginal request (that I've been going on about for years,sorry!) is indeed missing. It is the case of a fast producer anda slow consumer that is ideal for parallelisation.
I believe it may be possible to use the current 1.0-M2 to addressmy bugbear by using the Actor integration to write an actor thathas N instances behind a router, but it feels hacky to have touse an Actor at all. What is really missing is a Junction thatmultiplies the next Flow into N parallel parts that run onseparate threads.
In general, I felt that the documentation was missing anycommentary on concurrency and parallelisation. I was leftwondering what thread everything was happening on. Some initialquestions I have in this area:1. Is everything actually executed in the same thread? What aboutwhen you have a Junction?
2. Is it possible to be doing work to populate the Source's cachewhile work is being executed in a Flow or Sink?
It would be good to have a section in the documentation thatdiscusses this in more detail.And, very importantly, it would be good to have the feature ofbeing able to split a Flow into N parallel parts!
I recentlylearnt how to do this in ScalazStreams but I'm much rather beable to do it in Akka Streams as I find everything else about thearchitecture to be so much easier to understand (plus integrationwith Akka Actors is just tremendous).PS: I'm also very excited by Slick 3.0 which appears to be aiming toward Reactive Streams and, I assume, integration with Akka Streams. e.g. produce a Source[Entity] from a SELECT with pushback on extremely large result sets.Best regards,Sam (fommil)
Err, Endre, yes. And when merging Seq[T] in a mapConcat we would all benefit greatly from faster merging, Seq[Seq[T]] => Seq[T]
You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/SYmCYGQppKw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.
Err, Endre, yes. And when merging Seq[T] in a mapConcat we would all benefit greatly from faster merging, Seq[Seq[T]] => Seq[T]
There was some discussion about merging Seq in the docs. Anyway, it is a side point and I don't want to get derailed from my original questions so I will read your other responses later today.
There was some discussion about merging Seq in the docs.
It's the wording "with merge being the preferred strategy". From your email it is clear that merge is *not* the strategy used in Akka streams, so perhaps best to drop this sentence as it confuses more than clarifies. Instead, it would be instructive to note that a Source is returned and perhaps talk about the strategy that is used.
On Wednesday, 21 January 2015 08:21:15 UTC, drewhk wrote:>> I believe it may be possible to use the current 1.0-M2 to address>> my bugbear by using the Actor integration to write an actor that>> has N instances behind a router, but it feels hacky to have to>> use an Actor at all. What is really missing is a Junction that>> multiplies the next Flow into N parallel parts that run on>> separate threads.>I actually missed this when reading the docs... it's a gem buriedin a sea of examples! :-) Is there anything like this in the "topdown" style documentation?
A convenience method to set up this sort of thing is exactly whatI mean. I should imagine that fanning out a Flow forembarrasingly parallel processing is a common enough pattern thatone would want to be able to do this in a one liner. You notesomething about work in this area later on (quoted out of order):
> In the future we will allow users to add explicit markers where> the materializer needs to cut up chains/graphs into concurrent> entities.This sounds very good. Is there a ticket I can subscribe to forupdates? Is there a working name for the materializer so that Iknow what to look out for?
> Also, you can try mapAsync or mapAsyncUnordered for similar> tasks.It would be good to read some discussion about these that goesfurther than the API docs. Do they just create a Future andtherefore have no way to tell a fast producer to slow down?
Howdoes one achieve pushback from these? Pushback on evaluation ofthe result is essential, not on the ability to create/schedulethe futures. I would very like to see some documentation thatexplains where this has an advantage over plain 'ol Scala Streamwith a map{Future{...}}.
>> In general, I felt that the documentation was missing any>> commentary on concurrency and parallelisation. I was left>> wondering what thread everything was happening on.>> ... as the default all stream processing element is backed by> an actor ...The very fact that each component is backed by an Actor is newsto me. This wasn't at all obvious from the docs and actuallythe "integration with actors" section made me think that streamsmust be implemented completely differently if it needs anintegration layer!
Actually, the "actor integration" reallymeans "low level streaming actors", doesn't it?
I would stronglyrecommend making this a lot clearer as it helps to de-mystify theimplementation.Now knowing that each vertex in the graph is backed by an actor,I wonder if in "balancing work to fixed pools" the Balance isjust adding a router actor with a balance strategy?
Theconvenience method I suggested above could simply create a routerto multiple instances of the same actor with a parallelism bound.
I'm not entirely sure why one would need a Merge strategy forthat, although the option to preserve input order at the outputwould be good for some circumstances (which could apply pushbackin preference to growing the queue of out-of-order results).
In addition, this revelation helps me to understand theperformance considerations of using akka-streams. Knowing this,it would only be appropriate for something I would've consideredsuitable (from a performance point of view) for hand-crafted flowcontrol in akka before streams was released. The main advantageof akka-streams is therefore that it has dramatically lowered thebarrier of entry for writing Actor code with flow control.
Thanks for this explanation Endre, I hope to see even moreinnovation in the coming releases and improved documentation.
Best regards,Sam