Combining Kafka, Akka Streams and (Persistent) Actors

[Richard Rodseth]

Short version: is it fair to say the traditional warnings against ask() hold less weight because we have back-pressure?

In the past I've built an Akka app (no ask() pattern except at the outer edge), and a tool that used Akka Streams (no visible actors except a monitor updated with alsoTo), but am now trying to combine the two concepts.

Imagine a service which consumes a Kafka topic, sends an email (service returns Future) and updates an aggregate (persistent Actor). I can imagine an infinite stream for this, with mapAsync generating back pressure from the email service, and the persistent actor as a Sink. Email retries could be handled at the Future level, though I'm still a little unclear on how error scenarios would be handled.

But what if the flow needs to thread through other (persistent) actors on the way to the Email service, perhaps to gather some information for the email?

Would it make sense to use an ask() here (perhaps in combination with per-request actors). 

Is it fair to say the traditional warnings against ask() hold less weight because we have back-pressure?

Could the command to update the the aggregate persistent actor also be issued with an ask() and acked, leading to a more functional style overall?

Advice or examples appreciated. 

[Justin du coeur]

On Thu, Oct 20, 2016 at 5:55 PM, Richard Rodseth <rrod...@gmail.com> wrote:

Short version: is it fair to say the traditional warnings against ask() hold less weight because we have back-pressure?

Well, keep in mind that at least some of the usual warnings against ask() have nothing obviously to do with that sort of thing.  The biggest traditional problem with ask() is really a warning about using Future -- it's always dangerous to map/flatMap on a Future inside of an Actor, because the Future will generally execute in parallel rather than properly synchronized in receive; as a result, Future messes up the pseudo-single-threading invariant of Actors.  And since a for comprehension of ask() is simply mapping over Futures, it's automatically a bit risky.  I don't see how back-pressure helps with that.

I haven't thought about how it interacts with streams, but fixing the ask() problem is the whole point of the Requester library. You might want to give that a look, and see if it helps with your needs...

[Richard Rodseth]

I've had no problems using Futures within actors and piping the result to the actor, after mapping. I have had problems when an actor gets flooded and you get timeouts because you're simply starting up too many Futures. That, I think, is where back-pressure changes the landscape.

Thanks for the Requester link.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Richard Rodseth]

Isn't your comment about "messes up the pseudo-single-threading invariant of Actors" more about not closing over mutable state? In any case, you can't avoid Futures if you're using Slick or HTTP clients, for example.

[Justin du coeur]

On Fri, Oct 21, 2016 at 10:32 AM, Richard Rodseth <rrod...@gmail.com> wrote:

I've had no problems using Futures within actors and piping the result to the actor, after mapping. I have had problems when an actor gets flooded and you get timeouts because you're simply starting up too many Futures. That, I think, is where back-pressure changes the landscape.

Ah -- okay, that's not the *traditional* problem with using ask() within Actors (which, as you say, is somewhat about closing over state, although it's even more about allowing code to access the Actor's state outside of receive), but I can see how it could become an issue.

> Isn't your comment about "messes up the pseudo-single-threading invariant of Actors" more about not closing over mutable state? In any case, you can't avoid Futures if you're using Slick or HTTP clients, for example.

Correct, but you can corral those Futures to make them less risky -- that's what Requester does.  (At least in Plain Old Akka Actors -- like I said, I haven't done enough with streams to know how they would interact in that environment.)

[Richard Rodseth]

Anyone else? Suppose I need a stage that just looks up something that is contained in a (persistent) actor.

Is it good practice to use mapAsync with an ask() to that actor?

Or is there some other stage that would let me use the actor to transform one stream element to another without sacrificing backpressure.

This SO question advises against making an actor a Processor.

http://stackoverflow.com/questions/31272267/creating-an-actorpublisher-and-actorsubscriber-with-same-actor

[Richard Rodseth]

I should add that the ask() I would be inserting would actually be to the ShardRegion for a shared, persistent actor.

[Patrik Nordwall]

ask is fine as long as you don't close over something that is not thread-safe in callbacks of the Future. It should be low risk of mistake in mapAsync since you typically only return the Future (after casting/mapTo to the right type).

I updated the Streams documentation for integration with actors the other day:

http://doc.akka.io/docs/akka/snapshot/scala/stream/stream-integrations.html

http://doc.akka.io/docs/akka/snapshot/java/stream/stream-integrations.html

Regards,

Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall
Akka Tech Lead
Lightbend -  Reactive apps on the JVM
Twitter: @patriknw

[Richard Rodseth]

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.