Akka-streams supervision: best practices

[Alexander Temerev]

Hi everyone,

Is there any example or document describing best practices for supervising and monitoring akka streams? E.g. restarting Websocket streams on disconnect, backing off on repeated failures, creating supervision groups (e.g. if some stream failed, I also want to fail other streams in the group and restart them at once), making supervision hierarchies, etc.

Streams are incredibly handy and well-typed, but I don't see yet how I can recreate these powerful supervision tools available in plain Akka, especially in dynamic flow graphs where streams can be attached and detached at runtime.

"Introduction to streams lifecycle" would be incredibly handy, if it exists somewhere.

Best,

Alexander

[Konrad Malawski]

Hi Alexander,

since you asked the same question on the issue tracker at the same time allow me to link those and let's continue the discussion here if you want;

https://github.com/akka/akka/issues/21192#issuecomment-298806620

Firstly: I don't think that's entirely true. Existing supervision indeed only works when stages "support it". One has to notice that it's not as simple as it may seem from a high level perspective, since there's buffers involved between stages so what would "restart a section", or what should it really mean etc.

However, there's many ways to add restarting or other behaviours.

Including, but not limited to:

• Retry - https://github.com/akka/akka-stream-contrib/blob/master/contrib/src/main/scala/akka/stream/contrib/Retry.scala
• recoverWith http://doc.akka.io/docs/akka/2.5.1/scala/stream/stages-overview.html#recoverWith
• recoverWithRetries http://doc.akka.io/docs/akka/2.5.1/scala/stream/stages-overview.html#recoverWithRetries
• mapError http://doc.akka.io/docs/akka/2.5.1/scala/stream/stages-overview.html#mapError
• and more stages, such as watchTermination, monitor etc.
• the existing supervision when it's enough
• or hosting within Actors and reacting to a termination with how you want.

Do bear in mind that Akka was, is and remains a toolkit. The tools are there, with killswitches, a spec how failure is propagated (reactive streams, which will be in JDK9)

Would we want to improve the current supervision scheme?
Yes, we'd like to. Are we right now going to work on it? No, we're focused on other areas right now that are more pressing.

What would be helpful now: What I think would be the best next step here, is to actually come up with patterns - be it as blogs, or writeups in issues or PRs that pre-package behaviour such as the Retry stage did, and once we have the real use cases we can think about it again, because a generic "supervision" is very hard (note the generic part in that phrase, specialized ones are simple), and very likely not what you'd actually want (it would do unexpected surprising things, we experimented with some restarting schemes).

I do agree though that all those patterns should be then collected and put to: http://doc.akka.io/docs/akka/2.5.1/scala/stream/stream-error.html We're right now working on a new docs and website page, so once that's done we'll focus on such content.

Hope this answers the question about status, next steps, and how you can help :-)

-- 
Konrad `ktoso` Malawski
Akka @ Lightbend

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.