Stream API Fundamentals: Activator Template on Akka Http with Websockets

[Dagny T]

Newbie with Akka and Streams here!

I'm set on understanding the fundamentals of the latest Akka Streams API through standing up a prototype.  ;0)

I've looked to the Typesafe Activator Template on Akka Http with Websockets to do this; and have posted questions on the Disqus comments section; but apparently there isn't anyone monitoring that.

SO, I'm just going to re-post those here to see if anyone with relevant experience is able and willing to answer!  Would be grateful to be referred to a coherent Blog/GitHub repo which would answer most of these questions in any case!  

THANKS so much in advance!

1) TweetActor publishes to EventStream; and TweetPublisher receives this and forwards to WebSocket. What's the logic for separating the TweetActor and TweetPublisher? i.e. TweetActor is essentially acting as a Publisher, and Source of 'flow' for Tweets; HOWEVER, I'm confused about the distinction between, and when to use:
- context.system.eventStream.publish(tweet) within TweetActor 
VS
- Flow instantiated with TweetPublisher Source

2) What if I wanted to integrate with an (external) REAL Tweets Stream API from Twitter;
but I don't know how their API implements the Reactive Streams Protocol?

How would I use AkkaHttp Streams as a Client to GET tweets from that external Source; i.e. What are the latest recommended APIs I should be using to handle that?
- I'm seeing the Akka Template for AkkaSampleTwitterStream using this:
response.entity.dataBytes.runForeach
VS
- following THIS Akka Template; wouldn't one be constructing a Flow that MIRRORs what's in THIS OUTWARD flow; instead with an INWARD flow? i.e. Source.ignore; and 
class TweetSubscriber extends ActorSubscriber[Tweet]?

3) What is the default serialization format for AkkaStreams; and is it set to a reasonably efficient one? Otherwise, where can I override this; and what's the best-practice for setting this to?

4) How do local Stream buffers work roughly? 
- Where and how are the default and custom configuration property files for cache size in terms of number of typed Objects setup so as not to blow-up local memory? 
- Is each Stream dedicated to only one type of message; albeit permitting bi-directional flows?
- Where and how are the cache-eviction policies defaulted/configured in a property file? i.e. drop latest; or drop earliest?
- What is the distinction between when you use a configured Flow that's Materialized later;
OR, the context.system.eventStream (which is materialized by default with import of Akka Streams libraries?)

5) Handling versioning of incoming messages from external sources:

- Is there a good example for error-handling for serialization errors based on version
differences between the incoming external message; vs the assumed message structure processed internally?

6) Configuration of routes in externalized configuration files:

- Is there a coherent example which shows how to setup configuration properties files for mapping URLs for REST and WebSocket endpoints?
i.e. the nested code blocks within main.scala in this example could get hard to maintain in a larger application!

[Akka Team]

Hi Dagny,

That activator template is very dated, last update was last summer and we have put a huge amount of work into both Akka HTTP and Akka Streams since then. We'll make sure to remove it from the Lightbend web page.

The template was not written by me so I might not know the original reasons for the design decisions but I'll try to answer your questions none the less. In general I cannot say that the template is an especially good example of best practices as it is so dated, maybe it never was and just tries to cover as many concepts as possible, I can't say.

1) TweetActor publishes to EventStream; and TweetPublisher receives this and forwards to WebSocket. What's the logic for separating the TweetActor and TweetPublisher? i.e. TweetActor is essentially acting as a Publisher, and Source of 'flow' for Tweets; HOWEVER, I'm confused about the distinction between, and when to use:
- context.system.eventStream.publish(tweet) within TweetActor 
VS
- Flow instantiated with TweetPublisher Source

The eventStream is a pub-sub mechanism inside of the ActorSystem, it does not provide any back pressure and is not a Akka Streams component, it is always available to actors, so one reason could be to keep concepts separate, additionally it allows for many actors to subscribe for events dynamically while a stream would have to be materialized and then stay the same (this will probably change with a "Hub" stage we want to implement in the future). The tweet publisher is the glue from the eventStream to Akka Streams. As you say, there are other ways of doing this. For example passing a Source.queue to the TweetActor and let it publish into that instead of eventStream -> ActorPublisher. In general I think you should avoid the ActorPublisher and ActorSubscriber if you can, and if you can't then a custom GraphStage may anyways be a better solution as the APIs look today.

 

2) What if I wanted to integrate with an (external) REAL Tweets Stream API from Twitter;
but I don't know how their API implements the Reactive Streams Protocol?

There isn't (yet) a inter-node reactive streams protocol, it is only local. Luckily TCP already contains back-pressure so by simply using the Akka HTTP client you will be able to consume incoming elements in the pace your application can handle and backpressure will make sure the twitter API isn't sending you unbounded amounts of data.

 

How would I use AkkaHttp Streams as a Client to GET tweets from that external Source; i.e. What are the latest recommended APIs I should be using to handle that?
- I'm seeing the Akka Template for AkkaSampleTwitterStream using this:
response.entity.dataBytes.runForeach
VS
- following THIS Akka Template; wouldn't one be constructing a Flow that MIRRORs what's in THIS OUTWARD flow; instead with an INWARD flow? i.e. Source.ignore; and 
class TweetSubscriber extends ActorSubscriber[Tweet]?

Yes, it would probably mirror the sample in that you would have a Source[ByteString, _] from the HTTP response which you would have to split, and parse into a Source[Tweet, _] and then process further.

(There is however no such thing as a Source.ignore, only Sink.ignore)

 

3) What is the default serialization format for AkkaStreams; and is it set to a reasonably efficient one? Otherwise, where can I override this; and what's the best-practice for setting this to?

There is no default serialization for Akka Streams, its a library for local streams so nothing needs to be serialized. If you want to serialize elements in a stream you can use whatever serialization solution you like/need.

4) How do local Stream buffers work roughly? 
- Where and how are the default and custom configuration property files for cache size in terms of number of typed Objects setup so as not to blow-up local memory? 
- Is each Stream dedicated to only one type of message; albeit permitting bi-directional flows?
- Where and how are the cache-eviction policies defaulted/configured in a property file? i.e. drop latest; or drop earliest?
- What is the distinction between when you use a configured Flow that's Materialized later;
OR, the context.system.eventStream (which is materialized by default with import of Akka Streams libraries?)

I'd recommend you to read the docs and see if they answer your questions, they cover a lot of different aspects so you should be able to find most of the things you need to know in there:

http://doc.akka.io/docs/akka/2.4.8/scala/stream/index.html

The buffer stage will allow you to explicitly state buffer size and what to do upon overflow, no configuration defaults involved (as it wouldn't make sense to set default for buffers that are meant to be generic building blocks in systems we do not know anything about).

5) Handling versioning of incoming messages from external sources:

- Is there a good example for error-handling for serialization errors based on version
differences between the incoming external message; vs the assumed message structure processed internally?

This depends on the serialization format you are consuming. Some serialization formats (Protobuf for example) contains built in support for message versions. Some other does not but you can still handle support it by being careful with changes (json for example). Some does not support it at all (Java serialization for example)

 

6) Configuration of routes in externalized configuration files:

- Is there a coherent example which shows how to setup configuration properties files for mapping URLs for REST and WebSocket endpoints?
i.e. the nested code blocks within main.scala in this example could get hard to maintain in a larger application!

There is no such support built into Akka HTTP, you would have to implement this yourself, or use a third party library of some kind (but I can't say I know of any).

As Akka HTTP routes is just code you are free to modularize and split the code over as many classes and files you like, just like any other code. Additionally it provides type safety with compile time checks, something a property-file based route definition language would probably not.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Akka Team

Lightbend - Reactive apps on the JVM

Twitter: @akkateam