akka-http, websockets, and cluster sharding

[Eric Swenson]

I have a web service that up until my addition of server-side web socket support, worked fine.

I use akka-http to handle HTTP requests, and a cluster sharding region to route messages (sourced from the HTTP request) to actors that handle a particular "job" (the job id is the id used for sharding).  The actors are using akka persistence to maintain state.  This all works fine.

I recently updated the http handling to accept web socket connections from clients. In the initial ws:// request, the above-mentioned "job id" is included in the URI. The web socket Flow (modeled after the one found in this article:  http://blog.scalac.io/2015/07/30/websockets-server-with-akka-http.html) routes incoming web socket messages to the "correct" actor (based on the akka cluster sharding, by using the sharding region as the target actor).  This also works fine.  When the web socket is connected, a message is sent to the appropriate actor providing the actor source (to respond to).   My cluster sharded actor updates its state to make note of that actor, so it can send messages to the websocket.  That is what doesn't work.  When the cluster sharded actor tries to send a message to the actorRef it received during websocket connection, I get akka errors:

----

background log: info: route: ws-connect

background log: info: route: experimentInstanceId=187785cd-0276-4f04-a1a4-12e3d4487b81

background log: info: created new connection: org.genecloud.eim.WebSocketConnection@4b496e66

background log: info: [WARN] [10/30/2015 11:23:48.830] [ClusterSystem-akka.remote.default-remote-dispatcher-6] [akka.cluster.ClusterActorRefProvider] Error while resolving address [akka://HttpSystem] due to [No transport is loaded for protocol: [akka], available protocols: [akka.tcp]]

background log: info: [INFO] [10/30/2015 11:23:48.831] [ClusterSystem-akka.actor.default-dispatcher-20] [ExperimentInstance(akka://ClusterSystem)] Web socket connected for experiment instance 187785cd-0276-4f04-a1a4-12e3d4487b81

background log: info: [INFO] [10/30/2015 11:23:48.842] [ClusterSystem-akka.actor.default-dispatcher-15] [akka://HttpSystem/user/$a/flow-28-7-publisherSource-stageFactory-stageFactory-bypassRouter-flexiRoute-actorRefSource-actorRefSource] Message [org.genecloud.eim.WsMessage] from Actor[akka://ClusterSystem/system/sharding/ExperimentInstance/86/187785cd-0276-4f04-a1a4-12e3d4487b81#-126303652] to Actor[akka://HttpSystem/user/$a/flow-28-7-publisherSource-stageFactory-stageFactory-bypassRouter-flexiRoute-actorRefSource-actorRefSource#1354295246] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

background log: info: txt=[187785cd-0276-4f04-a1a4-12e3d4487b81] message #1

background log: info: WsIncomingMessage: experimentInstanceId=187785cd-0276-4f04-a1a4-12e3d4487b81

background log: info: txt=[187785cd-0276-4f04-a1a4-12e3d4487b81] message #2

background log: info: WsIncomingMessage: experimentInstanceId=187785cd-0276-4f04-a1a4-12e3d4487b81

---

The first 3 log messages show the HTTP route accepting the web socket connection with path /ws-connect/:JobId.  The WebSocketConnection object holds the flow.  The 5th log message shows the cluster sharded actor logging a message that the web socket has been connected.  The 4th log message is a mystery to me and probably going to cause the issue when the cluster sharded actor attempts to send a message back to the actor handling the web socket.  The code that sends the message looks like this:

    case WsConnected(experimentInstanceId: String, websocketActor: ActorRef) =>

      logger.info(s"Web socket connected for experiment instance $experimentInstanceId")

      persist(WebSocketConnectedEvent(websocketActor)) { evt =>

        state = state.updated(evt)

        websocketActor ! WsMessage("foo", "bar")

      }

The websocketActor that is supplied in the message came from the Flow, where the relevant portion is here in the WebSocketConnection class:

          // Materialized value of Actor

          val actorAsSource = builder.materializedValue.map(actor => WsConnected(experimentInstanceId, actor))

I'm not entirely sure where that actor comes from, but it appears that you can't send messages back to that actor from a potentially remote, clustered actor.  Is there some limitation here that I'm fighting with?

Note: I fully realize that I'm expecting a lot from akka here.  In the system I'm building, I'm intended to have multiple akka-http nodes behind a load balancer.  A user will therefore send http requests to one of the many akka-http nodes. But he will end up establishing a web socket connection with exactly one of these nodes (at a time).  Both http and web socket messages will get routed to an appropriate worker actor based on a sharded id.  That actor may be on some other node (not he one handling the http requests and not the one holding the server-side end of the web socket connection).  I'm expecting akka to be able to route any message from that actor back to the web socket.  Is that possible?

-- Eric

[Patrik Nordwall]

This is the interesting log entry:

ClusterSystem-akka.remote.default-remote-dispatcher-6] [akka.cluster.ClusterActorRefProvider] Error while resolving address [akka://HttpSystem] due to [No transport is loaded for protocol: [akka], available protocols: [akka.tcp]]

Looks like you have one ActorSystem named HttpSystem that is using LocalActorRefProvider. This can also happen if you share actor refs between two actor systems in the same jvm in a wrong way.

I think it would be easiest if you don't use a separate HttpSystem, and instead run those things in the ActorSystem named ClusterSystem.

/Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Eric Swenson]

Thank you, Patrick. That was, indeed, the problem. I had had earlier problems trying to get things working with a single actor system and that prompted me to move to two. I had no problems with the two actor systems until I added web sockets, and particularly when I tried to send messages from actors in the ClusterSystem to the HttpSystem actor Source for the web socket stream. That issue was caused by the local actor system. 

However, despite changing the HttpSystem to use RemoteActorRefProvider, and making sure that each actor used distinct ports, things still didn't work. I ended up going back to a single actor system. I ran back into a non-functioning aka-http -- the same issue that had prompted me to use two. It turned out that that issue was due to not waiting until the cluster actor system was "ready" before calling Http().bindAndHandle(...).

In any case, all is working now, with the single cluster actor system.

-- Eric

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/CwOJUcYzORM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.