ANNOUNCE: New Remoting Milestone 2

[Patrik Nordwall]

Dear hakkers,

We’re excited to announce that we have released the second development milestone of the new Akka Remoting, which has the code named Artery. It’s an early development preview and we encourage you to try it out and give us feedback, but it’s not intended for production usage yet.

The version number is 2.4-ARTERY-M2 with same artifacts as usual.

It is enabled with the following configuration:

akka.remote.artery {

 enabled = on

 # The hostname or ip clients should connect to.

 hostname = localhost

 # use 0 if you want a random available port

 port = 20200

}

The protocol part of the actor system address is artery (*), so you need to change previous akka.tcp to artery in for example configuration of cluster seed-nodes.

A summary of what is included in M2 compared to M1:

• performance improvements

• initial flight recorder to capture low and high frequency events in memory mapped files for debugging (also production issues) and testing

• make it possible to use efficient serialization with ByteBuffers directly

• various bug fixes, including issues with remote deployment

• test coverage by porting most of the old remoting tests

The full list of changes since the last milestone is available under the 2.4-ARTERY-M2 milestone on github for your reference.

We are using Aeron as the underlying transport and are implementing the layers on top using Akka Streams. You find more details in the design document.

The development branch is artery-dev, in case you want to take a look or contribute. Issues are labeled with t:remoting:artery.

(*) The protocol name is not final and may change before the new remoting infrastructure goes stable.

--

Patrik Nordwall
Akka Tech Lead
Lightbend -  Reactive apps on the JVM
Twitter: @patriknw

[Andrew Gaydenko]

Patrik,

In the Aeron-as-a-transport context (UDP) what is the akka-http's transport (TCP) planned future?

[Patrik Nordwall]

I'm not sure I understand the context of the question? Are you looking for Akka Remoting (actor messaging) over TCP or are you just curious in general of any plans for Akka Http transport, such as HTTP/2?

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Andrew Gaydenko]

Patrik, sorry I wasn't clear. If my guess is valid, at the moment akka-http is using TCP/Netty transport provided by akka-remote. As far as you are introducing new light-speed transport for remote, what are the plans wrt netty transport used for http? Is it going to just be kept intact? Or - I'm just wrong with my guess :)

[Guido Medina]

Hi Patrik,

Firstable many thanks for the great effort you guys are putting into this project.

I would like to test it, where can I find a mini-doc to make akka-cluster and akka-remote work with Artery?

Regards,

Guido.

[Roland Kuhn]

Andrew,

there seem to be a few misunderstandings here:

• HTTP by design requires TCP as its transport
• akka-http implements HTTP using akka-streams which in turn uses Akka IO for the TCP part—no Netty anywhere
• akka-remote has nothing to do with Akka IO, akka-stream, or akka-http

Regards,

Roland

[Andrew Gaydenko]

Roland, thanks  - all is absolutely clear now! :)

[Patrik Nordwall]

Hi Guido,

The mini-doc is in this announcement email.
1. Change the version number in your build.
2. Add the config listed
3. Replace akka.tcp with artery in any address paths that you use, e.g. seed-nodes

That should be all. Then it should just work. Let us know how it goes or if you have more questions.

/Patrik

--

[Guido Medina]

This stack trace is better as it shows the jar it comes from:

ERROR 15:03:06,933 ActorSystemImpl - Uncaught error from thread [DevCluster-akka.remote.default-remote-dispatcher-8] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled

java.lang.NoClassDefFoundError: org/jboss/netty/channel/ChannelHandler

at java.lang.Class.getDeclaredConstructors0(Native Method) ~[?:1.8.0_92]

at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671) ~[?:1.8.0_92]

at java.lang.Class.getConstructor0(Class.java:3075) ~[?:1.8.0_92]

at java.lang.Class.getDeclaredConstructor(Class.java:2178) ~[?:1.8.0_92]

at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(ReflectiveDynamicAccess.scala:30) ~[akka-actor_2.11-2.4-ARTERY-M2.jar:?]

at scala.util.Try$.apply(Try.scala:192) ~[scala-library-2.11.8.jar:?]

at akka.actor.ReflectiveDynamicAccess.createInstanceFor(ReflectiveDynamicAccess.scala:27) ~[akka-actor_2.11-2.4-ARTERY-M2.jar:?]

at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(ReflectiveDynamicAccess.scala:38) ~[akka-actor_2.11-2.4-ARTERY-M2.jar:?]

at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(ReflectiveDynamicAccess.scala:38) ~[akka-actor_2.11-2.4-ARTERY-M2.jar:?]

at scala.util.Success.flatMap(Try.scala:231) ~[scala-library-2.11.8.jar:?]

at akka.actor.ReflectiveDynamicAccess.createInstanceFor(ReflectiveDynamicAccess.scala:38) ~[akka-actor_2.11-2.4-ARTERY-M2.jar:?]

at akka.remote.EndpointManager$$anonfun$9.apply(Remoting.scala:813) ~[akka-remote_2.11-2.4-ARTERY-M2.jar:?]

at akka.remote.EndpointManager$$anonfun$9.apply(Remoting.scala:805) ~[akka-remote_2.11-2.4-ARTERY-M2.jar:?]

at scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:683) ~[scala-library-2.11.8.jar:?]

at scala.collection.Iterator$class.foreach(Iterator.scala:893) ~[scala-library-2.11.8.jar:?]

at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) ~[scala-library-2.11.8.jar:?]

at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) ~[scala-library-2.11.8.jar:?]

at scala.collection.AbstractIterable.foreach(Iterable.scala:54) ~[scala-library-2.11.8.jar:?]

at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:682) ~[scala-library-2.11.8.jar:?]

at akka.remote.EndpointManager.akka$remote$EndpointManager$$listens(Remoting.scala:805) ~[akka-remote_2.11-2.4-ARTERY-M2.jar:?]

at akka.remote.EndpointManager$$anonfun$receive$2.applyOrElse(Remoting.scala:537) ~[akka-remote_2.11-2.4-ARTERY-M2.jar:?]

at akka.actor.Actor$class.aroundReceive(Actor.scala:484) ~[akka-actor_2.11-2.4-ARTERY-M2.jar:?]

at akka.remote.EndpointManager.aroundReceive(Remoting.scala:433) ~[akka-remote_2.11-2.4-ARTERY-M2.jar:?]

at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) ~[akka-actor_2.11-2.4-ARTERY-M2.jar:?]

at akka.actor.ActorCell.invoke(ActorCell.scala:495) ~[akka-actor_2.11-2.4-ARTERY-M2.jar:?]

at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) ~[akka-actor_2.11-2.4-ARTERY-M2.jar:?]

at akka.dispatch.Mailbox.run(Mailbox.scala:224) ~[akka-actor_2.11-2.4-ARTERY-M2.jar:?]

at akka.dispatch.Mailbox.exec(Mailbox.scala:234) ~[akka-actor_2.11-2.4-ARTERY-M2.jar:?]

at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[scala-library-2.11.8.jar:?]

at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[scala-library-2.11.8.jar:?]

at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[scala-library-2.11.8.jar:?]

at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[scala-library-2.11.8.jar:?]

Caused by: java.lang.ClassNotFoundException: org.jboss.netty.channel.ChannelHandler

at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_92]

at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_92]

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[?:1.8.0_92]

at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_92]

... 32 more

[Guido Medina]

I got this exception because I'm explicitly excluding Netty 3 as dependency, should I leave it? but then, why is Netty 3 still in there?

[Patrik Nordwall]

Leave the netty dependency in. We have not made a clean separation of that yet.
/Patrik

[Guido Medina]

Hi Patrik,

I got it running and will test it during the week with low frequency messages (FX market data)

I see now two new threads which I assume are event loops threads:

• driver-conductor
• aeron-client-conductor

I'm guessing the load falls now into the remote-dispatcher?, do you have any comment on that?

I know it is new for you too but you must have some preliminary suggestion on that matter.

Cheers,

Guido.

[Patrik Nordwall]

That's Aeron threads. We launch the Aeron media driver embedded in the same jvm as the application by default. It's also possible to run it as a separate process, which can be an advantage if you have several jvms on the same machine.

You will also see a new thread named taskrunner, that is created by Akka. It's used for backoff poll and offer tasks. There is a config setting related to that, which you might be interested in for low latency testing. See idle-cpu-level https://github.com/akka/akka/blob/artery-dev/akka-remote/src/main/resources/reference.conf#L136

Read this when setting up the performance tests https://github.com/real-logic/Aeron/wiki/Performance-Testing

Looking forward to your results and what we can improve.

Cheers,
Patrik

[Guido Medina]

Hi Patrik,

Have you seen this before? It is happening on one PC but not in the other, I changed advanced.idle-cpu-level to 3, got that error, then removed that setting and recompiled but still getting the following exception:

ERROR 11:05:07,979 ArteryTransport(akka://DevCluster) - Aeron error: 8 observations from 2016-06-12 11:02:26.023+0100 to 2016-06-12 11:05:05.435+0100 for:
 .lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
 at io.aeron.driver.buffer.MappedRawLog.allocatePages(MappedRawLog.java:185)
 at io.aeron.driver.buffer.MappedRawLog.<init>(MappedRawLog.java:71)
 at io.aeron.driver.buffer.RawLogFactory.newInstance(RawLogFactory.java:121)
 at io.aeron.driver.buffer.RawLogFactory.newNetworkedImage(RawLogFactory.java:95)
 at io.aeron.driver.DriverConductor.newPublicationImageLog(DriverConductor.java:626)
 at io.aeron.driver.DriverConductor.onCreatePublicationImage(DriverConductor.java:228)
 at io.aeron.driver.cmd.CreatePublicationImageCmd.execute(CreatePublicationImageCmd.java:62)
 at io.aeron.driver.DriverConductor.onDriverConductorCmd(DriverConductor.java:839)
 at org.agrona.concurrent.OneToOneConcurrentArrayQueue.drain(OneToOneConcurrentArrayQueue.java:106)
 at io.aeron.driver.DriverConductor.doWork(DriverConductor.java:171)
 at org.agrona.concurrent.AgentRunner.run(AgentRunner.java:106)
 at java.lang.Thread.run(Thread.java:745)

My remote config:

  remote {

    log-remote-lifecycle-events = off

    artery {

      enabled = on

      port = 0

    }

    default-remote-dispatcher {

      type = Dispatcher

      executor = "fork-join-executor"

      fork-join-executor {

        parallelism-min = 8

        parallelism-factor = 1.0

        parallelism-max = 16

      }

    }

  }

Cheers,

Guido.

[Guido Medina]

I fixed this on Linux by deleting the folders created by aeron:

rm -Rf /dev/shm/aeron-*

Linux shared memory get's full after restarting micro-services few times, maybe this is some cleanup that needs to be done as part of ActorSystem shutdown?

Or maybe an aeron option I'm missing that should mark these temp folder to delete?

Guido.

[Guido Medina]

I read that if you start the driver you have to stop it in order to delete the sampling folder which for Linux is usually at /dev/shm (shared memory)

so, a bug?

[Patrik Nordwall]

I have seen this, and the reason is full disk as you found. We have not yet figured out why its not allways removing all files on shutdown. We try to remove on shutdown. I saw that it left directories that did not correspond to the directory of the embedded media driver. Need further investigation.

--

[Guido Medina]

Hi Patrik,

About performance, it seems to perform worse than the old akka remote, I usually get averages of 1ms and peaks between 10-20ms:

INFO  07:43:01,539 PriceDistributor - Price stats: avg 2.5896ms, max 75ms

INFO  07:43:01,540 PriceDistributor - Price stats: avg 2.6026ms, max 75ms

INFO  07:43:01,540 PriceDistributor - Price stats: avg 2.609ms, max 75ms

INFO  07:43:01,541 PriceDistributor - Price stats: avg 2.5917ms, max 63ms

There messages go from node A to B and from B to C (all in localhost), suffice to say that that's dev environment connected to test systems, not the real thing.

Also, messages are sent to a local round-robin router created out of N instances (N usually is CPUs) in a remote node.

These peaks might be an indication of too much garbage generated somewhere? but I haven't honestly measure that.

Hope that helps with the progress,

Guido.

[Endre Varga]

On Mon, Jun 13, 2016 at 8:53 AM, Guido Medina <oxy...@gmail.com> wrote:

Hi Patrik,

About performance, it seems to perform worse than the old akka remote, I usually get averages of 1ms and peaks between 10-20ms:

That is *very* surprising and contradicts our test results on different machines. Would be interesting if you could share a simplified version of the test you used so we can check the reason for the discrepancy.

 

INFO  07:43:01,539 PriceDistributor - Price stats: avg 2.5896ms, max 75ms

INFO  07:43:01,540 PriceDistributor - Price stats: avg 2.6026ms, max 75ms

INFO  07:43:01,540 PriceDistributor - Price stats: avg 2.609ms, max 75ms

INFO  07:43:01,541 PriceDistributor - Price stats: avg 2.5917ms, max 63ms

There messages go from node A to B and from B to C (all in localhost), suffice to say that that's dev environment connected to test systems, not the real thing.

Also, messages are sent to a local round-robin router created out of N instances (N usually is CPUs) in a remote node.

These peaks might be an indication of too much garbage generated somewhere? but I haven't honestly measure that.

We removed many-many allocations compared to the old remoting. I would be surprised if Artery produced more garbage than the old. Something fishy is going on here.

-Endre

 

Hope that helps with the progress,

Guido.

[Endre Varga]

On Sat, Jun 11, 2016 at 4:01 PM, Guido Medina <oxy...@gmail.com> wrote:

My first try failed, I thought Netty-3 wasn't being used so I excluded it as it is being pulled by akka-remote artery version, and I got the following exception:

That is because the old remoting is still included (hence the flag in the configuration to explicitly enable Artery) and that needs Netty.

 

java.lang.NoClassDefFoundError: org/jboss/netty/channel/ChannelHandler

at java.lang.Class.getDeclaredConstructors0(Native Method)

at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)

at java.lang.Class.getConstructor0(Class.java:3075)

at java.lang.Class.getDeclaredConstructor(Class.java:2178)

at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(ReflectiveDynamicAccess.scala:30)

at scala.util.Try$.apply(Try.scala:192)

at akka.actor.ReflectiveDynamicAccess.createInstanceFor(ReflectiveDynamicAccess.scala:27)

at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(ReflectiveDynamicAccess.scala:38)

at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(ReflectiveDynamicAccess.scala:38)

at scala.util.Success.flatMap(Try.scala:231)

at akka.actor.ReflectiveDynamicAccess.createInstanceFor(ReflectiveDynamicAccess.scala:38)

at akka.remote.EndpointManager$$anonfun$9.apply(Remoting.scala:813)

at akka.remote.EndpointManager$$anonfun$9.apply(Remoting.scala:805)

at scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:683)

at scala.collection.Iterator$class.foreach(Iterator.scala:893)

at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)

at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)

at scala.collection.AbstractIterable.foreach(Iterable.scala:54)

at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:682)

at akka.remote.EndpointManager.akka$remote$EndpointManager$$listens(Remoting.scala:805)

at akka.remote.EndpointManager$$anonfun$receive$2.applyOrElse(Remoting.scala:537)

at akka.actor.Actor$class.aroundReceive(Actor.scala:484)

at akka.remote.EndpointManager.aroundReceive(Remoting.scala:433)

at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)

at akka.actor.ActorCell.invoke(ActorCell.scala:495)

at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)

at akka.dispatch.Mailbox.run(Mailbox.scala:224)

at akka.dispatch.Mailbox.exec(Mailbox.scala:234)

at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Caused by: java.lang.ClassNotFoundException: org.jboss.netty.channel.ChannelHandler

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

... 32 more

So, are you guys still using part of Netty artery with akka-remote? Th

[Guido Medina]

We discussed this via Gitter and got some tips of what I probably did wrong, like too many threads, etc etc, scratch these results, I have to re-test again.

About Netty, yes, that was addressed also.

Thanks for the follow up.

Guido.

[Guido Medina]

Here is the timing with Netty in the same dev PC, I will soon re-test with Artery again with the new setup I got for Netty, will also change these times to nano:

  remote {

    log-remote-lifecycle-events = off

    netty.tcp {

      port = 0

      server-socket-worker-pool {

        pool-size-min = 4

        pool-size-factor = 1.0

        pool-size-max = 8

      }

      client-socket-worker-pool {

        pool-size-min = 4

        pool-size-factor = 1.0

        pool-size-max = 8

      }

    }

    default-remote-dispatcher {

      type = Dispatcher

      executor = "fork-join-executor"

      fork-join-executor {

        parallelism-min = 4

        parallelism-factor = 1

        parallelism-max = 8

      }

    }

  }

INFO  10:45:31,541 PriceDistributor - Price stats: avg 0.881ms, max 4ms
INFO  10:45:31,542 PriceDistributor - Price stats: avg 0.9ms, max 4ms
INFO  10:45:31,542 PriceDistributor - Price stats: avg 0.879ms, max 5ms
INFO  10:45:31,542 PriceDistributor - Price stats: avg 0.873ms, max 5ms
INFO  10:46:00,417 PriceDistributor - Price stats: avg 0.971ms, max 5ms
INFO  10:46:00,418 PriceDistributor - Price stats: avg 0.967ms, max 9ms
INFO  10:46:00,418 PriceDistributor - Price stats: avg 0.975ms, max 5ms
INFO  10:46:00,418 PriceDistributor - Price stats: avg 0.966ms, max 5ms

Note: My Netty versions is the un-released 3.10.6.Final (what 3.10 branch has now which allows Netty 3 to use default JVM CHM hence defaulting to CHMv8)