Akka-remote docker<->docker works fine with netty.tcp, but fails with artery - ideas?

[johan.kjolhed...@gmail.com]

I'm pretty new to Akka and am currently trying out Akka's new Artery backend for akka-remoting (http://doc.akka.io/docs/akka/2.4/scala/remoting-artery.html) while also testing the regular netty.tcp transport (http://doc.akka.io/docs/akka/2.4/scala/remoting.html).

It looks fairly straight forward and I have two test applications running:

* A producer application which sends a message to the consumer every 3 seconds, and prints some stuff when the reply comes back

* A consumer application which prints a message when it receives one, and then sends back a reply to sender()

However I cannot get the communication to work when running both my applications with artery inside docker containers on the same host - it perfectly when I use netty.tcp though!.

Below are the results I achieved:

Results using artery
Producer  Consumer  Result
-------------------------
IntelliJ  Docker    ok
Docker    IntelliJ  ok
IntelliJ  IntelliJ  ok
Docker    Docker    fail (no message reaches the consumer, and the producer eventually prints handshake timeouts after 20 seconds/default timeout in akka-remote-artery - the same error message I get when not running any Consumer at all)

Results using netty.tcp

Producer  Consumer  Result
-------------------------
IntelliJ  Docker    ok
Docker    IntelliJ  ok
IntelliJ  IntelliJ  ok
Docker    Docker    ok

As you see above, I got all combinations to work except when using artery to communicate between two local docker containers.

The akka configurations for the failing artery case are as follows 

Consumer conf (prettified output of typesafe Config object logged):

{

    "akka": {

        "actor": {

            "provider": "remote"

        },

        "remote": {

            "artery": {

                "bind": {

                    "hostname": "0.0.0.0",

                    "port": 12345

                },

                "canonical": {

                    "hostname": "192.168.99.100",

                    "port": 12345

                },

                "enabled": "on"

            }

        }

    }

}

Producer conf (prettified output of typesafe Config object logged):

{

    "akka": {

        "actor": {

            "provider": "remote"

        },

        "remote": {

            "artery": {

                "bind": {

                    "hostname": "0.0.0.0",

                    "port": 12346

                },

                "canonical": {

                    "hostname": "192.168.99.100",

                    "port": 12346

                },

                "enabled": "on"

            }

        }

    }

}

192.168.99.100 is the IP of my docker machine executing both containers.

The equivalent netty.tcp config works fine (as seen in the results table earlier in this message)

However using artery I just get zero activity on the consumer, and eventually the producer starts logging errors:

[ERROR] [11/09/2016 16:24:17.201] [producer-system-akka.remote.default-remote-dispatcher-6] [akka.remote.artery.Association(akka://producer-system)] Outbound message stream to [akka://consume...@192.168.99.100:12345] failed. Restarting it. Handshake with [akka://consume...@192.168.99.100:12345] did not complete within 20000 ms (akka.remote.artery.OutboundHandshake$HandshakeTimeoutException) after about 20 seconds, indicating what I assume is a timeout.

When I use netty.tcp it works just fine between the containers.

What is also strange is that it works with artery when I move one of the applications outside of a docker container. Both 

When running each application outside docker (e.g. from within intelliJ during development) they can communicate with each other fine, but as soon as I try to make them talk to each other with artery from docker container to docker container, no messages get through to the consumer

Each application can be configured (cmd line args) to communicate with either netty.tcp or artery.

I've repeatedly gone over the configurations, code and startup arguments from scratch now so many times, and had colleagues look at it as well. We cannot figure out at all what could be wrong here :S.

The following are the docker startup commands. The command line arguments you see here are used to generate the conf you see above.

Consumer:

docker run --shm-size 1024M -p 12345:12345 -p 12345:12345/udp --rm akka-remote-eval-consumer:0.1.0-SNAPSHOT --transport tcp --host 192.168.99.100 --port 12345 --bind-host 0.0.0.0 --bind-port 12345

Producer:

docker run --shm-size 1024M -p 11001:11001/udp -p 12346:12346/udp --rm akka-remote-eval-producer:0.1.0-SNAPST --transport artery --host 192.168.99.100 --port 12346 --bind-host 0.0.0.0 --bind-port 12346 --consumer-host 192.168.99.100 --consumer-port 12345

Running this on a regular macbook pro with what I think are the latest updates, oracle jdk8, docker running with docker-machine.

Using akka-actor and akka-remote 2.4.12

Note: We are running an older docker version if that is of any importance: 1.10.2

Any ideas of what I could be doing wrong?

/ Johan

[Akka Team]

Hi Johan (good name!),

Artery/Aeron uses UDP, while the old remoting is TCP-based, did you configure your docker nodes with udp port mappings?

--

Johan

Akka Team

[ERROR] [11/09/2016 16:24:17.201] [producer-system-akka.remote.default-remote-dispatcher-6] [akka.remote.artery.Association(akka://producer-system)] Outbound message stream to [akka://consumer-system@192.168.99.100:12345] failed. Restarting it. Handshake with [akka://consumer-system@192.168.99.100:12345] did not complete within 20000 ms (akka.remote.artery.OutboundHandshake$HandshakeTimeoutException) after about 20 seconds, indicating what I assume is a timeout.

When I use netty.tcp it works just fine between the containers.

What is also strange is that it works with artery when I move one of the applications outside of a docker container. Both 

When running each application outside docker (e.g. from within intelliJ during development) they can communicate with each other fine, but as soon as I try to make them talk to each other with artery from docker container to docker container, no messages get through to the consumer

Each application can be configured (cmd line args) to communicate with either netty.tcp or artery.

I've repeatedly gone over the configurations, code and startup arguments from scratch now so many times, and had colleagues look at it as well. We cannot figure out at all what could be wrong here :S.

The following are the docker startup commands. The command line arguments you see here are used to generate the conf you see above.

Consumer:

docker run --shm-size 1024M -p 12345:12345 -p 12345:12345/udp --rm akka-remote-eval-consumer:0.1.0-SNAPSHOT --transport tcp --host 192.168.99.100 --port 12345 --bind-host 0.0.0.0 --bind-port 12345

Producer:

docker run --shm-size 1024M -p 11001:11001/udp -p 12346:12346/udp --rm akka-remote-eval-producer:0.1.0-SNAPST --transport artery --host 192.168.99.100 --port 12346 --bind-host 0.0.0.0 --bind-port 12346 --consumer-host 192.168.99.100 --consumer-port 12345

Running this on a regular macbook pro with what I think are the latest updates, oracle jdk8, docker running with docker-machine.

Using akka-actor and akka-remote 2.4.12

Note: We are running an older docker version if that is of any importance: 1.10.2

Any ideas of what I could be doing wrong?

/ Johan

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[johan.kjolhed...@gmail.com]

Yes - i Did. Please see my original post for details. You will see that I use the udp parameter for my docker port forwarding. Also please note that I've tested artery successfully in docker as long as one of the applications is running outside docker. However the problem occurs only when both producer and consumer run in containers. See the test results table and error log in my previous message for more details.

/Johan

[Akka Team]

Ah, sorry, didn't notice it in the command line.

Can you verify with some other tool that you can communicate using udp between the nodes? (using netcat for example)

Set logs to debug to see if there is anything helpful in there.

Additionally artery contains a "flight recorder" which you can enable using the setting akka.remote.artery.advanced.flight-recorder.enabled, it will write artery related events to a binary logfile, which you can then feed to akka.remote.artery.FlightRecorderDump in akka-remote to get human readable output, it may also contain some hints about what/why/how it is not working.

--

Johan

Akka Team

On Wed, Nov 9, 2016 at 7:14 PM, <johan.kjolhed...@gmail.com> wrote:

Yes - i Did. Please see my original post for details. You will see that I use the udp parameter for my docker port forwarding. Also please note that I've tested artery successfully in docker as long as one of the applications is running outside docker. However the problem occurs only when both producer and consumer run in containers. See the test results table and error log in my previous message for more details.

/Johan

[johan.kjolhed...@gmail.com]

Well, host <-> docker container artery works (both permutations). It's just docker <-> docker on the same host that doesn't. 

So udp forwarding seems to work. I will try to figure out a non-artery way of testing udp. Might end up writing a small scala echo server + client with udp (im terrible at linux/unix/mac command line tools :))

Also found this, but I'm confused as to what it says. Maybe it is related? https://github.com/docker/docker/issues/15127

/ Johan

Den onsdag 9 november 2016 kl. 19:29:44 UTC+1 skrev Akka Team:

Ah, sorry, didn't notice it in the command line.

Can you verify with some other tool that you can communicate using udp between the nodes? (using netcat for example)

Set logs to debug to see if there is anything helpful in there.

Additionally artery contains a "flight recorder" which you can enable using the setting akka.remote.artery.advanced.flight-recorder.enabled, it will write artery related events to a binary logfile, which you can then feed to akka.remote.artery.FlightRecorderDump in akka-remote to get human readable output, it may also contain some hints about what/why/how it is not working.

--

Johan

Akka Team

On Wed, Nov 9, 2016 at 7:14 PM, <johan.kjolhed...@gmail.com> wrote:

Yes - i Did. Please see my original post for details. You will see that I use the udp parameter for my docker port forwarding. Also please note that I've tested artery successfully in docker as long as one of the applications is running outside docker. However the problem occurs only when both producer and consumer run in containers. See the test results table and error log in my previous message for more details.

/Johan

--
>>>>>>>>>>      Read the docs: http://akka.io/docs/
>>>>>>>>>>      Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>>      Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

[Viktor Klang]

looks related

--
Cheers,
√

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.

[johan.kjolhed...@gmail.com]

New info.

I have now tested regular udp docker <-> docker - with the exact same application repurposed for raw java udp sockets. Using the same command line arguments.

It works fine with regular udp docker<->docker. It's just artery that doesn't want to work for me I suppose :(.

Could it be an artery bug?

/ Johan

[johan.kjolhed...@gmail.com]

Ok hm sorry.. this is a lot of new things to take in for me :). 

How do I change debug level for akka? Is there a typesafe/akka conf parameter?

I may try the FlightRecorder later if I have time. I think we're probably going to stick with netty.tcp for now though while artery has some time to mature. Seems nice though. 

/ Johan

Den onsdag 9 november 2016 kl. 19:29:44 UTC+1 skrev Akka Team:

Ah, sorry, didn't notice it in the command line.

Can you verify with some other tool that you can communicate using udp between the nodes? (using netcat for example)

Set logs to debug to see if there is anything helpful in there.

Additionally artery contains a "flight recorder" which you can enable using the setting akka.remote.artery.advanced.flight-recorder.enabled, it will write artery related events to a binary logfile, which you can then feed to akka.remote.artery.FlightRecorderDump in akka-remote to get human readable output, it may also contain some hints about what/why/how it is not working.

--

Johan

Akka Team

On Wed, Nov 9, 2016 at 7:14 PM, <johan.kjolhed...@gmail.com> wrote:

Yes - i Did. Please see my original post for details. You will see that I use the udp parameter for my docker port forwarding. Also please note that I've tested artery successfully in docker as long as one of the applications is running outside docker. However the problem occurs only when both producer and consumer run in containers. See the test results table and error log in my previous message for more details.

/Johan

--
>>>>>>>>>>      Read the docs: http://akka.io/docs/
>>>>>>>>>>      Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>>      Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

[johan.kjolhed...@gmail.com]

Even more new info :).

If I for fun launch the consumer as a regular udp socket I actually receive something. So the artery udp layer seems to work. However I suspect there could be some id/address mismatch in the handshake which makes akka-artery silently discard the info. Will see if I can get some debug info out of this..

Den onsdag 9 november 2016 kl. 19:29:44 UTC+1 skrev Akka Team:

Ah, sorry, didn't notice it in the command line.

Can you verify with some other tool that you can communicate using udp between the nodes? (using netcat for example)

Set logs to debug to see if there is anything helpful in there.

Additionally artery contains a "flight recorder" which you can enable using the setting akka.remote.artery.advanced.flight-recorder.enabled, it will write artery related events to a binary logfile, which you can then feed to akka.remote.artery.FlightRecorderDump in akka-remote to get human readable output, it may also contain some hints about what/why/how it is not working.

--

Johan

Akka Team

On Wed, Nov 9, 2016 at 7:14 PM, <johan.kjolhed...@gmail.com> wrote:

Yes - i Did. Please see my original post for details. You will see that I use the udp parameter for my docker port forwarding. Also please note that I've tested artery successfully in docker as long as one of the applications is running outside docker. However the problem occurs only when both producer and consumer run in containers. See the test results table and error log in my previous message for more details.

/Johan

--
>>>>>>>>>>      Read the docs: http://akka.io/docs/
>>>>>>>>>>      Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>>      Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

[johan.kjolhed...@gmail.com]

With akka.loglevel = DEBUG i only get two more lines on the consumer side:

First

[DEBUG] [11/09/2016 20:29:02.937] [aeron-client-conductor] [akka.remote.artery.ArteryTransport(akka://consumer-system)] onAvailableImage from 172.17.0.1:52515 session 2110811048

And then a few seconds later

[DEBUG] [11/09/2016 20:29:13.417] [aeron-client-conductor] [akka.remote.artery.ArteryTransport(akka://consumer-system)] onUnavailableImage from 172.17.0.1:52515 session 2110811048

Den onsdag 9 november 2016 kl. 19:29:44 UTC+1 skrev Akka Team:

Ah, sorry, didn't notice it in the command line.

Can you verify with some other tool that you can communicate using udp between the nodes? (using netcat for example)

Set logs to debug to see if there is anything helpful in there.

Additionally artery contains a "flight recorder" which you can enable using the setting akka.remote.artery.advanced.flight-recorder.enabled, it will write artery related events to a binary logfile, which you can then feed to akka.remote.artery.FlightRecorderDump in akka-remote to get human readable output, it may also contain some hints about what/why/how it is not working.

--

Johan

Akka Team

On Wed, Nov 9, 2016 at 7:14 PM, <johan.kjolhed...@gmail.com> wrote:

Yes - i Did. Please see my original post for details. You will see that I use the udp parameter for my docker port forwarding. Also please note that I've tested artery successfully in docker as long as one of the applications is running outside docker. However the problem occurs only when both producer and consumer run in containers. See the test results table and error log in my previous message for more details.

/Johan

--
>>>>>>>>>>      Read the docs: http://akka.io/docs/
>>>>>>>>>>      Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>>      Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

[johan.kjolhed...@gmail.com]

One additional piece of info that could be useful:

It works if I let the docker containers call each other by container IP instead of host IP+fowarding.

Den onsdag 9 november 2016 kl. 19:29:44 UTC+1 skrev Akka Team:

Ah, sorry, didn't notice it in the command line.

Can you verify with some other tool that you can communicate using udp between the nodes? (using netcat for example)

Set logs to debug to see if there is anything helpful in there.

Additionally artery contains a "flight recorder" which you can enable using the setting akka.remote.artery.advanced.flight-recorder.enabled, it will write artery related events to a binary logfile, which you can then feed to akka.remote.artery.FlightRecorderDump in akka-remote to get human readable output, it may also contain some hints about what/why/how it is not working.

--

Johan

Akka Team

On Wed, Nov 9, 2016 at 7:14 PM, <johan.kjolhed...@gmail.com> wrote:

Yes - i Did. Please see my original post for details. You will see that I use the udp parameter for my docker port forwarding. Also please note that I've tested artery successfully in docker as long as one of the applications is running outside docker. However the problem occurs only when both producer and consumer run in containers. See the test results table and error log in my previous message for more details.

/Johan

--
>>>>>>>>>>      Read the docs: http://akka.io/docs/
>>>>>>>>>>      Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>>      Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

[johan.kjolhed...@gmail.com]

Another piece to the puzzle: 
docker <-> docker does work routed through the host as long as the containers run on different hosts.

It's only when doing docker <-> docker routed through host on the same host that it doesn't work.

Seems like an issue in the artery or aeron protocol.

Den onsdag 9 november 2016 kl. 19:29:44 UTC+1 skrev Akka Team:

Ah, sorry, didn't notice it in the command line.

Can you verify with some other tool that you can communicate using udp between the nodes? (using netcat for example)

Set logs to debug to see if there is anything helpful in there.

Additionally artery contains a "flight recorder" which you can enable using the setting akka.remote.artery.advanced.flight-recorder.enabled, it will write artery related events to a binary logfile, which you can then feed to akka.remote.artery.FlightRecorderDump in akka-remote to get human readable output, it may also contain some hints about what/why/how it is not working.

--

Johan

Akka Team

On Wed, Nov 9, 2016 at 7:14 PM, <johan.kjolhed...@gmail.com> wrote:

Yes - i Did. Please see my original post for details. You will see that I use the udp parameter for my docker port forwarding. Also please note that I've tested artery successfully in docker as long as one of the applications is running outside docker. However the problem occurs only when both producer and consumer run in containers. See the test results table and error log in my previous message for more details.

/Johan

--
>>>>>>>>>>      Read the docs: http://akka.io/docs/
>>>>>>>>>>      Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>>      Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

[Vishnu Vardhan]

Hi Johan,

Was there a defect logged for this issue. I seem to be suffering from exactly the same symptoms. I am now checking the docker / docker on different hosts.

A related issue is the shutdown behavior if one of my producers is on bare metal, and the consumers is on a docker container. I am unable to get a clean shutdown. If both are running native (without containers ) shutdown is clean, but if not, then the cluster master does not shut. I have truncated and attached the two logs

1. Good log - non containerized (master and slave)

2. Bad log - cluster seed is not containerized, slave is containerized. Master does not exit (log for master only)

[Donovan Kim]

Is this resolved problem?