[Longish] Hazelcast and Docker (docker networking problems?)

[o haya]

Hi,

I just started working with Docker (on Linux) and one of the first things I was trying to do was to containerize one of our Springboot apps that uses Hazelcast clustering (docker container was built on a "build" machine), and also test "deploying" the Docker container to another machine, other than the build machine, using "docker save" and "docker load". 

The target machine initially did not have Docker installed, so I installed Docker on the target machine.

I was able to make a Dockerfile and build the container on the build machine.  I then did "docker save" to create the .tar file and SCP'ed the .tar file over to the target machine (which again, just had Docker installed), and use "docker load" to load the image.  

I then did a "docker run" command on the target machine.  

An instance (the "native" instance) of the same springboot app was already running on the target machine (SVC01), running just as a springboot app, and it was  joining the Hazelcast cluster.  The app listens on port 8885 for incoming requests.

This "native" instance was not containerized, so for the "docker run" command, I included "-p "18885:8885" on the "run" command line, but I wasn't aware of the Docker networking concepts, so I did NOT include any "--network" parameter on the "docker run" command line, and the containerized app was getting a bunch of "UnknownHostException" exceptions, and wasn't able to resolve the hostnames in the Hazelcast memberlist.

So, I did another test, where I changed the members in the memberlist to IP addresses and did the same "docker run" command, and this time, there were no "UnknownHostException".

Then, I had some discussions with one of our other team members, who is more familiar with docker and specifically with Docker networking, and he suggested doing the "docker run" but with an additional "--network host" parameter, so that the container would use Docker "Host" networking.

In order to do that, I had to shutdown the native springboot app, to avoid port conflict, and then I did "docker run" again, but this time, included "--port 8885:8885" and also "--network host" on the "docker run" command, and it WORKED, and I could see the containerized app joining the Hazelcast cluster.

So, now, I am circling back and want to do some testing about the Docker Networking with our containerized springboot app, because they (our project management) wants to potentially be able to run both the native springboot app and the dockerized springboot app on the same physical machine.

So, as part of that investiagation, the first thing I wanted to do was to go back to the Docker Bridge mode (i..e, do the "docker run" WITHOUT the "--network host" parameter, and with the "-p 18885:8885"), so I did that last night, and, surprisingly, this time, instead of getting any "UnknownHostExceptio" and even joined the Hazelcast cluster!!

But I am now wondering:  Why is the containerized app able to now start, when I do the same "docker run" command, without the "--network host" parameter (i.e., in Docker network "Host" mode), but NOT get the DNS resolution problems that it encountered previously? 

The only thing that I can think of is that maybe when I ran the "docker run" ONCE previously, with the "--network host" parameter, that somehow Docker is persisting the network mode to Host mode?   FYI, I have tried doing "docker stop" and "docker rm" on the SVC01 machine, to remove the container, multiple times.

For those here who are move familiar with Docker and Hazelcast, could you provide some insight into what might be happening?

Thanks!

Jim