MongoDB always binding to public ip

[ved]

Hello all, 

I'm having a strange issue (that admittedly may or not be related with mongodb) which I'll try to break down as much as I can below.

If anybody needs any more info please just ask.

My goal is to have mongodb bind on localhost and a vpn inteface that uses zeroconf. The issue is that mongod seems to always also bind to the public ip.

Setup:

Debian 9.5 (fully updated, network managed with systemd)

MongoDB 4.0.3 (official from repo.mongodb.org with fully default config except for changing bindIp)

Network:

lo -> localhost

ens3 -> public interface with public ip assigned by the hosting company

vpn -> Tinc vpn interface with zeroconf ip (169.254.x.y) assigned by systemd-networkd and that gets resolved by systemd-resolved as hostname.local 

All of the above is an updated setup (from scratch) of an older machine that was doing the exact same function but using debian 8, mongo 3.4, ifupdown scripts (instead of systemd-networkd) and Avahi (instead of systemd-resolved)

The old machine has the correct behavior so I'm assuming that the issue is with the updated network setup or mongodb.

The issue:

Below are the different behaviors according to what I set on net.bindIp in /etc/mongod.conf

bindIp: 127.0.0.1

Works as expected

bindIp: 127.0.0.1,169.254.x.y

This works correctly and binds only to those ips but it's not useful because zeroconf is a dynamic ip

bindIp: 127.0.0.1,hostname.local

Binds to localhost, the vpn ip (169.254.x.y) but also binds to my public ip which is the main issue here.

This is what I need and how it's setup and running correctly on the previous machine.

bindIp: hostname.local

Binds to the vpn ip (169.254.x.y) and again also binds on public ip.

As you can see, it appears as whenever I use hostname.local, mongo also binds to the public ip. 

From the console running "ping hostname.local" correctly resolves to the zeroconf 169.254 ip. Setting the assigned zeroconf ip directly on bindIp also seems to work and doesn't bind to the public ip, although it's not what I need.

I've tried increasing mongod's log verbosity to try and detect if anything was being outputted in relation to the network but couldn't find anything helpful.

So, although this issue is being manifested on mongo, it's still possible that it's some network configuration issue that's the real culprit.

Nevertheless, if anybody has any ideas on why this may be happening or any kind of suggestions about what to try I'd be greatly appreciated.

Cheers.

[Kevin Adistambha]

Hi Ved,

As you can see, it appears as whenever I use hostname.local, mongo also binds to the public ip.

Have you found out what’s causing your network issues?

I have tried to reproduce the issue you’re seeing but have been unsuccessful so far. I ran a basic mongod as:

mongod --port 27017 --bind_ip 127.0.0.1,xyz.local

where xyz.local is the name of my local machine, bound to a local address. The output of netstat shows:

$ netstat -an | grep 27017
tcp4       0      0  10.x.x.xxx.27017       *.*                    LISTEN
tcp4       0      0  127.0.0.1.27017        *.*                    LISTEN

where I can see that mongod binds to the exact two IPs I have provided (xyz.local maps to 10.x.x.xxx).

If you’re still having this issue, could you please post:

• How did you install MongoDB
• The output of db.serverCmdLineOpts()
• The content of your config files
• The output of your /etc/hosts file

Best regards,

Kevin

​

[ved]

Hi Kevin,

Thanks for replying. And no, I still haven't figured out what's the issue exactly.

In the meantime I've closed mongo's port on the public inteface at the firewall level as a kind of workaround for it binding to the public ip.

To reply to your questions:

I installed mongodb through mongodb.org's debian repositories (repo.mongodb.org) using "apt-get install mongodb-org"

Output of db.serverCmdLineOpts():

db.serverCmdLineOpts()
{
        "argv" : [
                "/usr/bin/mongod",
                "--config",
                "/etc/mongod.conf"
        ],
        "parsed" : {
                "config" : "/etc/mongod.conf",
                "net" : {
                        "bindIp" : "127.0.0.1,app.local",
                        "port" : 27017
                },
                "processManagement" : {
                        "timeZoneInfo" : "/usr/share/zoneinfo"
                },
                "storage" : {
                        "dbPath" : "/var/lib/mongodb",
                        "journal" : {
                                "enabled" : true
                        }
                },
                "systemLog" : {
                        "destination" : "file",
                        "logAppend" : true,
                        "path" : "/var/log/mongodb/mongod.log"
                }
        },
        "ok" : 1
}

The content of /etc/mongod.conf:

# mongod.conf


# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/


# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:


# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log


# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1,app.local




# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo


#security:


#operationProfiling:


#replication:


#sharding:


## Enterprise-Only Options:


#auditLog:


#snmp:

Output of /etc/hosts: (masked ip and domain name, but format is as follows)

127.0.0.1       localhost
a.b.c.d     app.mydomain.com   app

With all the above configuration I get the following from netstat: (public ip masked as a.b.c.d)

tcp        0      0 a.b.c.d:27017           0.0.0.0:*               LISTEN
tcp        0      0 169.254.69.130:27017    0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN

Running mongod from the command line as per your example yields the exact same results on netstat.

Any hints or suggestions on what else to try will be greatly appreciated.

Cheers and thanks again.

​

[Kevin Adistambha]

Hi,

First of all please note that from what you described, this is a network setup issue and not a MongoDB issue.

Having said that, I would presume that the unexpected binding you see is due to the app.local name binding to the IP address you did not expect it to. You can check if this is the case by trying to ping app.local from itself, and see what IP address it resolves to.

Since a database server is primarily a static resource, it’s best to assign a static IP address to important machines in your network (e.g. the machine running mongod), and instruct mongod to bind to that static address. This way, your application can just use the static address, aliased with a readable name in their own /etc/hosts if so desired.

Another point for using a static IP is that MongoDB does not auto-refresh the IP it’s binding to. So if for some reason the IP it binds to changes (since you’re binding it to the name), then MongoDB would not be aware of this and as a result needs to be restarted.

For the best response regarding network issues, I would recommend you to ask a question in a network-related forum such as ServerFault.

Best regards,
Kevin

​

[ved]

Hi again Kevin,

First of all please note that from what you described, this is a network setup issue and not a MongoDB issue. 

Having said that, I would presume that the unexpected binding you see is due to the app.local name binding to the IP address you did not expect it to. You can check if this is the case by trying to ping app.local from itself, and see what IP address it resolves to.

.local resolution is achieved with multicast dns. There is no wrong binding or hardcoding of the ".local" part of the hostname anywhere. I'm assuming mongodb isn't doing the exact equivalent to a "ping app.local" internally because running "ping app.local" on the local machine (or any other machine on that vpn) resolves to the correct ips and never the public interface ip.

Also, even with verbosity at maximum, mongodb barely logs anything network related which does not help. Knowing how exactly mongodb resolves it's hostnames would be greatly helpful since it appears as it's bypassing the local nsswitch.conf file which is what I assume would have it correctly resolve the .local hostnames.

 

Since a database server is primarily a static resource, it’s best to assign a static IP address to important machines in your network (e.g. the machine running mongod), and instruct mongod to bind to that static address. This way, your application can just use the static address, aliased with a readable name in their own /etc/hosts if so desired.

Another point for using a static IP is that MongoDB does not auto-refresh the IP it’s binding to. So if for some reason the IP it binds to changes (since you’re binding it to the name), then MongoDB would not be aware of this and as a result needs to be restarted.

This is always a good tip for people starting up or building apps in 1998. But surely you can't be suggesting we hardcode ip's on N client nodes just because mongod is resolving a name differently that all existing net tools on an unix system?

 

For the best response regarding network issues, I would recommend you to ask a question in a network-related forum such as ServerFault.

Thanks for trying and for the suggestion.

​