Arbiter as an hidden member

[Amit Wankhede]

Hi

We are having an application connection issue. So, we are trying to lower down the application connection to the database. The application also makes connection with the arbiter.

We found a hidden member doesn't connect with the application.

So, it is possible to make arbiter as a hidden member

If yes, then will there be any impact like failover will not work, etc.

https://docs.mongodb.com/manual/core/replica-set-hidden-member/

[Kevin Adistambha]

Hi,

We are having an application connection issue. So, we are trying to lower down the application connection to the database

What is the nature of the connection issues? Can you elaborate?

So, it is possible to make arbiter as a hidden member

It is possible, but without knowing your actual issue, it’s hard to say what is the solution. Since arbiters do not carry data and do not participate in insert/update/find operations, I’d like to understand what the actual issue is.

If yes, then will there be any impact like failover will not work, etc.

It shouldn’t affect failovers.

Having said that, a better understanding of your issue could probably lead to a better solution. Could you post:

• Your MongoDB and your OS version
• The topology of your deployment (how many nodes, is it a sharded cluster, etc.), also the output of rs.conf()
• What connection issues are you facing

If this is a production environment, please also ensure that your setup follows the recommendations in the production notes.

Best regards,
Kevin

​

[ritesh mane]

Hi Kevin,

Thanks for your comments. Let me elaborate more on the application side issue that Amit had raised -

In our application we are connecting to multiple databases (replica-sets). Each of these replica-set has around 7 members (PSSSAAA - 4 data-bearing and 3 arbiters). Its not a mongo sharded cluster and we are on MongoDB 3.6 (Java driver 3.7.1).

In our Java application what we have observed is that Mongo Java driver creates multiple threads to connect to each and every member in the replica-set. Following is one of such mongo monitor thread from the application thread-dump -

"cluster-ClusterId{value='5ce4f51453dc0b321edb842f', description='null'}-192.169.67.28:27020" #70111 daemon prio=5 os_prio=0 tid=0x440029a00000 nid=0xe2d  [ JVM locked by VM (w/poll advisory bit) waiting on VM lock 'self_suspend_monitor for TID 3629', polling bits: safep ]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:226)

        at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:207)

        at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:154)

        - locked <0x0000200b5874fe50> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)

        at java.lang.Thread.run(Thread.java:748)

The strange thing is that mongo java driver also connects with the Arbiter members and above monitor threads are spawned for arbiter members as well. Since we have multiple database replica-sets and also 3 arbiters in each RS, the number of such arbiter threads are more in application. For better performance for our Java application we need to optimize the threads and we suppose that these threads connecting to arbiter can be avoided. We came across the 'hidden' replica-set member and found that the java driver does not make any connection with hidden member. We configured the arbiters as hidden and observed that the threads count had came down.

We assume that making the arbiter hidden should not affect the voting capability of the Arbiter and also help us reduce the unnecessary connections/threads with application. 

Thanks,

-Ritesh.

[ritesh mane]

Hi Kevin and team, 

Any thoughts over below behavior and making Arbiter as hidden member?

Thanks,

Ritesh.

[Kevin Adistambha]

Hi Ritesh,

Any thoughts over below behavior and making Arbiter as hidden member?

In my limited testing, I haven’t found any detrimental effect on using hidden arbiters using simple failure scenarios (e.g. killing a couple of nodes and ensuring that a new primary is still elected). However please continue testing with your own failure scenarios just in case you have a more complex scenario that my simple tests didn’t anticipate.

Each of these replica-set has around 7 members (PSSSAAA - 4 data-bearing and 3 arbiters)

Having said that, I’d like to understand why you’re using this setup. Do you have a specific deployment that requires the use of that many arbiters? I’m asking because arbiters are part of the voting nodes, and would therefore asked to confirm writes with majority write concern. Having that many in your replica set would result in non-intuitive majority write concern behaviour. For example, if I kill one node in the replica set (1 data-bearing node down out of 7 nodes) and do a majority write, it will result in this:

> db.test.insert({a:1}, {writeConcern: {w:'majority', wtimeout:3000}})
Inserted 1 record(s) in 3021ms
WriteResult({
  "nInserted": 1,
  "writeConcernError": {
    "code": 64,
    "codeName": "WriteConcernFailed",
    "errmsg": "waiting for replication timed out",
    "errInfo": {
      "wtimeout": true
    }
  }
})

So the document was inserted (Inserted 1 record(s) in 3021ms) and replicated to online data-bearing node. However the write returns that it failed the majority write (i.e. the write can be rolled back in some situation, so it’s not a safe write). I find this is completely non-intuitive, since the set still has the majority of data-bearing node online. Killing one data-bearing node should not result in a failed majority write. However the write should not be rolled back, since it should be replicated to the online nodes, which still makes up the majority of data-bearing nodes. Thus, the error message doesn’t reflect the reality, and I simply don’t know whether the write is safe or not. This is due to the large number of arbiters in the replica set.

Best regards,
Kevin

​