Frequent Error:- com.mongodb.MongoException$Network: can't call something

[Avise Sudhakar Rao]

Hi Team,

We are frequently getting below error (please suggest how to resolve this issue):-

Reason:

Caused by: com.mongodb.MongoException$Network: can't call something : mongo-perf-db1/172.25.89.119:27017/performance_data

        at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:295)

        at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:257)

        at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:310)

        at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:295)

        at com.mongodb.DBCollection.findOne(DBCollection.java:727)

        at com.mongodb.DBCollection.findOne(DBCollection.java:669)

        at org.springframework.data.mongodb.core.MongoTemplate$FindOneCallback.doInCollection(MongoTemplate.java:1846)

        at org.springframework.data.mongodb.core.MongoTemplate$FindOneCallback.doInCollection(MongoTemplate.java:1831)

        at org.springframework.data.mongodb.core.MongoTemplate.executeFindOneInternal(MongoTemplate.java:1617)

        ... 41 more

 

ERROR nioEventLoopGroup-7-20 com.netenrich.ra.netty.handler.NSGMessageHandler - Transaction rolled back because it has been marked as rollback-only

org.springframework.transaction.UnexpectedRollbackException: Transaction rolled back because it has been marked as rollback-only

        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:718)

        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:148)

        at com.netenrich.ra.proxy.events.ProxyEvent$ExecuteEventTransaction.execute(Unknown Source)

        at com.netenrich.ra.netty.handler.NSGMessageHandler.channelRead(Unknown Source)

        at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)

        at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)

        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:153)

        at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)

        at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)

        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:253)

        at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)

        at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)

        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785)

        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:132)

        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:485)

        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:452)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:346)

        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)

        at java.lang.Thread.run(Thread.java:744)

1) Current mongo-java-driver.version is 2.10.1 and spring-data-mongodb.version is 1.3.1.RELEASE

2) below is the ulimit where we can check open file limit (do we still need to increase more than 65536? )

root@mdb504:/home/neadmin# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 458890
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 458890
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Please provide your suggestions.

Thank you,
Sudhakar

[Avise Sudhakar Rao]

Our spring configuration:-

Spring-mongo configuration

                  <mongo:mongo id="mongo" host="172.25.89.119" port="27017">

                <mongo:options connections-per-host="100"

                                                threads-allowed-to-block-for-connection-multiplier="5"

                                                connect-timeout="10000" max-wait-time="120000" auto-connect-retry="true"

                                                socket-keep-alive="true" socket-timeout="15000" slave-ok="true"

                                                write-number="1" write-timeout="0" write-fsync="true" />

                </mongo:mongo>

Please suggest if there is any changes or additional options need to include.

Thanks,
Sudhakar

[Rod Adams]

Greetings Sudhakar.

Network Exceptions from MongoDB can mean any number of things, and can be used as a general "Something Bad Happened" alert, and unfortunately, I do not have enough information yet to offer any suggestions.

Thus, could you please answer the following questions, which should help narrow down what it happening?

1) How frequent is "frequent"? Every second? Every Minute? Roughly every 100 queries? Only under heavy load?

2) Is there an additional error in your client log preceding the "com.mongodb.MongoException$Network" exception?

3) Can you look in the log file for the mongod server, and look for an error there?

4) What kind of network do you have between client and MongoDB? It's possible that your network connections are getting corrupted. Note: in this event, my suggestion will be to either improve your network, or capture the exceptions and retry the query.

As to your question, a limit of 65536 open files is typically enough, and I've not yet seen any evidence of you needing to raise it.

Thanks!

[Avise Sudhakar Rao]

Is this because of the connection timeout? will the below settings resolve the problem?

connectTimeout=20000;
socketTimeout=60000;
maxWaitTime=180000;

Thank you,
Sudhakar

--
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb
 
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Rod Adams]

Sudhakar --

Connection timeouts may or may not be part of your issue.

However, without the information which I requested on Feb 28, it would be impossible for me to do anything except speculate as to what is actually happening, and what an appropriate remedy would be.

Could you please provide some more details as to what is happening?

Thanks,

-- Rod

[Avise Sudhakar Rao]

1) How frequent is "frequent"? Every second? Every Minute? Roughly every 100 queries? Only under heavy load?

Frequently

2) Is there an additional error in your client log preceding the "com.mongodb.MongoException$Network" exception?

No

3) Can you look in the log file for the mongod server, and look for an error there?

No error in the mongod server

4) What kind of network do you have between client and MongoDB? It's possible that your network connections are getting corrupted. Note: in this event, my suggestion will be to either improve your network, or capture the exceptions and retry the query.

App servers and Mongod servers are on same VMware ESX server.

Please let me know, if require any more information.

thank you,

Sudhakar

[Rod Adams]

Hmm.

Generally, Network Exceptions fall into two broad categories:

- Other errors, which use Network Exceptions as a catch all notification that something has gone wrong.

- Actual problems with your network.

Since you're not seeing any other errors in your logs, that would point away from some other error happening.

Since your servers are on the same machine, there's a very limited number of network issues that can crop up.

At this point the two leading theories for me are:

1) Some other error, which does not stand out in the logs.

2) The queries which are getting the exceptions are taking exceptionally long to execute.

Starting with theory #1:

- What version of MongoDB are you running? [db.version()]

- What command line options are you using? [db.serverCmdLineOpts()]

- Would it be possible to show me an area of the mongod logfile around the time of one of these exceptions, posting it to pastebin or the like?

Working towards #2:

- Do the exceptions always happen for the same query? 

- If you run that query mongo shell, does it return quickly?

- Do you see any "slow queries" in your mongod log?

Regardless of what's causing this, however, it is generally a good standard practice to capture these exceptions, and then to perform a few retries, preferably with exponential backoff. This should absorb any transient issues which come about in a production system.

If, however, a single query fails to run several times in a row, while others continue to work, then that would suggest there is something special about that query, which can then merit further investigation.

Hope this helps.

-- Rod

[Jeff Yemin]

It would also be helpful to see a log of the cause of the com.mongodb.MongoException$Network exception, which should be an IOException of some sort.

Regards,

Jeff

[Mohan Kumar]

Hi

For my self have thae same issue like taht,When i am getting my opcursor, It will show like this error.


WARNING: can't clean cursors
com.mongodb.MongoException$Network: Exception opening the socket
    at com.mongodb.DBPort.<init>(DBPort.java:94)
    at com.mongodb.DBPortFactory.create(DBPortFactory.java:28)
    at com.mongodb.PooledConnectionProvider$ConnectionItemFactory.create(PooledConnectionProvider.java:186)
    at com.mongodb.PooledConnectionProvider$ConnectionItemFactory.create(PooledConnectionProvider.java:183)
    at com.mongodb.ConcurrentPool.createNewAndReleasePermitIfFailure(ConcurrentPool.java:150)
    at com.mongodb.ConcurrentPool.get(ConcurrentPool.java:118)
    at com.mongodb.PooledConnectionProvider.get(PooledConnectionProvider.java:75)
    at com.mongodb.DefaultServer.getConnection(DefaultServer.java:61)
    at com.mongodb.BaseCluster$WrappedServer.getConnection(BaseCluster.java:254)
    at com.mongodb.DBTCPConnector$MyPort.getConnection(DBTCPConnector.java:503)
    at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:434)
    at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:163)
    at com.mongodb.DBApiLayer.killCursors(DBApiLayer.java:190)
    at com.mongodb.DBApiLayer.cleanCursors(DBApiLayer.java:159)
    at com.mongodb.Mongo$CursorCleanerThread.run(Mongo.java:825)
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:579)
    at com.mongodb.DBPort.ensureOpen(DBPort.java:255)
    at com.mongodb.DBPort.<init>(DBPort.java:89)
    ... 14 more

Mar 09, 2015 3:27:30 PM com.mongodb.DBApiLayer cleanCursors