Getting connection refused because too many open connections: 819 of 819 every 12 hours or so

[Chris Chen]

Here's some lines from the log:

Fri May  4 02:17:47 [initandlisten] connection accepted from 127.0.0.1:60640 #8402

Fri May  4 02:17:47 [initandlisten] connection refused because too many open connections: 819 of 819

Fri May  4 02:17:47 [initandlisten] connection accepted from 127.0.0.1:60641 #8403

Fri May  4 02:17:47 [initandlisten] connection refused because too many open connections: 819 of 819

Fri May  4 02:17:47 [initandlisten] connection accepted from 127.0.0.1:60642 #8404

Fri May  4 02:17:47 [initandlisten] connection refused because too many open connections: 819 of 819

Fri May  4 02:17:47 [initandlisten] connection accepted from 127.0.0.1:60643 #8405

Fri May  4 02:17:47 [initandlisten] connection refused because too many open connections: 819 of 819

Fri May  4 02:17:47 [initandlisten] connection accepted from 127.0.0.1:60644 #8406

It just started happening recently and not sure what's causing it. My site isn't that heavily trafficked.

Running mongo gives:

MongoDB shell version: 1.8.0

connecting to: test

Fri May  4 02:17:09 DBClientCursor::init call() failed

exception: DBClientBase::findOne: transport error: 127.0.0.1 query: { whatsmyuri: 1 }

And:

$ ulimit -n

64000

Mongod version:

db version v1.8.0, pdfile version 4.5

Fri May  4 02:25:00 git version: 9c28b1d608df0ed6ebe791f63682370082da41c0

A simple restart of mongod fixes it until it happens again hours later, but I'd like to get to the root of the problem.

I'm using the PHP driver but I also run a cron script of a js file that does a map reduce job every 15 minutes. I run that js file directly through the mongo shell "mongo thefile.js."

The map reduce job usually takes less than a minute to run though.

Any idea what's causing this?

Thanks!

[gregor]

The ulimit -n you have pasted  - is that from the shell that you are starting mongod from or mongo?

[Chris Chen]

That was my user.

mongod is run with user www-data

Here is ulimit -n for www-data:

$ sudo su - www-data

$ ulimit -n

64000

[gregor]

Can you run 

lsof | grep mongod | grep TCP | wc -l

[gregor]

It sounds like you are getting the system limits of 1024 open files - 80% used for connections - 1024 * 80% = 819. 

Is your startup script setting ulimit -n maybe?

[Scott Hernandez]

Here is a page discussing this :
http://www.mongodb.org/display/DOCS/Too+Many+Open+Files

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/1eH_ZDoQrIgJ.
>
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

[Chris Chen]

$ lsof | grep mongod | grep TCP | wc -l

1

And the startup script doesn't seem to be setting ulimit.

Also:

$ lsof | grep mongod | grep data | wc

      4      35     495

$ lsof | grep mongod | grep TCP

mongodump  3501     mydb    4u     IPv4           66763105       0t0      TCP localhost:60429->localhost:27017 (ESTABLISHED)

[Chris Chen]

I seemed to have solved the issue by upgrading the mongo php driver to the latest version. I will update this thread if the issue returns.

> mongodb-user+unsubscribe@googlegroups.com.

[gregor]

Great please do so - glad you got it fixed :)

[itpro...@gmail.com]

I experienced this same exact issue and it had nothing to do with the mongo php driver. It was a service account for a remote host that was not able to authenticate to the MongoDB. It was trying to authenticate so many times in a short period of time it was causing too many connections. Resetting the password for for the service account resolved the issue.

1. Connect to the server hosting MongoDB via SSH as the root user.
2. Log onto the MongoDB as the root user by issuing the command [mongo im -u admin -p <current_password> --authenticationDatabase admin]
3. Execute the command following command to change the password for the service account failing to authenticate [db.changeUserPassword('serviceaccount','<new_password>')]
4. Log off from the MongoDB [exit]
   

==