mongod fork

[mska...@redhat.com]

Hello,

how it is possible that mongod after forking is not ready accept connection?

I run it "mongod -f $MONGODB_CONFIG_PATH --fork --logpath=/var/lib/mongodb/create_users.log run"

Config file:

# mongodb.conf

port = 27017
pidfilepath = /var/lib/mongodb/mongodb.pid

# Set this value to designate a directory for the mongod instance to store its data.
# Default: /var/lib/mongodb/data
dbpath = /var/lib/mongodb/data

# Disable data file preallocation. Default: true
noprealloc = true

# Set MongoDB to use a smaller default data file size. Default: true
smallfiles = true

# Runs MongoDB in a quiet mode that attempts to limit the amount of output.
# Default: true
quiet = true

# Disable the HTTP interface (Defaults to localhost:28017).
nohttpinterface = true

Next command mongo $MONGODB_DATABASE --eval "db.addUser({user: '${MONGODB_USERNAME}', pwd: '${MONGODB_PASSWORD}', roles: [ 'readWrite' ]});"  fails with

MongoDB shell version: 2.4.9
connecting to: test
Thu Apr  9 13:01:24.973 Error: couldn't connect to server 127.0.0.1:27017 at src/mongo/shell/mongo.js:145
exception: connect failed

What should I change? Or how to check that mongod is ready?

[Stephen Steneker]

On Thursday, 9 April 2015 23:04:06 UTC+10, mska...@redhat.com wrote:

how it is possible that mongod after forking is not ready accept connection?

I run it "mongod -f $MONGODB_CONFIG_PATH --fork --logpath=/var/lib/mongodb/create_users.log run"

Hi,

Can you check the log info in /var/lib/mongodb/create_users.log? It's possible there is an error preventing MongoDB from starting successfully.

Alternatively, if you are running your `mongo` command line immediately after forking you could put in a sleep/retry loop in case it is trying to connect before the service has started.

Regards,

Stephen

[mska...@redhat.com]

There are no errors.

This is log before the error:

Fri Apr 10 06:01:59.859 [initandlisten] MongoDB starting : pid=17 port=27017 dbpath=/var/lib/mongodb/data 64-bit host=19359945a4c0
Fri Apr 10 06:01:59.859 [initandlisten] db version v2.4.9
Fri Apr 10 06:01:59.859 [initandlisten] git version: nogitversion
Fri Apr 10 06:01:59.859 [initandlisten] build info: Linux i-0001298a 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 x86_64 BOOST_LIB_VERSION=1_53
Fri Apr 10 06:01:59.859 [initandlisten] allocator: tcmalloc
Fri Apr 10 06:01:59.859 [initandlisten] options: { command: [ "run" ], config: "/var/lib/mongodb/mongodb.conf", dbpath: "/var/lib/mongodb/data", fork: true, logpath: "/var/lib/mongodb/create_users.log", nohttpinterface: "true", noprealloc: "true", pidfilepath: "/var/lib/mongodb/mongodb.pid", port: 27017, quiet: "true", smallfiles: "true" }
Fri Apr 10 06:01:59.872 [initandlisten] journal dir=/var/lib/mongodb/data/journal
Fri Apr 10 06:01:59.872 [initandlisten] recover : no journal files present, no recovery needed
Fri Apr 10 06:02:00.308 [initandlisten] preallocateIsFaster=true 3.64
Fri Apr 10 06:02:00.814 [initandlisten] preallocateIsFaster=true 5.52
Fri Apr 10 06:02:02.709 [initandlisten] preallocateIsFaster=true 7.5
Fri Apr 10 06:02:02.709 [initandlisten] preallocating a journal file /var/lib/mongodb/data/journal/prealloc.0
Fri Apr 10 06:02:06.263 [initandlisten] preallocating a journal file /var/lib/mongodb/data/journal/prealloc.1
Fri Apr 10 06:02:09.847 [initandlisten] preallocating a journal file /var/lib/mongodb/data/journal/prealloc.2
Fri Apr 10 06:02:12.915 [FileAllocator] allocating new datafile /var/lib/mongodb/data/local.ns, filling with zeroes...
Fri Apr 10 06:02:12.915 [FileAllocator] creating directory /var/lib/mongodb/data/_tmp

 This problem occurs only sometimes. It seems to me, that preallocation is not finished...

Yes, running sleep helps. But what time is enought to sleep to be sure that on every machine (even on the slow one) it will work? I thought that running mongod with forking stop blocking after mongod is ready to receive connection, it is not true?

[mska...@redhat.com]

And sometimes it works (for example with that sleep).

At the end of the log I mentioned in previous post, there is added:

Fri Apr 10 06:13:20.387 [FileAllocator] done allocating datafile /var/lib/mongodb/data/local.ns, size: 16MB,  took 0.019 secs
Fri Apr 10 06:13:20.387 [FileAllocator] allocating new datafile /var/lib/mongodb/data/local.0, filling with zeroes...
Fri Apr 10 06:13:20.399 [FileAllocator] done allocating datafile /var/lib/mongodb/data/local.0, size: 16MB,  took 0.011 secs
Fri Apr 10 06:13:20.402 [initandlisten] waiting for connections on port 27017
Fri Apr 10 06:13:20.554 [FileAllocator] allocating new datafile /var/lib/mongodb/data/test.ns, filling with zeroes...
Fri Apr 10 06:13:20.574 [FileAllocator] done allocating datafile /var/lib/mongodb/data/test.ns, size: 16MB,  took 0.019 secs
Fri Apr 10 06:13:20.575 [FileAllocator] allocating new datafile /var/lib/mongodb/data/test.0, filling with zeroes...
Fri Apr 10 06:13:20.585 [FileAllocator] done allocating datafile /var/lib/mongodb/data/test.0, size: 16MB,  took 0.01 secs
Fri Apr 10 06:13:20.586 [conn1] build index test.system.users { _id: 1 }
Fri Apr 10 06:13:20.587 [conn1] build index done.  scanned 0 total records. 0 secs
Fri Apr 10 06:13:20.587 [conn1] build index test.system.users { user: 1, userSource: 1 }
Fri Apr 10 06:13:20.587 [conn1] build index done.  scanned 0 total records. 0 secs

[Stephen Steneker]

Hi,

Can you elaborate on your use case .. are you standing up mongod for unit testing or is this for general usage?

In the example logs it looks like there are always data files being preallocated, which will happen before mongod starts listening for connections.

If this isn't a production environment you could make the startup time more predictable with --noprealloc:
 http://docs.mongodb.org/v2.4/reference/program/mongod/#cmdoption--noprealloc

Instead of a sleep & retry, a more indicative signal that the process is ready would be tailing the log for the initandlisten line, eg:

 "waiting for connections on port 27017"

Here's an example bash script with a loop to wait until the message appears in the log (which will be rotated when mongod starts):

#!/bin/bash

mongod --dbpath /data/db --logpath mongod.log --fork

until grep -q -m 1 "waiting for connections on port 27017" mongod.log; do

    sleep .5

    echo "... waiting for MongoDB to start ..."

done

echo "MongoDB is ready";

Regards,

Stephen

[mska...@redhat.com]

Hi Stephen,

thank you for the code.

So why is mongod putting "about to fork child process, waiting until server is ready for connections." into a log?

Thanks,
M.

[Eric Milkie]

I believe this is https://jira.mongodb.org/browse/SERVER-16292.