Cluster Shutdown Order

[Allen]

Is there a certain order in which the mongod processes of a shard
should be shut down? For example, if I'm running a config server,
shard master, and (other) shard slave on the same box, does the
shutdown order matter?

[Eliot Horowitz]

Order doesn't matter, but if you wanted to reduce the number of errors
in the logs, you should shutdown mongos first, then shards, then
config servers.

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>

[Greg Studer]

If there are no mongos processes still running, the three processes are
unrelated and not talking to one another and may be shut down in any
order. If there are still mongos processes active, shutting down the
config server instance first should prevent any shard metadata changes
while the other servers go down.

[Allen]

Thanks guys. Might your comments be good information to include on
http://www.mongodb.org/display/DOCS/Starting+and+Stopping+Mongo?

Now, another shutdown-related question: can you think of any reason
why "sudo kill -2" would be a more reliable way to shutdown mongod as
opposed to "mongo host:port/admin --eval "db.shutdownServer()""?
Sometimes, when I try to shutdown my cluster while one or more queries
are in progress, using the db.shutdownServer() approach fails:

MongoDB shell version: 1.8.1
connecting to: localhost:29017/admin
assert failed : unexpected error: "shutdownServer failed: db assertion
failure"
Error("Printing Stack Trace")@:0
()@shell/utils.js:21
("assert failed : unexpected error: \"shutdownServer failed: db
assertion failure\"")@shell/utils.js:32
(false,"unexpected error: \"shutdownServer failed: db assertion failure
\"")@shell/utils.js:42
()@shell/db.js:179
@(shell eval):1

The server's log file (-vvv):

Tue May 10 11:06:46 [initandlisten] MongoDB starting : pid=1965
port=29017 dbpath=/mongo_alpha/mongo/shard_data 64-bit
Tue May 10 11:06:46 [initandlisten] db version v1.8.1, pdfile version
4.5
Tue May 10 11:06:46 [initandlisten] git version:
a429cd4f535b2499cc4130b06ff7c26f41c00f04
Tue May 10 11:06:46 [initandlisten] build sys info: Linux bs-
linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28
EST 2009 x86_64 BOOST_LIB_VERSION=1_41
...
Tue May 10 11:28:49 [conn6] terminating, shutdown command received
Tue May 10 11:29:46 [DataFileSync] flushing mmap took 3ms for 25
files
Tue May 10 11:30:46 [DataFileSync] flushing mmap took 3ms for 25
files
Tue May 10 11:30:49 [conn6] query admin.$cmd ntoreturn:1 assertion
command: { shutdown: 1.0 } reslen:150 120015ms
Tue May 10 11:30:49 [conn6] MessagingPort recv() conn closed?
127.0.0.1:53622
Tue May 10 11:30:49 [conn6] SocketException: remote: error: 9001
socket exception [0]
Tue May 10 11:30:49 [conn6] end connection 127.0.0.1:53622
Tue May 10 11:31:46 [DataFileSync] flushing mmap took 3ms for 25
files
Tue May 10 11:32:46 [DataFileSync] flushing mmap took 2ms for 25
files
...

However, after this shutdown failure, I was able to gracefully shut
down this server by executing "sudo kill -2 1965." Further
experimentation revealed that using the kill approach is consistently
better at shutting down an active server in my linux environment. Any
ideas?

[Greg Studer]

Seems to be a bug, opened a JIRA case. I'd use -2 in the meanwhile.

https://jira.mongodb.org/browse/SERVER-3077