mongos running slowly
94 views
Skip to first unread message
Jon Schneider
Apr 30, 2012, 4:43:02 PM4/30/12
to mongodb-user
I am having this odd issue I can't get past:
I am trying to load some test data into a non-sharded database in a
sharded cluster.
When I run mongorestore to load some test data to a mongos instance
the load speed is about 1MB/second. When I run the same command but
against the primary in the shard containing the database the load
speed is more like 50MB/sec.
During the dump the mongos server uses about 10% cpu, io is minimal,
the system seems fairly quiet in general. In both cases I am runnig
mongorestore from the mongos server itself.
Running MongoDB 2.0.4 on CentOS 6.2. All servers are VMWare VM's.
Ideas?
I am trying to load some test data into a non-sharded database in a
sharded cluster.
When I run mongorestore to load some test data to a mongos instance
the load speed is about 1MB/second. When I run the same command but
against the primary in the shard containing the database the load
speed is more like 50MB/sec.
During the dump the mongos server uses about 10% cpu, io is minimal,
the system seems fairly quiet in general. In both cases I am runnig
mongorestore from the mongos server itself.
Running MongoDB 2.0.4 on CentOS 6.2. All servers are VMWare VM's.
Ideas?
Jon Schneider
Apr 30, 2012, 6:50:09 PM4/30/12
to mongodb-user
I setup a smaller test cluster, 1 confg server, 1 router, 2 set
members and am unable to duplicate to slowness. I even reconfigured
one of the prod mongos instances to point to the test cluster and
tested restoring and it was fast eliminating the router as the
problem. So I know the mongo primary is fast and the router is fast,
so why arn't they fast together? I have done everything I can do to
find an explination for this and am not having much luck so far.
members and am unable to duplicate to slowness. I even reconfigured
one of the prod mongos instances to point to the test cluster and
tested restoring and it was fast eliminating the router as the
problem. So I know the mongo primary is fast and the router is fast,
so why arn't they fast together? I have done everything I can do to
find an explination for this and am not having much luck so far.
Adam C
May 1, 2012, 7:53:17 AM5/1/12
to mongod...@googlegroups.com
Interesting behavior, can I ask - have you restarted the mongos at all when testing this?
Also, are you running with authentication enabled? I have seen this type of degradation in performance when running into this issue:
This is fixed in the 2.1 branch, and will eventually be back ported to the 2.0 branch but not in time for 2.0.5, so at the moment on 2.0 you have to restart the mongos to resolve.
Adam
Jon Schneider
May 1, 2012, 10:04:59 AM5/1/12
to mongodb-user
We have 6 mongos instances in this cluster, have tried all of them,
restarted etc, no dice. I restarted all the config servers, the repl
set members, no change. We are using authentication, come to think of
it I didn't enable auth in my test setup. I'll try that and make sure
its still fast. Another thing to note in our setup, 2 of the 5
replica set members in each shard are located in a different
datacenter 80 milliseconds away. Those members are set to a priority
of 0 and hidden.
restarted etc, no dice. I restarted all the config servers, the repl
set members, no change. We are using authentication, come to think of
it I didn't enable auth in my test setup. I'll try that and make sure
its still fast. Another thing to note in our setup, 2 of the 5
replica set members in each shard are located in a different
datacenter 80 milliseconds away. Those members are set to a priority
of 0 and hidden.
Jon Schneider
May 1, 2012, 10:48:54 AM5/1/12
to mongodb-user
Well I enabled auth on the test setup and guess what, now its slow,
although not quite as slow as our production setup. Possibly more set
members and shards compounds the problem. I am guessing this is a
bug. Example of what I am seeing:
Before Auth: (fast)
Restore to mongos:
[root@testserver scratch]# /opt/point2/mongodb/bin/mongorestore --host
mongotest --port 27017 -d testdb properties.bson
connected to: mongotest:27017
Mon Apr 30 15:52:12 properties.bson
Mon Apr 30 15:52:12 going into namespace [testdb.properties]
9168618/20325604307 0%
21463896/20325604307 0%
33556308/20325604307 0%
45867960/20325604307 0%
After Auth: (slow)
Restore to mongos:
[root@testserver scratch]# /opt/point2/mongodb/bin/mongorestore --host
mongotest --port 27017 -d testdb -u user -p password properties.bson
connected to: mongotest:27017
Tue May 1 08:37:05 properties.bson
Tue May 1 08:37:05 going into namespace [testdb.properties]
2656954/20325604307 0%
7127948/20325604307 0%
10420282/20325604307 0%
15139072/20325604307 0%
Restore to replica set primary: (fast)
[root@testserver scratch]# /opt/point2/mongodb/bin/mongorestore --host
mongotest --port 27018 -d testdb -u user -p password properties.bson
connected to: mongotest:27018
Tue May 1 08:39:32 properties.bson
Tue May 1 08:39:32 going into namespace [testdb.properties]
9899255/20325604307 0%
23405742/20325604307 0%
36914260/20325604307 0%
50356611/20325604307 0%
although not quite as slow as our production setup. Possibly more set
members and shards compounds the problem. I am guessing this is a
bug. Example of what I am seeing:
Before Auth: (fast)
Restore to mongos:
[root@testserver scratch]# /opt/point2/mongodb/bin/mongorestore --host
mongotest --port 27017 -d testdb properties.bson
connected to: mongotest:27017
Mon Apr 30 15:52:12 properties.bson
Mon Apr 30 15:52:12 going into namespace [testdb.properties]
9168618/20325604307 0%
21463896/20325604307 0%
33556308/20325604307 0%
45867960/20325604307 0%
After Auth: (slow)
Restore to mongos:
[root@testserver scratch]# /opt/point2/mongodb/bin/mongorestore --host
mongotest --port 27017 -d testdb -u user -p password properties.bson
connected to: mongotest:27017
Tue May 1 08:37:05 properties.bson
Tue May 1 08:37:05 going into namespace [testdb.properties]
2656954/20325604307 0%
7127948/20325604307 0%
10420282/20325604307 0%
15139072/20325604307 0%
Restore to replica set primary: (fast)
[root@testserver scratch]# /opt/point2/mongodb/bin/mongorestore --host
mongotest --port 27018 -d testdb -u user -p password properties.bson
connected to: mongotest:27018
Tue May 1 08:39:32 properties.bson
Tue May 1 08:39:32 going into namespace [testdb.properties]
9899255/20325604307 0%
23405742/20325604307 0%
36914260/20325604307 0%
50356611/20325604307 0%
Jon Schneider
May 1, 2012, 11:38:46 AM5/1/12
to mongodb-user
Just to add, mongodump is really fast. When using auth on the sharded
cluster I can dump 20-30 times faster than I can load. So perhaps
just a mongorestore bug? Let me know, I can create the jira if its
actually an issue.
cluster I can dump 20-30 times faster than I can load. So perhaps
just a mongorestore bug? Let me know, I can create the jira if its
actually an issue.
Jon Schneider
May 1, 2012, 1:22:43 PM5/1/12
to mongodb-user
Adam C
May 1, 2012, 1:38:03 PM5/1/12
to mongod...@googlegroups.com
Did you try on the 2.1 build? I'd be curious to see if it was also fixed there, given that 5405 fix is committed already.
Jon Schneider
May 1, 2012, 1:52:25 PM5/1/12
to mongodb-user
I will update the test machines to 2.1 and see what happens.
Adam C
May 1, 2012, 1:54:23 PM5/1/12
to mongod...@googlegroups.com
If it does fix it, then it is likely to be a dupe of 5405, and that is already being evaluated for back porting into the 2.0 branch. If not, then the issue is the right way to go.
Adam.
Thanks for all the testing :)
Adam.
Jon Schneider
May 1, 2012, 2:07:26 PM5/1/12
to mongodb-user
I upgraded the test cluster to 2.1.1-pre- (latest nightly) and
success, restore speed seemed to be about the same against the mongos
and the mongod. Now I really want these fixes in 2.0 branch :)
I'll update the issue, thanks!
success, restore speed seemed to be about the same against the mongos
and the mongod. Now I really want these fixes in 2.0 branch :)
I'll update the issue, thanks!
Adam C
May 3, 2012, 2:37:27 PM5/3/12
to mongod...@googlegroups.com
FWIW, I have already successfully tested a patched 2.0 build with the backported fix for SERVER-5405 in it (for a different issue). Assuming this is indeed the source of your problems, then it is looking good for a backport (barring a discovery of some sort of backward breaking issue) - so, probably into the 2.0.6 build once 2.0.5 is out the door. Keep an eye on the issue for a confirmation of the backport and target version.
Adam.
Adam.
Adam C
May 9, 2012, 6:48:26 AM5/9/12
to mongod...@googlegroups.com
Just as a follow up - I can confirm that 2.0.6 will have this backport as well as a couple of other bug fixes, so keep an eye out for that release candidate if you want to test out the fixes.
Adam
Adam
Reply all
Reply to author
Forward
0 new messages