very slow removeshard
106 views
Skip to first unread message
Azat Khuzhin
May 5, 2012, 4:04:39 AM5/5/12
to mongod...@googlegroups.com
Hi all
I execute "removeshard" command, and it is very slow
To delete 2 chunks need 8 hours
Yes, there is concurrent read/write (insert/query/update - 94/47/47) but this - is very slow
gregor
May 5, 2012, 10:16:51 AM5/5/12
to mongod...@googlegroups.com
Can you paste the output of mongostat
and also are there any warnings/errors in the mongos log file?
Azat Khuzhin
May 5, 2012, 10:22:54 AM5/5/12
to mongod...@googlegroups.com
I couldn't, because I already migrate all data, myself
Stop mongos, on the second shards execute insert to first shards and delete from it
However, I did not see any error messages in log, at all mongod, mongos instances
Azat Khuzhin.
Send from phone.
--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To view this discussion on the web visit https://groups.google.com/d/msg/mongodb-user/-/FjK6adPdz-cJ.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
gregor
May 7, 2012, 2:28:10 PM5/7/12
to mongod...@googlegroups.com
It was the long running query that was stopping the removeShard.
To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com.
Azat Khuzhin
May 7, 2012, 2:59:35 PM5/7/12
to mongod...@googlegroups.com
There's a query, to another collection, this played role?
And also what about yields?
Azat Khuzhin.
Send from phone.
To view this discussion on the web visit https://groups.google.com/d/msg/mongodb-user/-/kQYyutBf4cAJ.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
gregor
May 14, 2012, 10:40:08 AM5/14/12
to mongod...@googlegroups.com
The collection you were querying to was also sharded to the same shard? Then this would slow the removeShard.
The yield is for pagefaults and long running queries to allow other operations to complete - but those operations also yield back to the original query.
Azat Khuzhin
May 14, 2012, 10:53:54 AM5/14/12
to mongod...@googlegroups.com
That collection not shaded
But even if was sharded, it must migrate 1 chunk in hour (1 hour is also very alloy I think), which mongo don't
Azat Khuzhin.
Send from phone.
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb
gregor
May 15, 2012, 4:02:51 AM5/15/12
to mongod...@googlegroups.com
It seems slow yes but I think to be sure you would need to try to removeShard again and run mongostat / iostat to check exactly what is going on.
Azat Khuzhin
May 15, 2012, 4:13:48 AM5/15/12
to mongod...@googlegroups.com
I'v already migrate all data myself.
--
Azat Khuzhin
So maybe at weekends I try doing this on ec2 instance.
Azat Khuzhin
gregor
May 15, 2012, 4:32:19 AM5/15/12
to mongod...@googlegroups.com
Are you using ext4?
Azat Khuzhin
Azat Khuzhin
May 15, 2012, 4:34:56 AM5/15/12
to mongod...@googlegroups.com
ext3, but you already write that ext4 is better
--
Azat Khuzhin
Are you using ext4?
Azat Khuzhin
gregor
May 22, 2012, 2:57:45 AM5/22/12
to mongod...@googlegroups.com
How did you get on with this?
On Tuesday, May 15, 2012 9:34:56 AM UTC+1, Azat Khuzhin wrote:
On Tuesday, May 15, 2012 9:34:56 AM UTC+1, Azat Khuzhin wrote:
ext3, but you already write that ext4 is better
On Tue, May 15, 2012 at 12:32 PM, gregor wrote:
Are you using ext4?
On Tuesday, May 15, 2012 9:13:48 AM UTC+1, Azat Khuzhin wrote:
I'v already migrate all data myself.
So maybe at weekends I try doing this on ec2 instance.
--
Azat Khuzhin
Azat Khuzhin
May 22, 2012, 3:24:14 AM5/22/12
to mongod...@googlegroups.com
if you ask me if I convert ext3 -> ext4, the answer is - no. I can't do this at this time.
Azat Khuzhin
gregor
May 22, 2012, 3:32:12 AM5/22/12
to mongod...@googlegroups.com
No I meant checking if it was the the queries using disk io that was slowing your migrates - by using iostat.
On Tuesday, May 22, 2012 8:24:14 AM UTC+1, Azat Khuzhin wrote:
On Tuesday, May 22, 2012 8:24:14 AM UTC+1, Azat Khuzhin wrote:
if you ask me if I convert ext3 -> ext4, the answer is - no. I can't do this at this time.
Azat Khuzhin
May 22, 2012, 4:07:37 AM5/22/12
to mongod...@googlegroups.com
This is not because of disk IO slows process
--
Azat Khuzhin
iostat shows that reads/writes to disk not so much, as it could be. The same situation with process resources.
Azat Khuzhin
gregor
May 22, 2012, 4:23:17 AM5/22/12
to mongod...@googlegroups.com
The non-sharded collection that you were making queries to is on which shard?
On Tuesday, May 22, 2012 9:07:37 AM UTC+1, Azat Khuzhin wrote:
Do you have enough disk space on all shards?
On Tuesday, May 22, 2012 9:07:37 AM UTC+1, Azat Khuzhin wrote:
This is not because of disk IO slows process
iostat shows that reads/writes to disk not so much, as it could be. The same situation with process resources.
Azat Khuzhin
May 22, 2012, 4:46:44 AM5/22/12
to mongod...@googlegroups.com
On shard00 (first, the same machine mongos runs)
--
Azat Khuzhin
Yes, space enough
Azat Khuzhin
gregor
May 22, 2012, 4:56:04 AM5/22/12
to mongod...@googlegroups.com
So you are not sending queries to the shard you are trying to remove?
On Tuesday, May 22, 2012 9:46:44 AM UTC+1, Azat Khuzhin wrote:
On Tuesday, May 22, 2012 9:46:44 AM UTC+1, Azat Khuzhin wrote:
On shard00 (first, the same machine mongos runs)
Yes, space enough
Azat Khuzhin
May 22, 2012, 6:03:13 AM5/22/12
to mongod...@googlegroups.com
Maybe but don't sure.
--
Azat Khuzhin
And if there is a queries to shard, that I what to remove, not many and that will be find() by field, by that field collection have index
Azat Khuzhin
Azat Khuzhin
May 22, 2012, 6:04:59 AM5/22/12
to mongod...@googlegroups.com
Now I do some tests on ec2
shard (moveChunk) & unshard (removeshard)
When I finished, and if not forget, I post my results here
--
Azat Khuzhin
Azat Khuzhin
gregor
May 22, 2012, 6:22:27 AM5/22/12
to mongod...@googlegroups.com
You could try to lower the chunk size this might make remove faster
On Tuesday, May 22, 2012 11:04:59 AM UTC+1, Azat Khuzhin wrote:
And please run iostat and mongostat and capture output while doing remove.
Have you considered installing mms?
On Tuesday, May 22, 2012 11:04:59 AM UTC+1, Azat Khuzhin wrote:
Now I do some tests on ec2shard (moveChunk) & unshard (removeshard)
When I finished, and if not forget, I post my results here
On Tue, May 22, 2012 at 2:03 PM, Azat Khuzhin <a3at...@gmail.com> wrote:
Maybe but don't sure.
And if there is a queries to shard, that I what to remove, not many and that will be find() by field, by that field collection have index
--
Azat Khuzhin
Azat Khuzhin
May 22, 2012, 7:18:15 AM5/22/12
to mongod...@googlegroups.com
iostat & mongostat is running
--
Azat Khuzhin
About MMS install, if I there is enough time, I will. Thanks
Azat Khuzhin
Azat Khuzhin
May 23, 2012, 10:24:45 AM5/23/12
to mongod...@googlegroups.com
I create a ticket in you jira
--
Azat Khuzhin
"removeshard" in that tests faster that this one
Maybe some of queries slow this progress
Azat Khuzhin
Reply all
Reply to author
Forward
0 new messages