Balancing Window (activeWindow) don't work in MongoDB 3.0.6

366 views
Skip to first unread message

Filip Havlíček

unread,
Oct 16, 2015, 9:08:32 AM10/16/15
to mongodb-user
I have small testing sharded cluster (1 mongos, 1 config, 2 shards - replica sets) and I set balancing window according to the instructions: http://docs.mongodb.org/manual/tutorial/manage-sharded-cluster-balancer/#schedule-the-balancing-window

my db.setting.find():
{ "_id" : "chunksize", "value" : 64 }
{ "_id" : "balancer", "stopped" : false, "activeWindow" : { "start" : "23:00", "stop" : "6:00" } }

sh.status():
balancer:
    Currently enabled:  yes
    Currently running:  no
        Balancer active window is set between 23:00 and 6:00 server local time
    Failed balancer rounds in last 5 attempts:  0
    Migration Results for the last 24 hours:
        6 : Success


But in the mongos log file there arestill informations about balancing rounds:
2015-10-16T15:03:14.475+0200 I SHARDING [Balancer] distributed lock 'balancer/d845a340857a:27017:1444999599:1804289383' acquired, ts : 5620f592d1dbbe9a92bb7835
2015-10-16T15:03:14.515+0200 I SHARDING [Balancer] distributed lock 'balancer/d845a340857a:27017:1444999599:1804289383' unlocked.
2015-10-16T15:03:24.517+0200 I SHARDING [Balancer] distributed lock 'balancer/d845a340857a:27017:1444999599:1804289383' acquired, ts : 5620f59cd1dbbe9a92bb7837
2015-10-16T15:03:24.549+0200 I SHARDING [Balancer] distributed lock 'balancer/d845a340857a:27017:1444999599:1804289383' unlocked.
2015-10-16T15:03:34.551+0200 I SHARDING [Balancer] distributed lock 'balancer/d845a340857a:27017:1444999599:1804289383' acquired, ts : 5620f5a6d1dbbe9a92bb7839
2015-10-16T15:03:34.628+0200 I SHARDING [Balancer] distributed lock 'balancer/d845a340857a:27017:1444999599:1804289383' unlocked.


And balancing of chunks between shards really works, as I can see it in sh.status().
Maybe it's a bug, maybe it's known.. I use ballancing window it in MongoDB 2.6 and it works normally.

Asya Kamsky

unread,
Oct 18, 2015, 5:03:52 PM10/18/15
to mongodb-user
I don't see any evidence in your logs that the balancer is *running*.
Remember, the balancer thread needs to wake up just to check whether
or not it can or needs to run.

Asya
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user"
> group.
>
> For other MongoDB technical support options, see:
> http://www.mongodb.org/about/support/.
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mongodb-user/05caced7-522a-49cb-964b-5fef08786dce%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers

Filip Havlíček

unread,
Oct 26, 2015, 3:59:15 AM10/26/15
to mongodb-user
Hi, sorry for late answer, I was on holiday.
It's only a part of log. I will make more tests and send you new logs.
I compared logs from testing cluster 3.0 with logs from production cluster 2.6 and in 2.6 there are no logs from Balancer (distributed lock acquired/unlocked) outside the blanacing window (logs from Balancer occure only from 23:00 to 6:00).

Filip


Dne neděle 18. října 2015 23:03:52 UTC+2 Asya Kamsky napsal(a):

Filip Havlíček

unread,
Oct 26, 2015, 4:17:59 AM10/26/15
to mongodb-user
simple test with mongoimport of testing dataset (https://raw.githubusercontent.com/mongodb/docs-assets/primer-dataset/dataset.json)
more logs, where you can see chunk migration:

2015-10-26T09:07:45.612+0100 I SHARDING [Balancer] distributed lock 'balancer/d845a340857a:27017:1445846694:1804289383' acquired, ts : 562ddf511d6fa5d7d503b81f
2015-10-26T09:07:45.658+0100 I SHARDING [Balancer] distributed lock 'balancer/d845a340857a:27017:1445846694:1804289383' unlocked.
2015-10-26T09:07:47.082+0100 I NETWORK  [mongosMain] connection accepted from 127.0.0.1:34103 #8 (2 connections now open)
2015-10-26T09:07:47.099+0100 I ACCESS   [conn8] Successfully authenticated as principal root on admin
2015-10-26T09:07:48.376+0100 I SHARDING [conn8] ChunkManager: time to load chunks for sharding1.wroar: 0ms sequenceNumber: 5 version: 4|4||5620f86de05ed1826544bae9 based on: 4|1||5620f86de05ed1826544bae9
2015-10-26T09:07:48.376+0100 I SHARDING [conn8] autosplitted sharding1.wroar shard: ns: sharding1.wroar, shard: repl2:repl2/localhost:27040,localhost:27041, lastmod: 3|3||000000000000000000000000, min: { restaurant_id: "41476658" }, max: { restaurant_id: "50018995" } into 3 (splitThreshold 16777216)
2015-10-26T09:07:48.390+0100 I NETWORK  [conn8] end connection 127.0.0.1:34103 (1 connection now open)
2015-10-26T09:07:54.527+0100 I SHARDING [LockPinger] cluster localhost:27019 pinged successfully at Mon Oct 26 09:07:54 2015 by distributed lock pinger 'localhost:27019/d845a340857a:27017:1445846694:1804289383', sleeping for 30000ms
2015-10-26T09:07:55.660+0100 I SHARDING [Balancer] distributed lock 'balancer/d845a340857a:27017:1445846694:1804289383' acquired, ts : 562ddf5b1d6fa5d7d5041b30
2015-10-26T09:07:55.662+0100 I SHARDING [Balancer]  ns: sharding1.wroar going to move { _id: "sharding1.wroar-restaurant_id_"40396152"", ns: "sharding1.wroar", min: { restaurant_id: "40396152" }, max: { restaurant_id: "40512788" }, version: Timestamp 4000|1, versionEpoch: ObjectId('5620f86de05ed1826544bae9'), lastmod: Timestamp 4000|1, lastmodEpoch: ObjectId('5620f86de05ed1826544bae9'), shard: "repl2" } from: repl2 to: repl1 tag []
2015-10-26T09:07:55.663+0100 I SHARDING [Balancer] moving chunk ns: sharding1.wroar moving ( ns: sharding1.wroar, shard: repl2:repl2/localhost:27040,localhost:27041, lastmod: 4|1||000000000000000000000000, min: { restaurant_id: "40396152" }, max: { restaurant_id: "40512788" }) repl2:repl2/localhost:27040,localhost:27041 -> repl1:repl1/localhost:27030,localhost:27031
2015-10-26T09:07:59.970+0100 I SHARDING [Balancer] ChunkManager: time to load chunks for sharding1.wroar: 0ms sequenceNumber: 6 version: 5|1||5620f86de05ed1826544bae9 based on: 4|4||5620f86de05ed1826544bae9
2015-10-26T09:07:59.988+0100 I SHARDING [Balancer] distributed lock 'balancer/d845a340857a:27017:1445846694:1804289383' unlocked.
2015-10-26T09:08:00.990+0100 I SHARDING [Balancer] distributed lock 'balancer/d845a340857a:27017:1445846694:1804289383' acquired, ts : 562ddf601d6fa5d7d5041b32
2015-10-26T09:08:01.028+0100 I SHARDING [Balancer] distributed lock 'balancer/d845a340857a:27017:1445846694:1804289383' unlocked.
2015-10-26T09:08:11.030+0100 I SHARDING [Balancer] distributed lock 'balancer/d845a340857a:27017:1445846694:1804289383' acquired, ts : 562ddf6b1d6fa5d7d5041b34
2015-10-26T09:08:11.087+0100 I SHARDING [Balancer] distributed lock 'balancer/d845a340857a:27017:1445846694:1804289383' unlocked.



Dne neděle 18. října 2015 23:03:52 UTC+2 Asya Kamsky napsal(a):
I don't see any evidence in your logs that the balancer is *running*.

renjith Vrs

unread,
Oct 26, 2015, 4:45:48 PM10/26/15
to mongodb-user
i got this error while running my project. i need help to solve this issue. the resources i used are got from here


i really need to solve this problem immediately. i am new to mongodb and also to php.  please help

 
Fatal error: Uncaught exception 'MongoConnectionException' with message 'Failed to connect to: 127.0.0.1:27017: SASL Authentication failed on database 'blog': Authentication failed.' in C:\xampp\htdocs\s\db.php:28 Stack trace: #0 C:\xampp\htdocs\s\db.php(28): MongoClient->__construct('mongodb://127.0...', Array) #1 C:\xampp\htdocs\s\db.php(19): Blog\DB\DB->connect(Array) #2 C:\xampp\htdocs\s\app.php(10): Blog\DB\DB->__construct(Array) #3 C:\xampp\htdocs\s\index.php(5): require('C:\\xampp\\htdocs...') #4 {main} thrown in C:\xampp\htdocs\s\db.php on line 28

Asya Kamsky

unread,
Oct 27, 2015, 1:56:11 PM10/27/15
to mongod...@googlegroups.com
Please don't reply to threads with messages that are unrelated - start a new thread and describe your setup *completely*. 
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/31fde7f7-4d65-46bb-999f-6a3766fdcc10%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Asya Kamsky

unread,
Oct 27, 2015, 2:00:34 PM10/27/15
to mongod...@googlegroups.com
Was this a new collection?  There is some special logic in balancing a new collection (or rather collections with a very small number of chunks) and I wonder if that's what you are seeing. 
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/f20f9116-f2fd-4c6d-a678-b9a589d23571%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Filip Havlíček

unread,
Oct 29, 2015, 5:32:19 AM10/29/15
to mongodb-user
Yes, it was new collection. But now, it's older and migration still occure.
It has small number of chunks.
from sh.status():
    {  "_id" : "sharding1",  "partitioned" : true,  "primary" : "repl2" }
        sharding1.wroar
            shard key: { "restaurant_id" : 1 }
            chunks:
                repl1    8
                repl2    8

Can you provide me more information plese? I can increase the number of chunks ant test if activeWindow is still ignored or not.
Thank You.

Dne úterý 27. října 2015 19:00:34 UTC+1 Asya Kamsky napsal(a):
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.

Filip Havlíček

unread,
Oct 30, 2015, 6:18:23 AM10/30/15
to mongodb-user
Now I have much more chunks and balancing window is still ignored:

LOG:
2015-10-30T11:11:17.628+0100 I SHARDING [Balancer] distributed lock 'balancer/d845a340857a:27017:1446196609:1804289383' acquired, ts : 563342450907eb237c2bc5a9
2015-10-30T11:11:17.631+0100 I SHARDING [Balancer]  ns: sharding1.wroar going to move { _id: "sharding1.wroar-restaurant_id_"40400270"", ns: "sharding1.wroar", min: { restaurant_id: "40400270" }, max: { restaurant_id: "40423532" }, version: Timestamp 42000|1, versionEpoch: ObjectId('5620f86de05ed1826544bae9'), lastmod: Timestamp 42000|1, lastmodEpoch: ObjectId('5620f86de05ed1826544bae9'), shard: "repl2" } from: repl2 to: repl1 tag []
2015-10-30T11:11:17.631+0100 I SHARDING [Balancer] moving chunk ns: sharding1.wroar moving ( ns: sharding1.wroar, shard: repl2:repl2/localhost:27040,localhost:27041, lastmod: 42|1||000000000000000000000000, min: { restaurant_id: "40400270" }, max: { restaurant_id: "40423532" }) repl2:repl2/localhost:27040,localhost:27041 -> repl1:repl1/localhost:27030,localhost:27031

PART FROM sh.status()
  balancer:
    Currently enabled:  yes
    Currently running:  yes
        Balancer lock taken at Fri Oct 30 2015 11:11:17 GMT+0100 (CET) by d845a340857a:27017:1446196609:1804289383:Balancer:846930886

        Balancer active window is set between 23:00 and 6:00 server local time
    Collections with active migrations:
        sharding1.wroar started at Fri Oct 30 2015 11:11:17 GMT+0100 (CET)

    Failed balancer rounds in last 5 attempts:  0
    Migration Results for the last 24 hours:
        30 : Success
        44 : Failed with error 'moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because  there are still 1 deletes from previous migration', from repl1 to repl2
.
.
.
.
.   
   {  "_id" : "sharding1",  "partitioned" : true,  "primary" : "repl2" }
        sharding1.wroar
            shard key: { "restaurant_id" : 1 }
            chunks:
                repl1    99
                repl2    107
            too many chunks to print, use verbose if you want to force print


Dne úterý 27. října 2015 19:00:34 UTC+1 Asya Kamsky napsal(a):
Was this a new collection?  There is some special logic in balancing a new collection (or rather collections with a very small number of chunks) and I wonder if that's what you are seeing. 
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.

Filip Havlíček

unread,
Nov 6, 2015, 4:57:19 AM11/6/15
to mongodb-user
My problem is solved by 3.0.7.
related tickets:
https://jira.mongodb.org/browse/SERVER-20557
https://jira.mongodb.org/browse/SERVER-19948

After upgrade to 3.0.7 balancer active window is not ignored.


Dne úterý 27. října 2015 19:00:34 UTC+1 Asya Kamsky napsal(a):
Was this a new collection?  There is some special logic in balancing a new collection (or rather collections with a very small number of chunks) and I wonder if that's what you are seeing. 
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages