Failover is happening, but system goes down anyway

[Sergei Tulentsev]

I have two shards. Initially they were replica sets of one node. Then I added one more node and arbiter to both sets. I crossed my fingers, threw some salt over the left shoulder and went production. And it was working well, for some time.

However, the primary server of one of the shards (namely, moskau8) had some hardware problem. Suddenly, the filesystem goes readonly and mongodb shuts down. Secondary( moskau4) takes over, but my mongos's still keep connecting to moskau8.
(last time when it happened, mongos's also went down. But today they survivied. They get tougher :-) )

Here's what listshards tells me

MongoDB shell version: 1.6.3
connecting to: admin
> db.runCommand( { listshards : 1 } )
{
    "shards" : [
        {
            "_id" : "moscow",
            "host" : "moscow/moskau7:27018"
        },
        {
            "_id" : "brussels",
            "host" : "brussels/moskau8:27018"
        }
    ],
    "ok" : 1
}

I believe I raised this issue here and was told that there's a known problem with somebody (mongos or mongod) not updating config data. Any advances here?

--
Best regards,
Sergei Tulentsev

[Eliot Horowitz]

This is being worked on in 1.7
You're best bet is to edit the data in the config.shards collection
with the full replset url so it can find the master.
In 1.7.2 this will be fully automated.

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>

[Sergei Tulentsev]

Arbiters do not need to be listed there, right?

[Eliot Horowitz]

They don't need to be - but the more the better.
Lets say you have servers X and Y and arbiter C Then you add Z. X and
Y go down, C is up, then via C, the mongos can find Z

On Thu, Oct 14, 2010 at 11:06 PM, Sergei Tulentsev

[Sergei Tulentsev]

Well, I tried that. Here's what I get in mongos

Fri Oct 15 07:33:27 [conn2] DBException in process: setShardVersion failed! { "errmsg" : "exception: gotShardHost different than what i had before  before [brussels/moskau8:27018]  got [brussels/moskau8:27018,moskau4:27018,moskau3:27019] ", "code" : 13299, "ok" : 0 }
Fri Oct 15 07:33:27 [conn2] ns: pravdorub_production.vk_users ClusteredCursor::query ShardConnection had to change attempt: 0
Fri Oct 15 07:33:27 [conn2]      setShardVersion failed: { errmsg: "exception: gotShardHost different than what i had before  before [brus...", code: 13299, ok: 0.0 }
Fri Oct 15 07:33:27 [conn2] Assertion: 10429:setShardVersion failed! { "errmsg" : "exception: gotShardHost different than what i had before  before [brussels/moskau8:27018]  got [brussels/moskau8:27018,moskau4:27018,moskau3:27019] ", "code" : 13299, "ok" : 0 }
0x5094de 0x60b8c4 0x60b38c
 /usr/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x1de) [0x5094de]
 /usr/bin/mongos(_ZN5mongo17checkShardVersionERNS_12DBClientBaseERKSsbi+0xa34) [0x60b8c4]
 /usr/bin/mongos(_ZN5mongo17checkShardVersionERNS_12DBClientBaseERKSsbi+0x4fc) [0x60b38c]

[Eliot Horowitz]

Right - you have to bounce the shard.
That is already fixed in 1.7 with replica sets.

On Thu, Oct 14, 2010 at 11:39 PM, Sergei Tulentsev

[Sergei Tulentsev]

'to bounce the shard'? How do I do that with 1.6.3? :-)

On Fri, Oct 15, 2010 at 5:40 AM, Sergei Tulentsev <sergei.t...@gmail.com> wrote:

And I am trying to update data like this:
> use config
> db.shards.update({_id : 'brussels'}, {$set : {host : 'brussels/moskau8:27018,moskau4:27018'}})

--
Best regards,
Sergei Tulentsev

[Eliot Horowitz]

Just restart mongod

On Thu, Oct 14, 2010 at 11:43 PM, Sergei Tulentsev

[Sergei Tulentsev]

And I am trying to update data like this:
> use config
> db.shards.update({_id : 'brussels'}, {$set : {host : 'brussels/moskau8:27018,moskau4:27018'}})

On Fri, Oct 15, 2010 at 5:39 AM, Sergei Tulentsev <sergei.t...@gmail.com> wrote:

[Sergei Tulentsev]

Thanks, now it's working as expected :-)