Expected behavior that DBs in a cluster might always have differing document counts?

[Matthew Woodward]

My last email was probably too much of a brain dump so my apologies for that. I figured I'd try again with a more simple version of one piece of the puzzle I'm trying to put together.

In a BigCouch cluster is it expected behavior that databases across members of the cluster would be in a permanent state of having differing document counts, or does that indicate a problem?

To put it another way, if I know for a fact no activity is occurring on a particular database should I expect that the document counts across cluster members would match, or is that not necessarily expected?

Thanks.

--
Matthew Woodward
ma...@mattwoodward.com
http://blog.mattwoodward.com
identi.ca / Twitter: @mpwoodward

Please do not send me proprietary file formats such as Word, PowerPoint, etc. as attachments.
http://www.gnu.org/philosophy/no-word-attachments.html

[Shawn Parrish]

I would expect them to match unless internal replication is not happening correctly.  Our doc counts on our shards are the same.

Shawn

[Matthew Woodward]

Thanks -- we're only seeing the count discrepancies on a specific DB so I'll probably wind up nuking the "bad" shards. Appreciate the info.

[Paul Davis]

Check and see if your validate_doc_update is perhaps checking for a
specific user context that isn't set during internal replication as
that's usually the cause of this.

[Matthew Woodward]

Ah-HA! Interesting stuff!

I do have a validate_doc_update in both the databases in question and it's checking for either the user/password I created for this database or 'admin' as the user context.

Does that need to include another user context for replication to work properly?

Thanks!

[Paul Davis]

You should short circuit the validation if the user has the "_admin"
role. Once you have that set then you should run an
_all_docs?include_docs=true over the database and it should fix itself
up.

[Matthew Woodward]

On Mon, Oct 29, 2012 at 11:09 AM, Paul Davis <paul.jos...@gmail.com> wrote:

You should short circuit the validation if the user has the "_admin"
role. Once you have that set then you should run an
_all_docs?include_docs=true over the database and it should fix itself
up.

Excellent. Can't thank you enough for that tip. I'll give it a shot later today and report back either way so the loop is closed.

Really appreciate it.

[Matthew Woodward]

On Mon, Oct 29, 2012 at 11:11 AM, Matthew Woodward <ma...@mattwoodward.com> wrote:

On Mon, Oct 29, 2012 at 11:09 AM, Paul Davis <paul.jos...@gmail.com> wrote:

You should short circuit the validation if the user has the "_admin"
role. Once you have that set then you should run an
_all_docs?include_docs=true over the database and it should fix itself
up.

Excellent. Can't thank you enough for that tip. I'll give it a shot later today and report back either way so the loop is closed.

Well, that doesn't seem to have fixed it -- even on a node that I know has shards on it I can watch the document count vacillate (which i guess means a node doesn't always use its own shards?), and the count across the nodes still changes on practically every refresh of Futon.

What I might do next is try to figure out which is the node whose shards have the highest document count (which should be "correct" given the nature of this database), copy those shards to the other two servers that have shards for this database, and see how things go from there.

Appreciate the tip on the _admin user -- great info regardless of whether or not it fixed my immediate issue.

[Matthew Woodward]

On Mon, Oct 29, 2012 at 12:59 PM, Matthew Woodward <ma...@mattwoodward.com> wrote:

What I might do next is try to figure out which is the node whose shards have the highest document count (which should be "correct" given the nature of this database), copy those shards to the other two servers that have shards for this database, and see how things go from there.

I think I have a bit more info on the issue, so we'll see if the logic hangs with the people who know a lot more about how all this works behind the scenes than I do.

As I mentioned before, the only databases with which we were having issues were ones where we had applications writing to a host name on our load balancer as opposed to writing directly to an individual cluster node.

After doing a lot more digging around I discovered that in the vm.args file, the user name was the same (the default of simply bigcouch) on all the nodes, as opposed to the recommended (required?) practice of having each node have its own full name, e.g. bigc...@server.com

Given the issue with the names being the same in vm.args and the fact that the only databases we ever had an issue with are ones where we were writing to multiple nodes instead of only one, I'm wondering if this is the root cause of the differing document counts.

I certainly hope that's the case because I'd love to be able to throw writes at the cluster as a whole instead of to individual nodes, but given the issues we had when we did that I'm a bit reluctant.

As for how I (hopefully) solved the problem, I picked the "correct" shards and copied those to the other servers that had shards for the offending databases in addition to adding the _admin user to any validate_doc_update functions and we'll see how that goes. With DC shut down because of the hurricane I won't know for a few days but I'm keeping my fingers crossed.

Thanks again to everyone for the input! I'd love to hear if having the same name in vm.args would potentially cause the sorts of issues we were seeing so I can spread the write load across our cluster, so if anyone has input on that it'd be much appreciated.

[Paul Davis]

You can check how the cluster is connected by hitting
http://hostname:5984/_membership on each node and comparing. The
username isn't really a username, its an Erlang node name. Assuming
your cluster is connected then those will be unique because they'll
have 'nodename@hostname' which is good enough to differentiate.

[Matthew Woodward]

On Tue, Oct 30, 2012 at 11:17 AM, Paul Davis <paul.jos...@gmail.com> wrote:

You can check how the cluster is connected by hitting
http://hostname:5984/_membership on each node and comparing. The
username isn't really a username, its an Erlang node name. Assuming
your cluster is connected then those will be unique because they'll
have 'nodename@hostname' which is good enough to differentiate.

Interesting -- did that previously (and double-checked again just now) to be sure and that all looks good (all have user@host), so guess that means the vm.args thing being wrong wasn't the culprit. Bummer.

Well, we'll see how things hold up after getting the shards realigned and maybe I'll set up a dummy database and a script that throws writes at the cluster to see if I can duplicate the issue.

Thanks so much again for all the info and suggestions.