Is there a good reason for "combinatorial limit of $in partitioning of result set exceeded errors"?

[Remon van Vliet]

Hi,

I'm trying to run a query that has two $in clauses on a sharded
collection. Both clauses are of equal size (in terms of amount of
elements per array) and can exceed 1000. So something like :

db.col.update({a:{$in:[1, 2, ..., 1000+]}, b:{$in:[1, 2, ...,
1000+]}}, {$set:{c:1}});

For this collection the shard key is {a:1}

When I try to execute this update the server throws an error
"combinatorial limit of $in partitioning of result set exceeded
errors" (due to an assert at line 917 in queryutils.cpp). I think this
is due to the fact that it doesn't allow the amount of possible
combinations of all the $in clauses to be higher than 1000000 :

uassert( 13385, "combinatorial limit of $in partitioning of result set
exceeded", size() < 1000000 );

I'm wondering why this limitation is in place. It seems to me that
mongos can create shard specific version of this update and target it
at the apprioriate shard. E.g. :

shard1 -> db.col.update({a:{$in:[<all values in original $in that
reside on shard 1>]}, b:{$in:[1, 2, ..., 1000+]}}, {$set:{c:1}});
shard2 -> db.col.update({a:{$in:[<all values in original $in that
reside on shard 2>]}, b:{$in:[1, 2, ..., 1000+]}}, {$set:{c:1}});
...
shardN -> db.col.update({a:{$in:[<all values in original $in that
reside on shard N>]}, b:{$in:[1, 2, ..., 1000+]}}, {$set:{c:1}});

From there on it should be the same as a query on a non-sharded
collection and that does seem to work for the same query regardless of
the sizes of the $in clauses.

Is there a reason for this limitation?

[Eliot Horowitz]

Its related to this:
https://jira.mongodb.org/browse/SERVER-4745
Might be worth following up there.

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>

[Remon van Vliet]

Hi,

That issue is related to compound shard keys where I see the need for
some combinatorial limit. But I have a simple 1 field shard key and I
do not understand why it would throw this error since shard targetting
should be relatively straightforward. Is there an implementation
detail I'm not taking into account?

In any case I'll comment on the issue. Thanks!