ERROR: too many regexes in query
skc
I cant find any documented limit on the number of regex clauses within
a query.
mongod emits the error message when querying using more than 4
regexes.
Any help?
Thanks,
Shimon Cohen
shimoco@skctp:~$ mongod --version
db version v1.2.4, pdfile version 4.5
Tue Apr 6 02:53:35 git version:
5cf582d3d96b882c400c33e7670b811ccd47f477
Eliot Horowitz
That's sort of arbitrary though.
Do you need more? What's the case?
> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>
skc
Well, the case is of querying on some selection of string items, while
matching on substrings.
Are there alternative methods to do that?
Shimon.
On Apr 6, 3:13 am, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> Right now there is a limit of 4.
> That's sort of arbitrary though.
> Do you need more? What's the case?
>
Eliot Horowitz
skc
Actually my application is a sort of a framework
where users are free to define their own selection forms, with
arbitrary number of fields.
Hence, queries are built up dynamically out of the selection values
posted.
I just didn't like to put a limit on the number
of values the user may enter on one form submission.
BTW,
Are there more quantitative limits on query elements ?
Shimon.
On Apr 6, 5:35 am, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> Can you send an example query?
>
Eliot Horowitz
I think that's the only one, and we should probably get rid of it.
Can you add a case here: http://jira.mongodb.org/ ?
skc
Thanks for your help.
The case added.
Shimon.
On Apr 6, 6:17 am, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> Ah - I see.
> I think that's the only one, and we should probably get rid of it.
> Can you add a case here:http://jira.mongodb.org/?
>
Joseph Wang
ArrayList<DBObject> list = new ArrayList<DBObject>();
for (int k = 0; k < MAX_OBJECT_CHUNKS; k++) {
...
list.add(myDb.getDBObject());
}
db.resetError();
collection.insert(list);
DBObject dbObject = db.getPreviousError()
if (dbObject != null && dbObject.get("err") != null) {
System.err.println("Info DbInsertListWorker:" + dbObject.get("err"));
}
After the program run, I only see less than 50% of documents in the collection.
However, I don't see any insertion error message.
According to
http://www.mongodb.org/display/DOCS/Last+Error+Commands
we should be using getPreviousError() for bulk insert.
1) Any idea why less than 50% of documents got inserted? I didn't set
unique index flag. [See
http://www.mongodb.org/display/DOCS/Indexes?focusedCommentId=6488159#comment-6488159]
Thus, all documents should be inserted.
2) Is there to determine which documents within the list failed insertion?
Dwight Merriman
Joseph Wang
Previously, I was able to insert 2.3 documents/ms (using multiple threads and bulk load).
After 8 hours run, the new batch is averaging 1 document/ms. Any suggestion what to
tune the system so that I can improve insertion rate? Thanks in advance.
> db.currentOp();
{
"inprog" : [
{
"opid" : 2014896,
"active" : false,
"lockType" : "write",
"waitingForLock" : true,
"op" : "insert",
"ns" : "?p.lp.mrtest",
"client" : "10.160.187.47:52985",
"desc" : "conn"
},
{
"opid" : 2014890,
"active" : false,
"lockType" : "write",
"waitingForLock" : true,
"op" : "insert",
"ns" : "?p.lp.mrtest",
"client" : "10.160.187.47:52988",
"desc" : "conn"
},
{
"opid" : 2014869,
"active" : true,
"lockType" : "write",
"waitingForLock" : false,
"secs_running" : 0,
"op" : "insert",
"ns" : "lp.lp.mrtest",
"client" : "10.160.187.47:52987",
"desc" : "conn"
},
{
"opid" : 2014881,
"active" : false,
"lockType" : "write",
"waitingForLock" : true,
"op" : "insert",
"ns" : "?p.lp.mrtest",
"client" : "10.160.187.47:52991",
"desc" : "conn"
},
{
"opid" : 2014887,
"active" : false,
"lockType" : "write",
"waitingForLock" : true,
"op" : "insert",
"ns" : "?p.lp.mrtest",
"client" : "10.160.187.47:52984",
"desc" : "conn"
},
{
"opid" : 2014877,
"active" : false,
"lockType" : "write",
"waitingForLock" : true,
"op" : "insert",
"ns" : "?p.lp.mrtest",
"client" : "10.160.187.47:52983",
"desc" : "conn"
},
{
"opid" : 2014878,
"active" : false,
"lockType" : "write",
"waitingForLock" : true,
"op" : "insert",
"ns" : "?p.lp.mrtest",
"client" : "10.160.187.47:52986",
"desc" : "conn"
},
{
"opid" : 2014872,
"active" : false,
"lockType" : "write",
"waitingForLock" : true,
"op" : "insert",
"ns" : "?p.lp.mrtest",
"client" : "10.160.187.47:52982",
"desc" : "conn"
},
{
"opid" : 2014893,
"active" : false,
"lockType" : "write",
"waitingForLock" : true,
"op" : "insert",
"ns" : "?p.lp.mrtest",
"client" : "10.160.187.47:52990",
"desc" : "conn"
},
{
"opid" : 2014884,
"active" : false,
"lockType" : "write",
"waitingForLock" : true,
"op" : "insert",
"ns" : "?p.lp.mrtest",
"client" : "10.160.187.47:52989",
"desc" : "conn"
}
]
}
> db.lp.mrtest.find().count();
78264000
Top:
Tasks: 64 total, 1 running, 63 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.3%us, 0.0%sy, 0.0%ni, 0.0%id, 98.0%wa, 0.0%hi, 0.0%si, 1.7%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 15736360k total, 15714872k used, 21488k free, 16628k buffers
Swap: 0k total, 0k used, 0k free, 14820388k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2277 root 15 0 56.3g 12g 12g S 0.3 82.3 63:52.34 mongod
[root@ip-10-160-58-3 bin]# iostat 5
Linux 2.6.21.7-2.fc8xen (ip-10-160-58-3) 04/06/10
avg-cpu: %user %nice %system %iowait %steal %idle
0.74 0.00 0.18 11.68 0.85 86.54
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda1 0.72 5.24 29.43 577050 3241112
sdb 413.46 409.32 10154.67 45083466 1118469584
avg-cpu: %user %nice %system %iowait %steal %idle
0.17 0.00 0.06 27.17 0.61 72.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda1 0.40 4.80 0.00 24 0
sdb 748.20 236.80 12041.60 1184 60208
[root@ip-10-160-58-3 bin]# vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 4 0 23944 15804 14818280 0 0 52 1273 36 35 1 0 87 12 1
0 3 0 22876 15820 14818876 0 0 134 6618 958 180 0 0 75 24 1
Aníbal Rojas
at a high rate to?
--
Aníbal Rojas
Ruby on Rails Web Developer
http://www.google.com/profiles/anibalrojas
Joseph Wang
Eliot Horowitz
Can you send a sample object and the indexes?
Also, can you call stats on the collection in the shell?
db.foo.stats()
Eliot Horowitz
http://www.mongodb.org/display/DOCS/Java+Driver+Concurrency
If not - are there any errors on the mongod log?
Joseph Wang
Extra Large Instance
15 GB memory
8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each)
1,690 GB instance storage (4×420 GB plus 10 GB root partition)
64-bit platform
I/O Performance: High
API name: m1.xlarge
> db.lp.mrtest.stats();
{
"ns" : "lp.lp.mrtest",
"count" : 144184100,
"size" : 21915983304,
"storageSize" : 24909152512,
"numExtents" : 96,
"nindexes" : 2,
"lastExtentSize" : 711991808,
"paddingFactor" : 1,
"flags" : 0,
"totalIndexSize" : 13356223840,
"indexSizes" : {
"_id_" : 8120837808,
"background_" : 5235386032
},
"ok" : 1
Joseph Wang
In the main program, we do
mongo = new Mongo(host, port);
After we fork out a thread, we do
DB db = mongo.getDB("lp");
db.requestStart();
if (db != null) {
DBCollection collection = db.getCollection("lp.mrtest");
...
db.resetError();
collection.insert(list);
DBObject dbObject = db.getPreviousError();
if (dbObject != null && dbObject.get("err") != null) {
System.err.println("Info DbInsertListWorker:" + dbObject.get("err"));
}
db.requestDone();
}
Any suggestion how we can determine which documents failed insertion when we
do insert(list)?
Saw the following messages in the master's log.
Tue Apr 6 09:21:11 Assertion: 10295:getFile(): bad file number value (corrupt db?): run repair
0x505bb4 0x5689bf 0x5d7e44 0x56e823 0x56f375 0x632316 0x63edbe 0x63f506 0x56115d 0x5fa2da 0x5feb2f 0x6a9a72 0x6bbde0 0x2aaaaacce4a7 0x2aaaab748c2d
/root/mongodb-linux-x86_64-1.4.0/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x204) [0x505bb4]
/root/mongodb-linux-x86_64-1.4.0/bin/mongod(_ZN5mongo8Database7getFileEiib+0x8f) [0x5689bf]
/root/mongodb-linux-x86_64-1.4.0/bin/mongod(_ZN5mongo19CoveredIndexMatcher7matchesERKNS_7BSONObjERKNS_7DiskLocEPNS_12MatchDetailsE+0xa4) [0x5d7e44]
/root/mongodb-linux-x86_64-1.4.0/bin/mongod(_ZN5mongo18FindingStartCursor4nextEv+0x213) [0x56e823]
/root/mongodb-linux-x86_64-1.4.0/bin/mongod(_ZN5mongo11UserQueryOp4nextEv+0xf5) [0x56f375]
/root/mongodb-linux-x86_64-1.4.0/bin/mongod(_ZN5mongo12QueryPlanSet6Runner6nextOpERNS_7QueryOpE+0x36) [0x632316]
/root/mongodb-linux-x86_64-1.4.0/bin/mongod(_ZN5mongo12QueryPlanSet6Runner3runEv+0x37e) [0x63edbe]
/root/mongodb-linux-x86_64-1.4.0/bin/mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x206) [0x63f506]
/root/mongodb-linux-x86_64-1.4.0/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpE+0x7ed) [0x56115d]
/root/mongodb-linux-x86_64-1.4.0/bin/mongod [0x5fa2da]
----- Original Message ----
From: Eliot Horowitz <elioth...@gmail.com>
To: mongod...@googlegroups.com
Eliot Horowitz
So the big problem is that message in the server log.
Did your server or mongod crash? I would recommend doing a repair asap.
Eliot Horowitz
Can you run mongostat for a while and send the output?
Joseph Wang
I'd removed all entries. Rerunning the insertion test now.
[root@ip-10-160-58-3 bin]# ./mongo localhost:10000
MongoDB shell version: 1.4.0
url: localhost:10000
connecting to: localhost:10000/test
type "help" for help
> use lp;
switched to db lp
> db.lp.mrtest.find().count();
0
> db.lp.mrtest.stats();
{
"ns" : "lp.lp.mrtest",
"count" : 0,
"size" : 0,
"storageSize" : 28835496960,
"numExtents" : 100,
"nindexes" : 2,
"lastExtentSize" : 711991808,
"paddingFactor" : 1,
"flags" : 0,
"totalIndexSize" : 9476835024,
"indexSizes" : {
"_id_" : 9476703952,
"mr_1_lt_1_lp_1_id_1_bbi_1_ir_1" : 131072
},
"ok" : 1
}
Begining of document insertion job:
[root@ip-10-160-58-3 bin]# ./mongostat -h localhost:10000 -d lp
connected to: localhost:10000
insert/s query/s update/s delete/s getmore/s command/s mapped vsize res % locked % idx miss conn time
160 0 0 0 2 323 75788 76180 14651 0.952 0 12 00:11:50
169 0 0 0 2 341 75788 76180 14652 0.942 0 12 00:11:51
165 0 0 0 1 328 75788 76180 14651 0.947 0 12 00:11:52
After 3 mins of document insertion job:
[root@ip-10-160-58-3 bin]# ./mongostat -h localhost:10000 -d lp
connected to: localhost:10000
insert/s query/s update/s delete/s getmore/s command/s mapped vsize res % locked % idx miss conn time
124 0 0 0 2 250 75788 76244 14656 0.971 0 12 00:13:55
140 0 0 0 1 279 75788 76244 14660 0.983 0 12 00:13:56
151 0 0 0 2 304 75788 76244 14661 0.976 0 12 00:13:57
After ~10 mins of document insertion job:
[[root@ip-10-160-58-3 bin]# ./mongostat -h localhost:10000 -d lp
connected to: localhost:10000
insert/s query/s update/s delete/s getmore/s command/s mapped vsize res % locked % idx miss conn time
38 0 0 0 1 77 75788 76244 14663 0.71 0 12 00:20:27
34 0 0 0 1 69 75788 76244 14665 1.11 0 12 00:20:28
14 0 0 0 0 29 75788 76244 14665 1.13 0 12 00:20:29
[root@ip-10-160-58-3 bin]# ./mongostat -h localhost:10000 -d lp
connected to: localhost:10000
insert/s query/s update/s delete/s getmore/s command/s mapped vsize res % locked % idx miss conn time
104 0 0 0 0 209 75788 76244 14661 0.985 0 12 00:22:50
12 0 0 0 0 25 75788 76244 14661 0.943 0 12 00:22:51
90 0 0 0 1 181 75788 76244 14665 1.12 0 12 00:22:52
146 0 0 0 0 296 75788 76244 14663 0.992 0 12 00:22:53
[root@ip-10-160-58-3 bin]# date
Wed Apr 7 00:24:11 EDT 2010
[root@ip-10-160-58-3 bin]# vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 2 0 20924 13712 14925088 0 0 668 1238 6 60 3 0 86 9 1
0 2 0 21060 13720 14924168 0 0 1016 15665 1233 363 3 1 76 18 3
[root@ip-10-160-58-3 bin]# iostat 5
Linux 2.6.21.7-2.fc8xen (ip-10-160-58-3) 04/07/10
avg-cpu: %user %nice %system %iowait %steal %idle
3.41 0.00 0.32 9.10 1.47 85.70
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda1 0.64 6.53 20.67 1090882 3452472
sdb 399.43 5337.34 9885.28 891295722 1650767864
avg-cpu: %user %nice %system %iowait %steal %idle
2.38 0.00 0.28 22.48 2.55 72.31
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda1 0.20 3.20 0.00 16 0
sdb 1144.80 1078.40 37216.00 5392 186080
> db.lp.mrtest.stats();
{
"ns" : "lp.lp.mrtest",
"count" : 5496900,
"size" : 527702448,
"storageSize" : 28835496960,
"numExtents" : 100,
"nindexes" : 2,
"lastExtentSize" : 711991808,
"paddingFactor" : 1,
"flags" : 0,
"totalIndexSize" : 10383493776,
"indexSizes" : {
"_id_" : 9775392464,
"mr_1_lt_1_lp_1_id_1_bbi_1_ir_1" : 608101312
},
"ok" : 1
}
Data Files
[root@ip-10-160-58-3]# ls -lt | head
total 79912312
drwxrwxrwx 2 root root 4096 Apr 6 19:26 _tmp
-rw------- 1 root root 2146435072 Apr 6 17:40 lp.30
-rwxrwxrwx 1 root root 5 Apr 6 15:54 mongod.lock
drwxrwxrwx 2 root root 4096 Apr 6 15:43 lp
-rw------- 1 root root 2146435072 Apr 6 15:36 lp.29
-rw------- 1 root root 2146435072 Apr 6 15:23 lp.28
-rw------- 1 root root 2146435072 Apr 6 14:51 lp.27
-rw------- 1 root root 2146435072 Apr 6 14:13 lp.26
-rw------- 1 root root 2146435072 Apr 6 14:00 lp.25
[root@ip-10-160-58-3 logs]# tail db
Wed Apr 7 00:25:56 insert lp.lp.mrtest 134ms
Wed Apr 7 00:25:57 insert lp.lp.mrtest 190ms
Wed Apr 7 00:25:57 insert lp.lp.mrtest 143ms
Wed Apr 7 00:25:57 insert lp.lp.mrtest 168ms
Wed Apr 7 00:25:58 insert lp.lp.mrtest 273ms
Wed Apr 7 00:25:59 insert lp.lp.mrtest 255ms
Wed Apr 7 00:26:00 insert lp.lp.mrtest 386ms
Wed Apr 7 00:26:00 insert lp.lp.mrtest 477ms
Wed Apr 7 00:26:01 insert lp.lp.mrtest 157ms
Wed Apr 7 00:26:01 insert lp.lp.mrtest 336ms
Joseph Wang
neither the server log nor the Java program log.
[root@ip-10-160-58-3 bin]# ./mongo localhost:10000
MongoDB shell version: 1.4.0
url: localhost:10000
connecting to: localhost:10000/test
type "help" for help
> use lp;
switched to db lp
> db.lp.mrtest.find().count();
12258300
Eliot Horowitz
Nothing we've done can reproduce.
Eliot Horowitz
I run and get exactly what I expect, 10k new objects per run.
One thing to note, you have:
DB db = mongo.getDB("lp");
DBCollection collection = db.getCollection("lp.mrtest");
That actually creates a full namespace called lp.lp.mrtest
is it possible you're looking at lp.mrtest and that's old?
You probably want to change the 2nd line to
DBCollection collection = db.getCollection("mrtest");
Joseph Wang
When running w/ 10 threads for 50M documents, after ~3 hours
stats() only shows less than 13M documents inserted.
Love to figure out the problem so that we can run multiple insertion
threads. Right now, have to resolve to do single thread insertion
s.t. I can run other tests. It will take over 6 hours to populate the
db.
What is the optimal size that insert(list) can support with the
smallest network overhead?
> show collections;
lp.mrtest
system.indexes
> db.lp.lp.mrtest.stats();
{ "errmsg" : "ns not found", "ok" : 0 }
> db.lp.mrtest.stats();
{
"ns" : "lp.lp.mrtest",
"count" : 15780000,
"size" : 1704240000,
"storageSize" : 28835496960,
"numExtents" : 100,
"nindexes" : 2,
"lastExtentSize" : 711991808,
"paddingFactor" : 1,
"flags" : 0,
"totalIndexSize" : 15658208640,
"indexSizes" : {
"_id_" : 11920926416,
"mr_1_lt_1_lp_1_id_1_bbi_1_ir_1" : 3737282224
},
"ok" : 1
Eliot Horowitz
Its possible you're hitting a weird nio bug.
Can you try building the jar from master on github?
Joseph Wang
EC2 server is running 1.6 JVM. I'm using 1.3 Mongo Java driver.
[root@ip-10-160-187-47 ~]# java -version
java version "1.6.0_14"
Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
Java HotSpot(TM) 64-Bit Server VM (build 14.0-b16, mixed mode)
[root@ip-10-160-187-47 ~]# ls -l mong*.jar
-rw-r--r-- 1 root root 231113 Apr 5 15:18 mongo-1.3.jar
Eliot Horowitz
see this for more info: http://jira.mongodb.org/browse/JAVA-100
if you could try master - would be great.
may want to drop old collection data before starting test
Joseph Wang
Eliot Horowitz
They're usually very easy to build, much simpler than the db.
Will be releasing 1.4 of the java driver soon though.
Kristina Chodorow
Joseph Wang
Still didn't see error message in neither mongo log nor program log.
> db.lp.mrtest.stats();
{
"ns" : "lp.lp.mrtest",
"count" : 4970000,
"size" : 437360032,
"storageSize" : 607047424,
"numExtents" : 20,
"nindexes" : 2,
"lastExtentSize" : 107698688,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 1031333760,
"indexSizes" : {
"_id_" : 396698560,
"mr_1_lt_1_lp_1_id_1_bbi_1_ir_1_pt_1" : 634635200
},
"ok" : 1
}
Could it be insertion failed because Mongo was allocating files? Still would be nice
to know if all inserts are successful. If there is an error, then have a way to determine
which documents we need to retry.
Fri Apr 9 09:26:34 allocating new datafile /mnt/data/test/lp.4, filling with zeroes...
Fri Apr 9 09:26:36 done allocating datafile /mnt/data/test/lp.4, size: 1024MB, took 1.92 secs
Eliot Horowitz
I tried the one you sent with various settings and can't get any issus.
Allocating files will not cause any writes to fail.
Have you looked at the server log? Are they any messages there?
Eliot Horowitz
Are there any errors in the logs?
I've run the program many times and its working fine.
Joseph Wang
MongoDB shell version: 1.4.0
Didn't see any error/exception in the mongo log.
Eliot Horowitz
Joseph Wang
long now = System.currentTimeMillis();
while (true) {
int count = threadPool.getActiveCount();
if (count < 1) {
break;
} else {
long diff = System.currentTimeMillis() - now;
if (diff > MAX_WAIT_TIME) {
break;
}
}
}
threadPool.shutdown();