mongo-php-driver patch to fix memory leak cause by cursor

[un44444444]

I got a memory leak problem in mongodb, when query data using php with
cursor.
but it goes OK with my c++ client.

Then i use tcpdump to capture the packages, and find that c++ client
send 11 package while php send 10.
the one missing is OP_KILL_CURSORS

So i add a killCursor() method in MongoCursor.
It seems memory leak problem fix with below code:
<code>
$db = $connection->selectDB($database_names);
$collection = $db->selectCollection($table_name);
$cursor = $collection->find($query,array("UID"))->limit($limit_size);
while($cursor->hasNext())
{
$user = $cursor->getNext();
var_dump($user);
}
$cursor->killCursor();
unset($cursor);
</code>

cursor.patch:
<pre>
diff --git a/cursor.c b/cursor.c
index 7efc5fa..26cacfd 100644
--- a/cursor.c
+++ b/cursor.c
@@ -810,6 +810,31 @@ PHP_METHOD(MongoCursor, count) {
zval_ptr_dtor(&db_z);
}

+/* {{{ MongoCursor::killCursor
+ */
+PHP_METHOD(MongoCursor, killCursor) {
+ mongo_cursor *cursor =
(mongo_cursor*)zend_object_store_get_object(getThis() TSRMLS_CC);
+ char quickbuf[128];
+ buffer buf;
+ zval temp;
+
+ //If the cursor_id is 0, the db is out of results anyway.
+ if (cursor->cursor_id == 0) {
+ return;
+ }
+
+ //
+ buf.pos = quickbuf;
+ buf.start = buf.pos;
+ buf.end = buf.start + 128;
+
+ php_mongo_write_kill_cursors(&buf, cursor TSRMLS_CC);
+ mongo_say(cursor->link, &buf, &temp TSRMLS_CC);
+
+ cursor->cursor_id = 0;
+}
+/* }}} */
+
static function_entry MongoCursor_methods[] = {
PHP_ME(MongoCursor, __construct, NULL, ZEND_ACC_CTOR|
ZEND_ACC_PUBLIC)
PHP_ME(MongoCursor, hasNext, NULL, ZEND_ACC_PUBLIC)
@@ -828,6 +853,7 @@ static function_entry MongoCursor_methods[] = {
PHP_ME(MongoCursor, sort, NULL, ZEND_ACC_PUBLIC)
PHP_ME(MongoCursor, hint, NULL, ZEND_ACC_PUBLIC)
PHP_ME(MongoCursor, addOption, NULL, ZEND_ACC_PUBLIC)
+ PHP_ME(MongoCursor, killCursor, NULL, ZEND_ACC_PUBLIC)

/* query */
PHP_ME(MongoCursor, doQuery, NULL, ZEND_ACC_PROTECTED)
diff --git a/cursor.h b/cursor.h
index 1be561e..507ac44 100644
--- a/cursor.h
+++ b/cursor.h
@@ -45,6 +45,7 @@ PHP_METHOD(MongoCursor, valid);
PHP_METHOD(MongoCursor, reset);
PHP_METHOD(MongoCursor, count);
PHP_METHOD(MongoCursor, info);
+PHP_METHOD(MongoCursor, killCursor);

#define preiteration_setup mongo_cursor
*cursor; \

PHP_MONGO_GET_CURSOR(getThis());
\

</pre>

[Kristina Chodorow]

Is MongoDB crashing?  How did you determine that it's leaking?

Thanks for the patch, but it shouldn't change memory usage at all.  If you exhaust a cursor's results (calling hasNext() or more() until it returns false), the DB should automatically clean up the cursor and there is no need to call kill_cursor.  In the patch you sent, cursor->cursor_id will be 0 (based on the code you sent) and so the kill_cursor message will never be sent. 

If cursor_id is not 0, when the cursor goes out of scope kill_cursor will be called anyway, so there's no need to make the user handle it.

--
You received this message because you are subscribed to the Google Groups "mongodb-dev" group.
To post to this group, send email to mongo...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-dev...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-dev?hl=en.

[wei chen]

Maybe few request to php can not reproduce the problem.
In my stress test, 100+ client request php page concurrently, each query return 20 record to client,
and in mongodb server side memory increase rapidly, up to about 10MB/s, finally die of bad_alloc().

In monogdb server http interface, i can see something like follow:
# databases 3
cursors: 100000
and when php get heavy stress, cursor increase by every query.

I don't know how mongodb server deals with cursors, but i guess there is a thread to recycle them.
if we not tell server to kill cursors,  server delete them after timeout?

best regards!

2010/9/27 Kristina Chodorow <kris...@10gen.com>

[wei chen]

In my environment, cursor_id is not 0 after $cursor->hasNext() return false.
I had verify that, by
- edit cursor.c add cursor_id in info() method
- then print info in php

while($cursor->hasNext())
{
       $user = $cursor->getNext();
       var_dump($user);
}

var_dump($cursor->info());

2010/9/27 Kristina Chodorow <kris...@10gen.com>

[Kristina Chodorow]

Looks like a bug, thanks for the report.  Looking into it.

[Kristina Chodorow]

Okay, fixed.  Could you give the code at http://github.com/mongodb/mongo-php-driver a try to make sure it works for you?

[wei chen]

The code works!
When i do stress test again, thing goes OK.
and throught tcpdump i also see the OP_KILL_CURSORS in my query.

Thanks for your fix!

2010/9/28 Kristina Chodorow <kris...@10gen.com>

[Kristina Chodorow]

Great, thanks for reporting it.