php persistent connection best practices

[Dominik]

I was wondering what the current best practices regarding php
persistent connections were and if they are documentation officially
somewhere.

I've seen at several places ideas like

$persistId = 'db_' . rand(1, 50);
$mongo = new Mongo($dbHost, array('persist' =>
$persistId));

Is this the recommended way of doing connection pooling in PHP ? What
would happing if two more more PHP processes would 'by chance' use the
same $persistId ? Does the PHP driver in this case correctly queue the
requests and answers ?

Thanks,
Dominik

[Kristina Chodorow]

Is this the recommended way of doing connection pooling in PHP ?

Yes.  It will probably become more automatic in the future.  Make sure you keep track of how many connections total you have open, since Apache (for example) forks off a bunch of PHP processes, each of which ends up with its own pool.

What
would happing if two more more PHP processes would 'by chance' use the
same $persistId ? Does the PHP driver in this case correctly queue the
requests and answers ?

Yes.

There is documentation is at http://www.php.net/manual/en/mongo.connecting.php and http://www.php.net/manual/en/mongo.construct.php.

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Dominik]

> Make sure you
> keep track of how many connections total you have open, since Apache (for
> example) forks off a bunch of PHP processes, each of which ends up with its
> own pool.

If each PHP process uses its own connection pool (I'm using Apache
with mpm-prefork), why not just use
$mongo = new Mongo($dbHost, array('persist' => "1"));
? This would, if I understand your message correctly, create 'n'
distinct connections to MongoDB where 'n' is the number of apache
children

Am I misunderstanding something ?

Thanks again,
Dominik

[Kristina Chodorow]

Correct. "1" should work fine.  You could use rand() if you're saturating n connections so you want a 2n pool, or 3n, or so on.
 

Dominik

[Dominik]

> Correct. "1" should work fine.  You could use rand() if you're saturating n
> connections so you want a 2n pool, or 3n, or so on.

Thanks for the clarification. Now, as long as I don't do any multi-
threading inside the PHP scripts, how could I possibly saturate the
MongoDB connection ? In which circumstances would I benefit from 2 or
3 different persistent connections inside the PHP script ?

Thanks,
Dominik

[Kristina Chodorow]

Say you have 3 connections to the database and each connection can send 5MB/sec.  If you're trying to send more than 15MB/sec, you need more connections.

Dominik

[tmountain]

Not trying to hijack this thread, but I'm not sure I understand the
logic here. Assuming you're running Mongo with Apache prefork, each
worker can do a maximum of one thing at a time and each worker will
have one connection to the DB when setting the persist key to a static
value. Adding a rand(N) for the persist key is going to make each
worker open N connections to the DB; however, the workers can still
only do one thing at a time, so you're essentially just picking a
random connection and sending data across it. I don't see how this
buys you any more or less bandwidth; instead, you're just increasing
the maximum number of connections to your mongo server which may
actually degrade performance slightly.

-Travis

> > mongodb-user...@googlegroups.com<mongodb-user%2Bunsubscribe@google groups.com>
> > .

[Kristina Chodorow]

Many setups let you have multiple threads. Using one connection across many threads can make it very busy.

To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.

[jdill]

Persist will reduce connection overhead for sure, but I am not sure
totally sold on using persistent connections in all situations. There
is added risk. If a member of your replica set goes down, or
generally your db stops working, the connection may remain but when
you try to query you will get 'couldn't send query: Broken pipe'
errors. Even when you bring your server back online, you still have
the same issue unless you specifically close the connection and re-
open it again or if you restart your php cgi or php-fpm, etc.

In general, you may not need the added boost. We have a very high-
volume system in place which handled over 10 million hits in a day
with 3 (HVM compute cluster) web servers and we did not use persist
option. I may consider using them again once I am sure I can mitigate
the broken pipe issues.

- Jeremy

[Kristina Chodorow]

If it doesn't reconnect, persistent or no, that's a bug (one that should be fixed in 1.1.0). 

To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.

[Gotys]

I am having issues with this "broken pipe" behaviour under PHP's mongo
driver . I've installed the latest 1.1.3 from PECL and that still
didn't help.

What is happening is that when I use persistant connections, and
mongoDB is shut down - then restarted, the PHP driver doesnt' seem to
reconnect and keeps giving me broken pipe error until I restart
apache.

You are saying this was fixed ? Is this mongodb issue, or driver-
specific issue ? How can I get around it ?

Thanks !

[Kristina Chodorow]

I can't reproduce this...after any sort of exception, the driver closes the connection to the database and tries to make a new connection on the next request.  Is it possible that you're hitting different persistent connections each time?

To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.

[jdill]

I can confirm that new driver is better, and it does self-heal. But
generally there is at least 1 more failure before connection is fixed
and I still have to mitigate the broken pipe error with a try catch
retry.

[jdill]

Could it be related to custom timeout setting? I timeout at 1 second.

array("timeout"=>1000,"persist" => "x")

[Kristina Chodorow]

The driver has to error once (on most failures) because you don't want the it to automatically retry and end up send a MapReduce twice or something.  It's generally better to let user APIs (or frameworks) figure out what to do next.

You'll get an error message like "Operation in progress" if it actually times out on connect/reconnect.

To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.

[Derick Rethans]

On Fri, 17 Aug 2012, Sagar Sonawane wrote:

> I am getting Too Many open sockets error even if i create instance with
> persists setting. When i debug the issue i came to know that after 912
> calls to Mongo constructor it gives this error. I am using MPM model with
> following configuration,

What is the exact error? And how are you calling this?

> # worker MPM
> <IfModule worker.c>
> StartServers 5
> MaxClients 2000
> MinSpareThreads 25
> MaxSpareThreads 75
> ThreadsPerChild 50
> ServerLimit 40
> MaxRequestsPerChild 100
> </IfModule>
>
> Please let me know if i am wrong thinking that persists feature in
> php-Mongo Client has some issues. Because,
>
> 1. If it makes only one connection, in case of persists, then there should
> not be any open connections exception,

There will always be one connection per thread - which means that if you
have 750 threads running PHP, then there will be atleast 750
connections. You *really* should avoid the worker MPM though. The
threaded MPM does not work optimally with PHP.

> 2. I am using Mongo 1.3.0dev version in my setup, i.e. > 1.2.0 version in
> which they claimed that persistent is default feature and one should only
> "Persistent connection in production!!! as per
> http://www.php.net/manual/en/mongo.connecting.php"

How old is the 1.3.0dev version? Also, be aware that this is *not*
officially released!

> 3. I am calling Mongo constructor in loop from one PHP script, so i think
> my Apache Model doesn't matters here

No, but how are you calling this in a loop?

> Kindly tell me what to do with Mongo connection in production under heavy
> traffic (viz. 550 qps)

Let the driver manage it - if you're not trying to outsmart it it will
work as planned.

cheers,
Derick

--
{
website: [ "http://mongodb.org", "http://derickrethans.nl" ],
twitter: [ "@derickr", "@mongodb" ]
}

[Jeremy Mikola]

Sagar,

Based on the code you shared, I have the following input:

• Unless there is an actual need for creating and using multiple database connections (e.g. different auth credentials), connect once and share the Mongo instance. Your code currently creates over a thousand connections with repeated invocations of the Mongo constructor, so it's no wonder that you are exhausting socket allowances.
  
  
• I'm not sure what you meant by the "Controversial fix" comment around "$mongo->close()", but it's important to understand the ramifications of closing the connection. As mentioned in its documentation, there is almost never a reason to call that method, as you will forfeit any benefit of persistent connection handling. In this case, close() allowing you to implement a poor design by freeing up sockets, which you can then use to hammer the Mongo server with new connections.
  
  
• The current stable version of the driver (1.2.12) seems a better choice than using an unreleased build of the master branch, especially if you are considering this for a production deployment. It's not clear what you were expecting to get out of the master branch, or why you had no other choice but to use it (as you say).
  
  
• While using 1.2.12, I would suggest you look into MongoPool, which you can use to monitor the driver's connection handling. In the 1.2.x version, connection pools exist for each unique connection string (determined by host, port and auth credentials). You can set a pool size of one (before the pool is created with its first connection) to ensure that you are not creating redundant Mongo connections. This assumes that you have single-threaded workers handling PHP requests, as Derek advised earlier.

[Sagar Sonawane]

Hi Jeremy,

  Thanks for the reply.

  Your code currently creates over a thousand connections with repeated invocations of the Mongo constructor, so it's no wonder that you are exhausting socket allowances.

  Code i have shared was sample to reproduce the scenario i was talking about. I still wonder (i dont know why you have not wondered :) ), that how come i am "exhausting socket allowances" while i am using persistent connection! I expect a single connection resource, to be returned by the client, even if i am making 10k connections!!

I'm not sure what you meant by the "Controversial fix" comment around "$mongo->close()", but it's important to understand the ramifications of closing the connection. As mentioned in its documentation, there is almost never a reason to call that method, as you will forfeit any benefit of persistent connection handling. In this case, close() allowing you to implement a poor design by freeing up sockets, which you can then use to hammer the Mongo server with new connections.

thats what my point was...it will forfeit any benefit of persistent connection but then persistent connection should work otherwise you have to move to poorer design. I called it as "Controversial fix" because in this thread some were encouraging to use close() and some were going for persistent. please forgive me for my english :)

It's not clear what you were expecting to get out of the master branch, or why you had no other choice but to use it (as you say).

I will definitely try "stable version of the driver (1.2.12) " Thank you very much for this. But when i installed the driver at that time no stable release of the driver was available thats why i said i was not having any option.


I would suggest you look into MongoPool ....single-threaded workers handling PHP requests as Derek advised earlier

I will definitely read and try for MongoPool as well as the Apache Model @Derek thanks for that.

Also, i was curious to know about the ramifications of using the "safe" option in "update" command in Mongo. As i have observed one thing, even if i am using "safe" option and persistent connection, in my warm-up script to upsert documents from Excel Spreadsheet data, CPU goes to high and after the script execution stops it remains High for say 4-5 seconds. I wonder if  "safe" option is getting used or not in update command, as it should be synchronus o/p(if we are using "safe" in command) and not an asynchronus o/p(where mongo sends "true" to client and actually does the o/p later), which is default behaviour of Mongo.

[Jeremy Mikola]

On Wednesday, September 5, 2012 2:32:43 PM UTC+1, Sagar Sonawane wrote:

  Code i have shared was sample to reproduce the scenario i was talking about. I still wonder (i dont know why you have not wondered :) ), that how come i am "exhausting socket allowances" while i am using persistent connection! I expect a single connection resource, to be returned by the client, even if i am making 10k connections!!

I ran your code locally to investigate why connections were not being re-used, and I traced it to the MongoDB reference you store in ADOMongo's $db property. By convention, driver classes keep a reference to their logical parent, but not vice versa. E.g. MongoCollection will keep a reference to MongoDB, but when you ask MongoDB for a collection, it constructs a new MongoCollection instance and returns it without storing it internally.

Your code creates a cyclic reference which in turn prevents PHP's garbage collector from doing its job. If you manually unset the $db property before the old ADOMongo object leaves scope in your for loop, you'll enable PHP to properly clean up the ADOMongo object, which will return its connection to the pool for re-use.

Oddly enough, on the 1.3.0beta2 driver, your original code runs just fine. And I believe that's with a 1024 file descriptor limit.

 

Also, i was curious to know about the ramifications of using the "safe" option in "update" command in Mongo. As i have observed one thing, even if i am using "safe" option and persistent connection, in my warm-up script to upsert documents from Excel Spreadsheet data, CPU goes to high and after the script execution stops it remains High for say 4-5 seconds. I wonder if  "safe" option is getting used or not in update command, as it should be synchronus o/p(if we are using "safe" in command) and not an asynchronus o/p(where mongo sends "true" to client and actually does the o/p later), which is default behaviour of Mongo.

I don't believe there would be any CPU ramifications to using safe writes. The mongod server is going to process the operation as it normally would; you're simply asking the driver to wait for an acknowledgement by sending an getLastError command internally.

[Jeremy Mikola]

On Monday, September 10, 2012 2:12:01 PM UTC+1, Jeremy Mikola wrote:

Oddly enough, on the 1.3.0beta2 driver, your original code runs just fine. And I believe that's with a 1024 file descriptor limit.

Correction: this is to be expected in the new driver. Since we got rid of pools, connections are now tracked by their host, port and auth credentials (the same signature previously used to organize pools). If you construct multiple Mongo instances with the same parameters, only one connection will be used internally, so the file descriptor limit is never reached. Thanks to Derick for reminding me :)

[Sagar Sonawane]

Thank You very much for the replies.

Kindly point me to latest stable release of driver.

Regards,
Sagar Sonawane

[Jeremy Mikola]

On Wednesday, September 12, 2012 9:36:39 AM UTC+1, Sagar Sonawane wrote:

Thank You very much for the replies.

Kindly point me to latest stable release of driver.

As mentioned above, 1.2.12 is the current stable release: http://pecl.php.net/package/mongo