PHP driver connecting to replica set error - No candidate servers found

4,042 views
Skip to first unread message

Justin

unread,
Feb 3, 2013, 1:38:10 PM2/3/13
to mongod...@googlegroups.com
I am seeing a very strange bug using the PHP driver. I am running a three member replica set, each member is running v2.2.3.

If I take a member in the replica set offline, sometime the PHP client fails to connect with:

    No candidate servers found

The only way to fix it, is to restart the php-fpm worker, and magically it goes back to normal and connects. Here is my PHP connection code:

try {
    $client = new MongoClient("mongodb://username-here:passwo...@mongodb1.local.mydomain.com,mongodb2.local.mydomain.com,mongodb3.local.mydomain.com/my-database", array("timeout" => 10000, "replicaSet" => "rs-san-jose");
catch(Exception $mongoException) {
    print_r($mongoException);
}

And each member in the replica set is binding to its local ip:

     bind_ip = 10.X.X.X
     port = 27017

Any ideas?

Thanks.

Hannes Magnusson

unread,
Feb 3, 2013, 2:06:43 PM2/3/13
to mongod...@googlegroups.com
On Sun, Feb 3, 2013 at 10:38 AM, Justin <comm...@nodesocket.com> wrote:
> I am seeing a very strange bug using the PHP driver. I am running a three
> member replica set, each member is running v2.2.3.
>
> If I take a member in the replica set offline, sometime the PHP client fails
> to connect with:
>
> No candidate servers found
>
> The only way to fix it, is to restart the php-fpm worker, and magically it
> goes back to normal and connects. Here is my PHP connection code:
>
> try {
> $client = new
> MongoClient("mongodb://username-here:passwo...@mongodb1.local.mydomain.com,mongodb2.local.mydomain.com,mongodb3.local.mydomain.com/my-database",
> array("timeout" => 10000, "replicaSet" => "rs-san-jose");
> catch(Exception $mongoException) {
> print_r($mongoException);
> }


I can only reproduce this when I don't have an primary (i.e. force
elections), which is expected behaviour as we can't login since your
Read Preferences only allow us to execute on the primary.

After adding adding "readPreference=" .
MongoClient::RP_PRIMARY_PREFERRED, to your connection string it should
work fine again...
Note that reads will then also be executed on the secondaries when
there is no primary available.


-Hannes

Justin

unread,
Feb 3, 2013, 2:31:51 PM2/3/13
to mongod...@googlegroups.com
Just tested again, by taking the primary offline, waiting a few minutes, then bringing the primary back. PHP keeps on throwing the exception:

   No candidate servers found

This is a big problem, since basically the replica set does not provide high availability. Any further details I can provide to help troubleshoot this?


On Sunday, February 3, 2013 7:38:10 PM UTC+1, Justin wrote:
I am seeing a very strange bug using the PHP driver. I am running a three member replica set, each member is running v2.2.3.

If I take a member in the replica set offline, sometime the PHP client fails to connect with:

    No candidate servers found

The only way to fix it, is to restart the php-fpm worker, and magically it goes back to normal and connects. Here is my PHP connection code:

try {
    $client = new MongoClient("mongodb://username-here:password-here@mongodb1.local.mydomain.com,mongodb2.local.mydomain.com,mongodb3.local.mydomain.com/my-database", array("timeout" => 10000, "replicaSet" => "rs-san-jose");

Hannes Magnusson

unread,
Feb 3, 2013, 2:54:10 PM2/3/13
to mongod...@googlegroups.com
What are your mongo.ping_interval and mongo.is_master_interval INI settings?
Did you check the server after those had elapsed? - And are you sure
you hit the same php process twice?

You will get that exception until we discover that there is a primary
somewhere, but we only do that check every "once in a while".
Note that it is normal that each php process will throw that exception
once after the interval has elapsed.



-Hannes

On Sun, Feb 3, 2013 at 11:31 AM, Justin <comm...@nodesocket.com> wrote:
> Just tested again, by taking the primary offline, waiting a few minutes,
> then bringing the primary back. PHP keeps on throwing the exception:
>
> No candidate servers found
>
> This is a big problem, since basically the replica set does not provide high
> availability. Any further details I can provide to help troubleshoot this?
>
>
> On Sunday, February 3, 2013 7:38:10 PM UTC+1, Justin wrote:
>>
>> I am seeing a very strange bug using the PHP driver. I am running a three
>> member replica set, each member is running v2.2.3.
>>
>> If I take a member in the replica set offline, sometime the PHP client
>> fails to connect with:
>>
>> No candidate servers found
>>
>> The only way to fix it, is to restart the php-fpm worker, and magically it
>> goes back to normal and connects. Here is my PHP connection code:
>>
>> try {
>> $client = new
>> MongoClient("mongodb://username-here:passwo...@mongodb1.local.mydomain.com,mongodb2.local.mydomain.com,mongodb3.local.mydomain.com/my-database",
>> array("timeout" => 10000, "replicaSet" => "rs-san-jose");
>> catch(Exception $mongoException) {
>> print_r($mongoException);
>> }
>>
>> And each member in the replica set is binding to its local ip:
>>
>> bind_ip = 10.X.X.X
>> port = 27017
>>
>> Any ideas?
>>
>> Thanks.
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
>
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

Justin

unread,
Feb 3, 2013, 3:32:00 PM2/3/13
to mongod...@googlegroups.com
mongo.ping_interval and mongo.is_master_interval are defaults, I am not setting them.

I just ran another test, here is result of rs.status() when all nodes are online:


And here is the replica set config. Really basic, the only thing that is custom is that I set priorities so mongodb1 is the preferred master.


Then I stopped mongod on mongodb1. Here is the result of rs.status():


PHP started throwing the exception (No candidate servers found). Waited a few minutes (like 3) and still throwing the exception. Then I restarted php-fpm and it works.

Thanks.

On Sunday, February 3, 2013 7:38:10 PM UTC+1, Justin wrote:
I am seeing a very strange bug using the PHP driver. I am running a three member replica set, each member is running v2.2.3.

If I take a member in the replica set offline, sometime the PHP client fails to connect with:

    No candidate servers found

The only way to fix it, is to restart the php-fpm worker, and magically it goes back to normal and connects. Here is my PHP connection code:

try {
    $client = new MongoClient("mongodb://username-here:password-here@mongodb1.local.mydomain.com,mongodb2.local.mydomain.com,mongodb3.local.mydomain.com/my-database", array("timeout" => 10000, "replicaSet" => "rs-san-jose");

Derick Rethans

unread,
Feb 3, 2013, 5:07:35 PM2/3/13
to mongod...@googlegroups.com
Maybe I missed this, but which version of the driver are you using?

cheers,
Derick


On Sun, 3 Feb 2013, Justin wrote:

> mongo.ping_interval and mongo.is_master_interval are defaults, I am not
> setting them.
>
> I just ran another test, here is result of rs.status() when all nodes are
> online:
>
> https://gist.github.com/4703511
>
> And here is the replica set config. Really basic, the only thing that is
> custom is that I set priorities so mongodb1 is the preferred master.
>
> https://gist.github.com/4703515
>
> Then I stopped mongod on mongodb1. Here is the result of rs.status():
>
> https://gist.github.com/4703529
>
> PHP started throwing the exception (*No candidate servers found*). Waited a
> few minutes (like 3) and still throwing the exception. Then I restarted
> php-fpm and it works.
>
> Thanks.
>
> On Sunday, February 3, 2013 7:38:10 PM UTC+1, Justin wrote:
> >
> > I am seeing a very strange bug using the PHP driver. I am running a three
> > member replica set, each member is running v2.2.3.
> >
> > If I take a member in the replica set offline, sometime the PHP client
> > fails to connect with:
> >
> > *No candidate servers found*
> >
> > The only way to fix it, is to restart the php-fpm worker, and magically it
> > goes back to normal and connects. Here is my PHP connection code:
> >
> > try {
> > $client = new MongoClient("mongodb://
> > username-here:passwo...@mongodb1.local.mydomain.com,
> > mongodb2.local.mydomain.com,mongodb3.local.mydomain.com/my-database",
> > array("timeout" => 10000, "replicaSet" => "rs-san-jose");
> > catch(Exception $mongoException) {
> > print_r($mongoException);
> > }
> >
> > And each member in the replica set is binding to its local ip:
> >
> > bind_ip = 10.X.X.X
> > port = 27017
> >
> > Any ideas?
> >
> > Thanks.
> >
>
>

--
{
website: [ "http://mongodb.org", "http://derickrethans.nl" ],
twitter: [ "@derickr", "@mongodb" ]
}

Justin

unread,
Feb 4, 2013, 8:16:52 AM2/4/13
to mongod...@googlegroups.com
Derick,

I just grabbed the latest from GitHub, I think it was early last week. php -i says:

Version => 1.3.4-dev


On Sunday, February 3, 2013 7:38:10 PM UTC+1, Justin wrote:
I am seeing a very strange bug using the PHP driver. I am running a three member replica set, each member is running v2.2.3.

If I take a member in the replica set offline, sometime the PHP client fails to connect with:

    No candidate servers found

The only way to fix it, is to restart the php-fpm worker, and magically it goes back to normal and connects. Here is my PHP connection code:

try {
    $client = new MongoClient("mongodb://username-here:password-here@mongodb1.local.mydomain.com,mongodb2.local.mydomain.com,mongodb3.local.mydomain.com/my-database", array("timeout" => 10000, "replicaSet" => "rs-san-jose");

Justin

unread,
Feb 5, 2013, 5:15:06 PM2/5/13
to mongod...@googlegroups.com
Bump, any ideas from the official PHP driver maintainers?

Hannes Magnusson

unread,
Feb 5, 2013, 10:22:46 PM2/5/13
to mongod...@googlegroups.com
On Sun, Feb 3, 2013 at 12:32 PM, Justin <comm...@nodesocket.com> wrote:
> mongo.ping_interval and mongo.is_master_interval are defaults, I am not
> setting them.
>
> I just ran another test, here is result of rs.status() when all nodes are
> online:
>
> https://gist.github.com/4703511
>
> And here is the replica set config. Really basic, the only thing that is
> custom is that I set priorities so mongodb1 is the preferred master.
>
> https://gist.github.com/4703515
>
> Then I stopped mongod on mongodb1. Here is the result of rs.status():
>
> https://gist.github.com/4703529
>
> PHP started throwing the exception (No candidate servers found). Waited a
> few minutes (like 3) and still throwing the exception. Then I restarted
> php-fpm and it works.


How many processes do you have running?

It is normal that you get this message once for each process...

Since you seem to be able to consistently reproduce this, could you
enable logging while this is happening?

Add to the top of your script:
MongoLog::setLevel(MongoLog::ALL);
MongoLog::setModule(MongoLog::ALL);


It will automatically log whole bunch of things the driver is doing
internally to your php error log by default.

-Hannes

Justin

unread,
Feb 12, 2013, 1:59:40 PM2/12/13
to mongod...@googlegroups.com
Hannes,

Sorry for the delay in responding. I have 16 php-fpm worker processes. Ok, the logging is a lot, and contains sensitive details. What is strange, is if, I stop mongod on the primary so mongodb1, then keep on refreshing my app, it alternates between connection successful and an error, even though mongodb2 and mongodb3 are still up. I shouldn't see this downtime and behavior correct?

Here is a snippet of logs which seem particularly interesting. Again, the logs contain sensitive details, my password, database names, hostnames, and IP addresses, so I replaced the sensitive details.

Thanks a lot.










On Sunday, February 3, 2013 7:38:10 PM UTC+1, Justin wrote:
I am seeing a very strange bug using the PHP driver. I am running a three member replica set, each member is running v2.2.3.

If I take a member in the replica set offline, sometime the PHP client fails to connect with:

    No candidate servers found

The only way to fix it, is to restart the php-fpm worker, and magically it goes back to normal and connects. Here is my PHP connection code:

try {
    $client = new MongoClient("mongodb://username-here:password-here@mongodb1.local.mydomain.com,mongodb2.local.mydomain.com,mongodb3.local.mydomain.com/my-database", array("timeout" => 10000, "replicaSet" => "rs-san-jose");

Hannes Magnusson

unread,
Feb 12, 2013, 2:23:30 PM2/12/13
to mongod...@googlegroups.com
It is alternating between success and failure because you are hitting
different worker processes (in your case, I guess you saw 16
connection failures..).
Each process needs to discover the fact the server is down by its own,
as the 1.3.x series of the driver doesn't use shared connection
pooling.
On the next request to the same process everything should be fine.
Keep in mind if your read preferences are to primary only, it will
take about a minute for a new primary to be elected and the app to
become fully functioning again.

We generally suggest MongoClient::RP_PRIMARY_PREFERRED to avoid such
scenarios, but you'll have to be aware that reads will go to a
secondary (with the lowest ping time) in the meanwhile, and you won't
be able to write anything (user session data for example).
Which secondary the read hits to can be "configured" by using
readPreferenceTags.

-Hannes

On Tue, Feb 12, 2013 at 10:59 AM, Justin <comm...@nodesocket.com> wrote:
> Hannes,
>
> Sorry for the delay in responding. I have 16 php-fpm worker processes. Ok,
> the logging is a lot, and contains sensitive details. What is strange, is
> if, I stop mongod on the primary so mongodb1, then keep on refreshing my
> app, it alternates between connection successful and an error, even though
> mongodb2 and mongodb3 are still up. I shouldn't see this downtime and
> behavior correct?
>
> Here is a snippet of logs which seem particularly interesting. Again, the
> logs contain sensitive details, my password, database names, hostnames, and
> IP addresses, so I replaced the sensitive details.
>
> Thanks a lot.
>
> https://gist.github.com/nodesocket/4772310
>
>
>
>
>
>
>
>
>
> On Sunday, February 3, 2013 7:38:10 PM UTC+1, Justin wrote:
>>
>> I am seeing a very strange bug using the PHP driver. I am running a three
>> member replica set, each member is running v2.2.3.
>>
>> If I take a member in the replica set offline, sometime the PHP client
>> fails to connect with:
>>
>> No candidate servers found
>>
>> The only way to fix it, is to restart the php-fpm worker, and magically it
>> goes back to normal and connects. Here is my PHP connection code:
>>
>> try {
>> $client = new
>> MongoClient("mongodb://username-here:passwo...@mongodb1.local.mydomain.com,mongodb2.local.mydomain.com,mongodb3.local.mydomain.com/my-database",
>> array("timeout" => 10000, "replicaSet" => "rs-san-jose");
>> catch(Exception $mongoException) {
>> print_r($mongoException);
>> }
>>
>> And each member in the replica set is binding to its local ip:
>>
>> bind_ip = 10.X.X.X
>> port = 27017
>>
>> Any ideas?
>>
>> Thanks.
>

Justin

unread,
Feb 12, 2013, 2:27:04 PM2/12/13
to mongod...@googlegroups.com
Hannes,

Thanks for the help, appreciated it. So, basically this means that until all php-fpm process fail once, connections will fail. That seems VERY counter-intuitive. Why can't the driver detect this, and not fail a request.


On Sunday, February 3, 2013 7:38:10 PM UTC+1, Justin wrote:
I am seeing a very strange bug using the PHP driver. I am running a three member replica set, each member is running v2.2.3.

If I take a member in the replica set offline, sometime the PHP client fails to connect with:

    No candidate servers found

The only way to fix it, is to restart the php-fpm worker, and magically it goes back to normal and connects. Here is my PHP connection code:

try {
    $client = new MongoClient("mongodb://username-here:password-here@mongodb1.local.mydomain.com,mongodb2.local.mydomain.com,mongodb3.local.mydomain.com/my-database", array("timeout" => 10000, "replicaSet" => "rs-san-jose");

Hannes Magnusson

unread,
Feb 12, 2013, 2:41:39 PM2/12/13
to mongod...@googlegroups.com
PHP is designed with "share nothing architecture", which means each
PHP process is a completely standalone entity with no knowledge of its
siblings and there is no communication between them.
This is what makes PHP so easy to scale out.

The unfortunate sideaffect of that system is the fact we can't really
communicate with the other PHP processes and tell them we detected
failed master. Each process needs to figure this out by it self.

We also cannot create a global ruleset that works for every usecase,
so we leave it in the hands of the developer to decide what to do in
this case - but we do offer you several possibilities with Read
Preferences.
You can easily attempt couple of times, or sleep for few microseconds,
or you may want to print out a "system currently in maintenance mode,
please click <here> to try again" to safe yourself couple of seconds
for the system to recover....
It really depends on what exactly the application is doing, and what
the expectations are.

$retry = 3;
$mc = null;
do {
try {
$mc = new MongoClient("host1,host2,host3", array("readPreference"
=> MongoClient::RP_PRIMARY_PREFERRED));
// Break out of the while loop if everything worked fine..
break;
} catch(MongoConnectionException $e) {
logException($e);
}
// Retry 3 times in case of minor interruptions
} while($retry--);
if (!$mc) {
// Our RS seems to be down! Emergency, emergency !
log("Something bad is going on!");
redirectToMaintenancePage();
}

Hope the explanation helps!

-Hannes

On Tue, Feb 12, 2013 at 11:27 AM, Justin <comm...@nodesocket.com> wrote:
> Hannes,
>
> Thanks for the help, appreciated it. So, basically this means that until all
> php-fpm process fail once, connections will fail. That seems VERY
> counter-intuitive. Why can't the driver detect this, and not fail a request.
>
>
> On Sunday, February 3, 2013 7:38:10 PM UTC+1, Justin wrote:
>>
>> I am seeing a very strange bug using the PHP driver. I am running a three
>> member replica set, each member is running v2.2.3.
>>
>> If I take a member in the replica set offline, sometime the PHP client
>> fails to connect with:
>>
>> No candidate servers found
>>
>> The only way to fix it, is to restart the php-fpm worker, and magically it
>> goes back to normal and connects. Here is my PHP connection code:
>>
>> try {
>> $client = new
>> MongoClient("mongodb://username-here:passwo...@mongodb1.local.mydomain.com,mongodb2.local.mydomain.com,mongodb3.local.mydomain.com/my-database",
>> array("timeout" => 10000, "replicaSet" => "rs-san-jose");
>> catch(Exception $mongoException) {
>> print_r($mongoException);
>> }
>>
>> And each member in the replica set is binding to its local ip:
>>
>> bind_ip = 10.X.X.X
>> port = 27017
>>
>> Any ideas?
>>
>> Thanks.
>

Justin

unread,
Feb 12, 2013, 2:56:26 PM2/12/13
to mongod...@googlegroups.com
Hannes,

I understand that PHP is share nothing, but it doesn't make sense, since every-time I call connect, it appears that the PHP driver is reading each server, detecting master, pinging, and doing the required work. It should be able to detect the new primary and connect to that, without failing one full request, and then working from them on.

The logs seem to indicate this behavior as well, lots of work to detect master, ping, auth, etc. The php processes shouldn't need to share anything, since they are all independent, and do this work on each connect.

Thanks again for the explanations.

On Sunday, February 3, 2013 7:38:10 PM UTC+1, Justin wrote:
I am seeing a very strange bug using the PHP driver. I am running a three member replica set, each member is running v2.2.3.

If I take a member in the replica set offline, sometime the PHP client fails to connect with:

    No candidate servers found

The only way to fix it, is to restart the php-fpm worker, and magically it goes back to normal and connects. Here is my PHP connection code:

try {
    $client = new MongoClient("mongodb://username-here:password-here@mongodb1.local.mydomain.com,mongodb2.local.mydomain.com,mongodb3.local.mydomain.com/my-database", array("timeout" => 10000, "replicaSet" => "rs-san-jose");

Hannes Magnusson

unread,
Feb 12, 2013, 3:19:21 PM2/12/13
to mongod...@googlegroups.com
We don't actually create new connections on each request, all
connections are persistent per process.
We iterate over the seedlist and look in our registry for these
connections, if we don't have one already we'll create it. If you
execute $mc = new MongoClient("host1,host2", array("replicaSet" =>
"RS")); 2 times in a row, we won't actually create two sets of
connections to all the hosts.

Because the connection is persistent we don't detect the failure until
we pickup the socket again and try to use it.
In many cases it makes sense to drop that socket and attempt to
reconnect, but I don't think we can make such an assumption that
_everyone_ wants that behaviour.
We therefore provide you with the means to implement the behaviour you
wish, be it a reconnect right away, wait a moment, or just log it and
move on.

-Hannes

On Tue, Feb 12, 2013 at 11:56 AM, Justin <comm...@nodesocket.com> wrote:
> Hannes,
>
> I understand that PHP is share nothing, but it doesn't make sense, since
> every-time I call connect, it appears that the PHP driver is reading each
> server, detecting master, pinging, and doing the required work. It should be
> able to detect the new primary and connect to that, without failing one full
> request, and then working from them on.
>
> The logs seem to indicate this behavior as well, lots of work to detect
> master, ping, auth, etc. The php processes shouldn't need to share anything,
> since they are all independent, and do this work on each connect.
>
> Thanks again for the explanations.
>
> On Sunday, February 3, 2013 7:38:10 PM UTC+1, Justin wrote:
>>
>> I am seeing a very strange bug using the PHP driver. I am running a three
>> member replica set, each member is running v2.2.3.
>>
>> If I take a member in the replica set offline, sometime the PHP client
>> fails to connect with:
>>
>> No candidate servers found
>>
>> The only way to fix it, is to restart the php-fpm worker, and magically it
>> goes back to normal and connects. Here is my PHP connection code:
>>
>> try {
>> $client = new
>> MongoClient("mongodb://username-here:passwo...@mongodb1.local.mydomain.com,mongodb2.local.mydomain.com,mongodb3.local.mydomain.com/my-database",
>> array("timeout" => 10000, "replicaSet" => "rs-san-jose");
>> catch(Exception $mongoException) {
>> print_r($mongoException);
>> }
>>
>> And each member in the replica set is binding to its local ip:
>>
>> bind_ip = 10.X.X.X
>> port = 27017
>>
>> Any ideas?
>>
>> Thanks.
>

Justin

unread,
Feb 12, 2013, 3:38:32 PM2/12/13
to mongod...@googlegroups.com
Hannes,

I will owe you a beer after all this help. :)

I just did further testing, and STILL seeing incorrect behavior. Started with all servers online (mongodb1, mongodb2, mongodb3). Mongodb1 as the primary. Shutdown mongod on mongodb1. Waited 3 minutes. Then refreshed a page which simply does:

<?php
    if(!MongoConnection::connect()) {
       echo 'MONGODB is DOWN!';
    } else {
       echo 'MONGODB is UP!';
    }
?>

I refreshed the page literally like 50 times, every-time I got MONGODB is DOWN. Here is my full custom MongoDB connection wrapper class. Perhaps you can bench and test on your end? Do you see any errors in my below custom MongoDB connection wrapper class.



On Sunday, February 3, 2013 7:38:10 PM UTC+1, Justin wrote:
I am seeing a very strange bug using the PHP driver. I am running a three member replica set, each member is running v2.2.3.

If I take a member in the replica set offline, sometime the PHP client fails to connect with:

    No candidate servers found

The only way to fix it, is to restart the php-fpm worker, and magically it goes back to normal and connects. Here is my PHP connection code:

try {
    $client = new MongoClient("mongodb://username-here:password-here@mongodb1.local.mydomain.com,mongodb2.local.mydomain.com,mongodb3.local.mydomain.com/my-database", array("timeout" => 10000, "replicaSet" => "rs-san-jose");

Hannes Magnusson

unread,
Feb 12, 2013, 5:38:36 PM2/12/13
to mongod...@googlegroups.com
I can't reproduce this....

This is the actual code I executed: http://pastebin.com/67QyhYVk
(just added some prints to your class incase it would have failed so I
knew what happened :)).

I ran it for few seconds, then killed the primary, and few seconds
later everything worked just fine again.

Can you tweak your fpm config to only have one process available to
guarantee that you are always hitting the same process (that will
remove some variables from the picture)?
Also, please test it from the command line like I just did to remove
all environment issues :)

-Hannes
>> MongoClient("mongodb://username-here:passwo...@mongodb1.local.mydomain.com,mongodb2.local.mydomain.com,mongodb3.local.mydomain.com/my-database",
>> array("timeout" => 10000, "replicaSet" => "rs-san-jose");
>> catch(Exception $mongoException) {
>> print_r($mongoException);
>> }
>>
>> And each member in the replica set is binding to its local ip:
>>
>> bind_ip = 10.X.X.X
>> port = 27017
>>
>> Any ideas?
>>
>> Thanks.
>

Justin

unread,
Feb 17, 2013, 12:16:26 PM2/17/13
to mongod...@googlegroups.com
Hannes,

Sorry for the delay, finally got some time to bench test. So I tried your script on the command line, and seeing something very strange. If I run:


And then stop mongod on the primary, I am NEVER seeing MongoDB [DOWN], i.e. MongoConnection::connect() always returns true. So very strange. However, if I uncomment the line to close the connection, and do the same test, this time I am seeing down for a few seconds, then back up.

Is this expected. In your example, you didn't call close(), yet yours worked.

I am truly confused. :) Thanks.

Hannes Magnusson

unread,
Feb 22, 2013, 6:55:40 PM2/22/13
to mongod...@googlegroups.com
I... Don't know what I can say.
You should never ever have to call ->close() manually, and its use is
super discouraged.

Pass in the debug option and paste the log from MongoLog, that should
clarify what is going on internally.

-Hannes
>> >> MongoClient("mongodb://username-here:passwo...@mongodb1.local.mydomain.com,mongodb2.local.mydomain.com,mongodb3.local.mydomain.com/my-database",

EastGhostCom

unread,
Feb 27, 2013, 5:50:35 AM2/27/13
to mongod...@googlegroups.com
We're seeing the exact same symptoms, messages from Mongo.
We made two side-by-side servers flip roles: primary became secondary, secondary became primary.
Nothing else changed anywhere with anything.
When our scripts connect to the former primary IP we get random errors, exactly as described above.
When our scripts connect to the new primary IP all errors vanish.

EastGhostCom

unread,
Feb 27, 2013, 6:01:53 AM2/27/13
to mongod...@googlegroups.com
Seeing all kinds of weird errors, scripts breaking in different places...
Restarting all apache procs on all servers so far seems to fix all problems, along with changing all IPs of former primary, to new primary, in all MongoClient __constructs.

EastGhostCom

unread,
Feb 27, 2013, 6:26:09 AM2/27/13
to mongod...@googlegroups.com
We did this on original primary
(original primary was member[0] and had priority 90 and votes 3)

cfg = rs.conf()
cfg.members[0].priority = 10
cfg.members[0].votes = 1
cfg.members[4[.priority = 90
cfg.members[4].votes = 3
rs,reconfig(cfg)

then all hell broke loose, as described.

Some combination of changing all the connection strings on all scripts to new primary IP (instead of former primary IP) and restarting all apaches on all servers seems to have regained sanity.  Not sure if culprit was due to old IP (tho driver docs say it doesn't matter which IP is reached, that any IP in the replica set will work) or due to latent / "pre-forked" apache procs retaining previously valid replicaset status.

EastGhostCom

unread,
Feb 27, 2013, 12:42:47 PM2/27/13
to mongod...@googlegroups.com
Is this then an unofficial gotcha ? -- it appears to be not just a matter of the failover time of mongo itself, but also crucial is the lifetime of all idle / waiting apache procs.  Shouldn't the drivers in the idle / waiting apache procs automatically (and fairly instantly) update upon change-up of primary ?

Hannes Magnusson

unread,
Feb 27, 2013, 3:52:07 PM2/27/13
to mongod...@googlegroups.com
I don't quite understand your setup.
You have a replicaset running, but in your connection string you pass
to MongoClient() you only pass in one server? And that was the "old
primary"?

Please create a new thread and explain your infrastructure, which
versions you are using (driver and mongod), paste your ReplicaSet
config, MongoClient arguments and what exactly you tested along with
the error messages.

All the idling/waiting apache processess will discover the ReplicaSet
topology change on the first request coming into that process, granted
you have reached the configured mongo.is_master_interval, or on the
first failed write.

-Hannes

EastGhostCom

unread,
Mar 7, 2013, 4:27:31 AM3/7/13
to mongod...@googlegroups.com
correct - but it doesn't matter which replicaset member we pass in as the hint - they all fail, about 20% of the time during normal use -- "no candidate servers found" -- and 100% of the time after some period after firefox browser sits idle (maybe 20 minutes?)

solution seems to be putting all replicaset members in the /etc/hosts file.  still testing, but this seems to have fixed 100% of all "no candidate servers" found problem

Hannes Magnusson

unread,
Mar 7, 2013, 4:04:20 PM3/7/13
to mongod...@googlegroups.com
It sounds like you are experiencing lot of failovers in your
replicaset for some reason.

In anycase, you should *always* provide more then one server in the
connection string when connecting to a ReplicaSet.

If you only pass in one server, and that server goes down.. There is
nothing the driver can do.

-Hannes

Justin

unread,
Mar 7, 2013, 4:13:21 PM3/7/13
to mongod...@googlegroups.com
Hannes,

According to the official PHP manual on the constructor

// you only need to pass a single seed, the driver will derive the full list and
// find the primary from this seed
$m2 = new MongoClient("mongodb://ny1.example.com", array("replicaSet" => "myReplSet"));

So, what you said doesn't appear to be accurate.

Hannes Magnusson

unread,
Mar 7, 2013, 4:18:57 PM3/7/13
to mongod...@googlegroups.com
Its a bad example, I'll fix it in a minute. The example above it is
the one you should be following.

If you only pass in one server in the connection string, and that
server goes down, we really have no way of establishing connection to
the other members of the replicaset.
We strongly recommend you pass in atleast two members of the
replicaset, and if you have multiple datacenters, then one member in
each of those datacenters.

-Hannes

Justin

unread,
Mar 10, 2013, 12:12:45 PM3/10/13
to mongod...@googlegroups.com
Hannes,

I think I may be narrowed down the bug. On our web1 server which only has 1 php-fpm pool, things work as expected. But our web2 and web3 servers with multiple php-fpm pools, require restarting php-fpm when the primary MongoDB server changes.

Maybe you can bench this?

Thanks.

EastGhostCom

unread,
Mar 14, 2013, 10:37:28 AM3/14/13
to mongod...@googlegroups.com
Confirmed - a week after adding all replicaset members to the /etc/hosts of all replicaset member machines, 99.999% of "no candidate servers found" errors have ceased.

Norganna's AddOns

unread,
Sep 8, 2013, 11:35:39 PM9/8/13
to mongod...@googlegroups.com
For some reason, the php-fpm server just keeps the old SECONDARY around for ages and ages (it doesn't see it ever get promoted to PRIMARY) thus when you dump $mongo->getConnections(), you just see a whole heap of SECONDARY servers.

Here is the dump of a test that I run which has debugging on during this situation (it never recovers from this, even hours after the switch has happened, it still has the server listed as SECONDARY even though it's definitely been promoted to PRIMARY):


Anyhow, for anyone else in this situation, I've adjusted the connection code on my servers to account for this and to close and retry connections if it detects this situation happening:


Hope this helps.

Justin

unread,
Sep 10, 2013, 11:26:31 PM9/10/13
to mongod...@googlegroups.com
I also confirm this. I have to restart all php-fpm services if the MongoDB master changes. This is unacceptable, and we shouldn't have to code this ourself like Norganna's AddOns.

This seems like a bug in the PHP driver.

Derick Rethans

unread,
Sep 11, 2013, 7:02:11 AM9/11/13
to mongod...@googlegroups.com
On Tue, 10 Sep 2013, Justin wrote:

> I also confirm this. I have to restart all php-fpm services if the MongoDB
> master changes. This is unacceptable, and we shouldn't have to code this
> ourself like Norganna's AddOns.
>
> This seems like a bug in the PHP driver.

Which version are you using?

cheers,
Derick

--
{
website: [ "http://mongodb.org", "http://derickrethans.nl" ],
twitter: [ "@derickr", "@mongodb" ]
}

Justin

unread,
Sep 12, 2013, 1:23:14 PM9/12/13
to mongod...@googlegroups.com
Hey Derick,

-- MongoDB --
db version v2.4.6
Thu Sep 12 10:21:49.192 git version: b9925db5eac369d77a3a5f5d98a145eaaacd9673

-- PHP Driver --
Version => 1.4.3

Thanks.

Paolo Mainardi

unread,
Sep 27, 2013, 6:29:23 AM9/27/13
to mongod...@googlegroups.com
I confirm the same exact bug, as with Apache (less frequent) as with php-fpm, right now i have this kind of situation on Mongod server side:

{ "set" : "drupalSet",
  "date" : { "$date" : "Fri Sep 27 12:12:04 2013" },
  "myState" : 2,
  "syncingTo" : "mongodb03.dcluster:27017",
  "members" : [ 
    { "_id" : 0,
      "name" : "mongodb01.dcluster:27017",
      "health" : 1,
      "state" : 2,
      "stateStr" : "SECONDARY",
      "uptime" : 29778,
      "optime" : { "$timestamp" : { "t" : 1380276683, "i" : 3 } },
      "optimeDate" : { "$date" : "Fri Sep 27 12:11:23 2013" },
      "errmsg" : "syncing to: mongodb03.dcluster:27017",
      "self" : true }, 
    { "_id" : 1,
      "name" : "mongodb02.dcluster:27017",
      "health" : 1,
      "state" : 2,
      "stateStr" : "SECONDARY",
      "uptime" : 10,
      "optime" : { "$timestamp" : { "t" : 1380276683, "i" : 3 } },
      "optimeDate" : { "$date" : "Fri Sep 27 12:11:23 2013" },
      "lastHeartbeat" : { "$date" : "Fri Sep 27 12:12:02 2013" },
      "lastHeartbeatRecv" : { "$date" : "Fri Sep 27 12:12:02 2013" },
      "pingMs" : 0,
      "lastHeartbeatMessage" : "syncing to: mongodb01.dcluster:27017",
      "syncingTo" : "mongodb01.dcluster:27017" }, 
    { "_id" : 2,
      "name" : "mongodb03.dcluster:27017",
      "health" : 1,
      "state" : 1,
      "stateStr" : "PRIMARY",
      "uptime" : 94,
      "optime" : { "$timestamp" : { "t" : 1380276683, "i" : 3 } },
      "optimeDate" : { "$date" : "Fri Sep 27 12:11:23 2013" },
      "lastHeartbeat" : { "$date" : "Fri Sep 27 12:12:02 2013" },
      "lastHeartbeatRecv" : { "$date" : "Fri Sep 27 12:12:02 2013" },
      "pingMs" : 0,
      "syncingTo" : "mongodb01.dcluster:27017" } ],
  "ok" : 1 }

This happened after i powered off "mongodb.dcluster01" that was the old elected primary and the used one by php5-pfm to connect.

Now, after 5 minutes i have php-side: 

string(51) "Couldn't get connection: No candidate servers found"

The unique solutions to refresh connection pools is to restart php-fpm workers, this is unacceptable.

My configuration:

vagrant@fe01:~$ pecl list
Installed packages, channel pecl.php.net:
=========================================
Package Version State
mongo   1.4.4   stable

vagrant@fe01:~$ php -v
PHP 5.3.10-1ubuntu3.8 with Suhosin-Patch (cli) (built: Sep  4 2013 20:00:51)
Copyright (c) 1997-2012 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2012 Zend Technologies

vagrant@mongodb03:~$ mongod -version
db version v2.4.6
Fri Sep 27 12:29:12.064 git version: b9925db5eac369d77a3a5f5d98a145eaaacd9673


Paolo Mainardi

unread,
Sep 27, 2013, 6:37:23 AM9/27/13
to mongod...@googlegroups.com
I forgot to paste mongodb configuration:

vagrant@fe01:~$ cat /etc/php5/fpm/conf.d/mongodb.ini
; File Managed by Puppet

extension=mongo.so
mongo.ping_interval = 1
mongo.is_master_interval = 1


On Thursday, September 12, 2013 7:23:14 PM UTC+2, Justin wrote:

Paolo Mainardi

unread,
Sep 27, 2013, 10:01:01 AM9/27/13
to mongod...@googlegroups.com
Some logs:

Mongo:

{ "set" : "drupalSet",
  "date" : { "$date" : "Fri Sep 27 15:59:11 2013" },
  "myState" : 2,
  "syncingTo" : "mongodb03.dcluster:27017",
  "members" : [ 
    { "_id" : 0,
      "name" : "mongodb01.dcluster:27017",
      "health" : 0,
      "state" : 8,
      "stateStr" : "(not reachable/healthy)",
      "uptime" : 0,
      "optime" : { "$timestamp" : { "t" : 1380285910, "i" : 3 } },
      "optimeDate" : { "$date" : "Fri Sep 27 14:45:10 2013" },
      "lastHeartbeat" : { "$date" : "Fri Sep 27 15:59:11 2013" },
      "lastHeartbeatRecv" : { "$date" : "Fri Sep 27 14:46:39 2013" },
      "pingMs" : 0 }, 
    { "_id" : 1,
      "name" : "mongodb02.dcluster:27017",
      "health" : 1,
      "state" : 2,
      "stateStr" : "SECONDARY",
      "uptime" : 4365,
      "optime" : { "$timestamp" : { "t" : 1380285910, "i" : 3 } },
      "optimeDate" : { "$date" : "Fri Sep 27 14:45:10 2013" },
      "self" : true }, 
    { "_id" : 2,
      "name" : "mongodb03.dcluster:27017",
      "health" : 1,
      "state" : 1,
      "stateStr" : "PRIMARY",
      "uptime" : 4365,
      "optime" : { "$timestamp" : { "t" : 1380285910, "i" : 3 } },
      "optimeDate" : { "$date" : "Fri Sep 27 14:45:10 2013" },
      "lastHeartbeat" : { "$date" : "Fri Sep 27 15:59:11 2013" },
      "lastHeartbeatRecv" : { "$date" : "Fri Sep 27 15:59:09 2013" },
      "pingMs" : 0,
      "syncingTo" : "mongodb01.dcluster:27017" } ],
  "ok" : 1 }

PHP:

2013/09/27 15:58:47 [error] 2089#0: *133 FastCGI sent in stderr: "PHP message: PHP Notice:  CON     INFO: mongo_get_read_write_connection: finding a REPLSET connection (read) in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  CON     FINE: found connection mongodb01.dcluster:27017;drupalSet;.;2246 (looking for mongodb01.dcluster:27017;drupalSet;.;2246) in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  CON     FINE: is_ping: skipping: last ran at 1380290327, now: 1380290327, time left: 1 in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  CON     WARN: Couldn't connect to 'mongodb01.dcluster:27017': Previous connection attempts failed, server blacklisted in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  CON     FINE: found connection mongodb02.dcluster:27017;drupalSet;.;2246 (looking for mongodb02.dcluster:27017;drupalSet;.;2246) in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  CON     FINE: is_ping: skipping: last ran at 1380290327, now: 1380290327, time left: 1 in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  CON     FINE: found connection mongodb02.dcluster:27017;drupalSet;.;2246 (looking for mongodb02.dcluster:27017;drupalSet;.;2246) in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  CON     FINE: is_ping: skipping: last ran at 1380290327, now: 1380290327, time left: 1 in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  CON     FINE: discover_topology: checking ismaster for mongodb01.dcluster:27017;drupalSet;.;2246 in /var/www/test-
2013/09/27 15:58:47 [error] 2089#0: *133 FastCGI sent in stderr: "in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  REPLSET FINE: - all servers in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  REPLSET FINE: filter_connections: adding connections: in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  REPLSET FINE: - connection: type: SECONDARY, socket: 42, ping: 1, hash: mongodb03.dcluster:27017;drupalSet;.;2246 in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  REPLSET FINE: - connection: type: SECONDARY, socket: 42, ping: 0, hash: mongodb02.dcluster:27017;drupalSet;.;2246 in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  REPLSET FINE: filter_connections: done in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  REPLSET FINE: limiting to servers with same replicaset name in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  REPLSET FINE: - connection: type: SECONDARY, socket: 42, ping: 1, hash: mongodb03.dcluster:27017;drupalSet;.;2246 in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  REPLSET FINE: - connection: type: SECONDARY, socket: 42, ping: 0, hash: mongodb02.dcluster:27017;drupalSet;.;2246 in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  REPLSET FINE: limiting to servers with same replicaset name: done in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  REPLSET FINE: limiting by credentials
2013/09/27 15:58:47 [error] 2089#0: *133 FastCGI sent in stderr: "che.inc on line 29
PHP message: PHP Notice:  REPLSET FINE: - connection: type: SECONDARY, socket: 42, ping: 1, hash: mongodb03.dcluster:27017;drupalSet;.;2246 in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  REPLSET FINE: selecting near server: done in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  REPLSET INFO: pick server: random element 0 in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  REPLSET INFO: - connection: type: SECONDARY, socket: 42, ping: 0, hash: mongodb02.dcluster:27017;drupalSet;.;2246 in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  IO      FINE: getting reply in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  IO      FINE: getting cursor header in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29
PHP message: PHP Notice:  IO      FINE: getting cursor body in /var/www/test-app/sites/all/modules/contrib/mongodb/mongodb_cache/mongodb_cache.inc on line 29" while reading response header from upstream, client: 10.0.1.1, server: dcluster-test.twinbitlabs.com, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "fe01.local:90", referrer: "http://fe01.local:90/views/banners"

Now the application is hanged, i have 2 solutions to make it works again:

1) Close the connections by php side
2) Restart php5-fpm pool

Josh Stuart

unread,
Apr 26, 2014, 10:57:08 PM4/26/14
to mongod...@googlegroups.com
I'm experiencing this issue also:

Steps to reproduce:

1. create a local replica set with 3 nodes eg (in linux):
mongod --port 27017 --dbpath /mnt/data/db/rs0-0 --logpath /mnt/data/log/mongo-0.log --keyFile /srv/mongodb/mongodb-rs.key --replSet rs0 --smallfiles --oplogSize 128 --auth --fork
mongod --port 27018 --dbpath /mnt/data/db/rs0-1 --logpath /mnt/data/log/mongo-1.log --keyFile /srv/mongodb/mongodb-rs.key --replSet rs0 --smallfiles --oplogSize 128 --auth --fork
mongod --port 27019 --dbpath /mnt/data/db/rs0-2 --logpath /mnt/data/log/mongo-2.log --keyFile /srv/mongodb/mongodb-rs.key --replSet rs0 --smallfiles --oplogSize 128 --auth --fork
mongo admin --port 27017 --eval 'rs.initiate({"_id": "rs0", "version": 1, "members": [{"_id": 0, "host": "localhost:27017"},{"_id": 1, "host": "localhost:27018"},{"_id": 2, "host": "localhost:27019"}]})'

2. connect via php eg. 
new MongoClient('mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs0');
3. shutdown the primary
4. connect via php again and you will get: No candidate servers found

Even if you have:

new MongoClient('mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs0&readPreference=primaryPreferred');

You will still get random errors when trying to write data.

Does anyone have a valid work around or fix yet. As people have mentioned, it's unacceptable to have to restart processes in order to overcome these errors.



On Monday, February 4, 2013 5:38:10 AM UTC+11, Justin wrote:
I am seeing a very strange bug using the PHP driver. I am running a three member replica set, each member is running v2.2.3.

If I take a member in the replica set offline, sometime the PHP client fails to connect with:

    No candidate servers found

The only way to fix it, is to restart the php-fpm worker, and magically it goes back to normal and connects. Here is my PHP connection code:

try {
    $client = new MongoClient("mongodb://username-here:password-here@mongodb1.local.mydomain.com,mongodb2.local.mydomain.com,mongodb3.local.mydomain.com/my-database", array("timeout" => 10000, "replicaSet" => "rs-san-jose");

Josh Stuart

unread,
Apr 26, 2014, 11:13:40 PM4/26/14
to mongod...@googlegroups.com
My apologies! I just updated to v1.5.1 and it seems that it is now working as expected.

Asya Kamsky

unread,
Apr 28, 2014, 12:20:17 AM4/28/14
to mongodb-user
No need to apologize, Josh!

For other folks who are having the same issue - are you able to try the latest (1.5) and let us know whether that resolves the issue?

Asya



--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.

---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.

Justin

unread,
May 6, 2014, 4:56:12 PM5/6/14
to mongod...@googlegroups.com
Just upgraded to MongoDB 2.6 and we switch the primary in our replica-set. The PHP driver worked as expected with a standard Mongo connection (auto switched to the new primary), but when we tried:

$mongo_db = $mongo_client->selectDB("database-here");
$mongo_db->getGridFS();

This failed, until we restarted php-fpm. Seems to be the same behavior as before. Should'nt the MongoDB PHP driver be able to recover from a primary switch and connect to GridFS without restarting php-fpm?

We are running the latest Mongo PHP extension as well: 1.5.1.

Justin

unread,
May 6, 2014, 5:08:08 PM5/6/14
to mongod...@googlegroups.com
Sorry, the GridFS code I provided was incomplete:

$mongo_db = $mongo_client->selectDB("database-here");
$mongo_grid_fs = $mongo_db->getGridFS();

try {
    return MongoConnection::$mongo_grid_fs->count();
 } catch(Exception $MongoException) {
    die("Temporarily unable to process request. Failed to establish a connection with MongoDB GridFS. Please retry.");
}

We hit the die() condition, until restarting php-fpm. Thanks.

Derick Rethans

unread,
May 7, 2014, 4:41:50 AM5/7/14
to mongod...@googlegroups.com
On Tue, 6 May 2014, Justin wrote:

> Just upgraded to MongoDB 2.6 and we switch the primary in our
> replica-set. The PHP driver worked as expected with a standard Mongo
> connection (auto switched to the new primary), but when we tried:
>
> $mongo_db = $mongo_client->selectDB("database-here");
> $mongo_db->getGridFS();
>
> This failed, until we restarted php-fpm. Seems to be the same behavior as
> before. Should'nt the MongoDB PHP driver be able to recover from a primary
> switch and connect to GridFS without restarting php-fpm?
>
> We are running the latest Mongo PHP extension as well: 1.5.1.

Although that is not the latest anymore, I think you are running into
"expected behavior". The PHP driver does not check on every query
whether there is a new primary, that would be way too slow. Instead on
average, it does replicaset discovery about every 15 seconds, but this
is something you can tune by changing mongo.is_master_interval
(http://docs.php.net/manual/en/mongo.configuration.php#ini.mongo.is-master-interval).
If you see any other behaviour, let us know and we can investigate.

Please note that there is no difference between standard mongo
connections, and ones to gridfs. GridFs is just a layer on top of (two)
normal collections.

Justin

unread,
May 7, 2014, 5:18:36 PM5/7/14
to mongod...@googlegroups.com
Derick,

We waited like 10 minutes, shouldn't it have discovered the new master by then? Until we forcefully restarted php-fpm is was down.

Derick Rethans

unread,
May 8, 2014, 6:04:34 AM5/8/14
to mongod...@googlegroups.com
On Wed, 7 May 2014, Justin wrote:

> We waited like 10 minutes, shouldn't it have discovered the new master
> by then? Until we forcefully restarted php-fpm is was down.

In that case, it should have discovered it. Is this something you can
easily reproduce?

Justin

unread,
May 8, 2014, 5:46:20 PM5/8/14
to mongod...@googlegroups.com
Derick,

Not easily tested, we are in production. However, if you setup a replica set, and then setup php-fpm. Then switch the primary and try connecting to GridFS, something like:

//Standard MongoDB connection first
$mongo_db = $mongo_client->selectDB("database-here");
$mongo_grid_fs = $mongo_db->getGridFS();

try {
    return $mongo_grid_fs->count();
 } catch(Exception $MongoException) {
    die("Temporarily unable to process request. Failed to establish a connection with MongoDB GridFS. Please retry.");
}

You should see it hit the catch() always, until you restart php-fpm.

Gleb Burov

unread,
Apr 22, 2015, 8:53:41 AM4/22/15
to mongod...@googlegroups.com
Got this error with php-driver 1.6.6. in production
When many update queries (1000 per sec / writeConcern=0), smth goes slow. Response on query can be 1, 2, 3,... secs (whole number)
Connection hostnames: mongodb://cs18632:27017,cs18633:27017,cs19096:27017/?replicaSet=rs0&readPreference=primaryPreferred

Actually there are several random errors (7-10 per sec):
- Couldn't get connection: No candidate servers found
- Unable to connect to MongoDB server at No candidate servers found
- Unable to connect to MongoDB server at MongoClient::connect(): php_network_getaddresses: getaddrinfo failed: Name or service not known.

MongoLog warns:
- connection_create: error while creating connection for cs19096:27017: Connection timed out
- Couldn't connect to 'cs19096:27017': Connection timed out
- discover_topology: could not connect to new host: cs19096:27017: Previous connection attempts failed, server blacklisted

mongostats:
writelock ~4% on updated db
connections on primary ~40
connections on secondaries ~1000

Enrico Cornacchini

unread,
Dec 11, 2015, 6:27:43 PM12/11/15
to mongodb-user
Same problem connection with php-driver, work fine if i connect with mongo shell.

Any solutions?

E

Wan Bachtiar

unread,
Dec 13, 2015, 5:28:22 PM12/13/15
to mongodb-user

Hi Enrico,

Although you may be seeing a similar error message, it may be a different issue that is causing it.

Please start a new discussion with relevant details of:

  • PHP driver version.
  • MongoDB version and deployment topology (standalone, replica set, sharded cluster)
  • Snippet of the connection PHP code.

Thanks.

Wan.

Reply all
Reply to author
Forward
This conversation is locked
You cannot reply and perform actions on locked conversations.
0 new messages