Sockets not being closed correctly with php extension

[Jeff Minard]

Hiya!

I'm using the php couchbase extension (see version at bottom of email) and I'm running into a rather troublesome error, namely it's causing my system to run out of file descriptors.

If I run this little command here:

lsof | grep httpd | awk '{print $9}' | sort | uniq -c | sort -rn | head -n 15;

I can watch the count just sky rocket as I click around my site (each page load always does at least one connect, using couchbase as a session store.)

Has any one else seen this? I would think that for some reason, a shutdown hook isn't triggering and closing file descriptors (sockets?) or something weird, but I don't really know how to diagnose this further. Pertinent details: centos5.5, apache2.2, mod_php, php5.3.

I have confirmed that a simple script like this, when executed through hitting the apache webpage will cause the "socket" value to go up by 2 and eventpoll up by 1. (Makes sense, send/rec socket, and one poller per connection). Running this script from the CLI does not cause the count to become incremented permanently.

<?php

$couchbaseClient = new Couchbase('127.0.0.1:8091', 'admin', 'password', 'default');

echo 'fsupandaway!'; 

What else can I do to help diagnose and fix the problem? Thanks!

php.ini information:

couchbase

couchbase support => enabled

version => 1.1.0-dp5

json support => yes

fastlz support => no

zlib support => yes

Directive => Local Value => Master Value

couchbase.compression_factor => 1.3 => 1.3

couchbase.compression_threshold => 2000 => 2000

couchbase.compressor => none => none

couchbase.serializer => php => php

[Matt Ingenthron]

Hi Jeff,

See inline…

On 11/3/12 7:15 PM, "Jeff Minard" <je...@jrm.cc> wrote:

Hiya!

I'm using the php couchbase extension (see version at bottom of email) and I'm running into a rather troublesome error, namely it's causing my system to run out of file descriptors.

If I run this little command here:

lsof | grep httpd | awk '{print $9}' | sort | uniq -c | sort -rn | head -n 15;

I can watch the count just sky rocket as I click around my site (each page load always does at least one connect, using couchbase as a session store.)

Has any one else seen this? I would think that for some reason, a shutdown hook isn't triggering and closing file descriptors (sockets?) or something weird, but I don't really know how to diagnose this further. Pertinent details: centos5.5, apache2.2, mod_php, php5.3.

I have confirmed that a simple script like this, when executed through hitting the apache webpage will cause the "socket" value to go up by 2 and eventpoll up by 1. (Makes sense, send/rec socket, and one poller per connection). Running this script from the CLI does not cause the count to become incremented permanently.

<?php

$couchbaseClient = new Couchbase('127.0.0.1:8091', 'admin', 'password', 'default');

echo 'fsupandaway!'; 

What else can I do to help diagnose and fix the problem? Thanks!

If you could, please file an issue at couchbase.com/issues, as this sure seems to be a bug.  There is likely something we're not closing down appropriately.  A workaround would be to shorten the time the Apache processes live, but we should fix whatever the main cause is.

One note, the 2nd and 3rd position parameters there above should be the ones for the bucket, or "default" and "" in the case of the default bucket.  It will work with admin permissions, but it's not necessary.

Could you also try something?  If you add a ,true as the last positional parameter? If you do then you'll be using persistent connections. This may make things better.

Thanks,

Matt

php.ini information:

couchbase

couchbase support => enabled

version => 1.1.0-dp5

json support => yes

fastlz support => no

zlib support => yes

Directive => Local Value => Master Value

couchbase.compression_factor => 1.3 => 1.3

couchbase.compression_threshold => 2000 => 2000

couchbase.compressor => none => none

couchbase.serializer => php => php

-- 

Matt Ingenthron - Director, Developer Solutions

Couchbase, Inc.

[Jeff Minard]

If you could, please file an issue at couchbase.com/issues, as this sure seems to be a bug.  There is likely something we're not closing down appropriately.  A workaround would be to shorten the time the Apache processes live, but we should fix whatever the main cause is.

Will do. I figured I'd check here first.

 

One note, the 2nd and 3rd position parameters there above should be the ones for the bucket, or "default" and "" in the case of the default bucket.  It will work with admin permissions, but it's not necessary.

What you're saying doesn't really jive with the docs (http://www.couchbase.com/docs/couchbase-sdk-php-1.1/api-reference-connection.html). I only put "default" here for the example. In my actual code I have different buckets being accessed. (Maybe the docs are out of date for the "next" version, since this URL (http://www.couchbase.com/develop/php/next) has examples like what you're mentioning.)

 

Could you also try something?  If you add a ,true as the last positional parameter? If you do then you'll be using persistent connections. This may make things better.

Yeah, it helps a bit, but it seems to make less connections (ie, a pooled connection is good for a little while) but even the pooled ones aren't being freed up, so it just grows slower.

Thanks for the tips (the Apache tweak on MaxRequestsPerChild does also work to keep the number down but, as noted, is just a workaround.)

 

[Matt Ingenthron]

On 11/3/12 11:00 PM, "Jeff Minard" <je...@jrm.cc> wrote:

If you could, please file an issue at couchbase.com/issues, as this sure seems to be a bug.  There is likely something we're not closing down appropriately.  A workaround would be to shorten the time the Apache processes live, but we should fix whatever the main cause is.

Will do. I figured I'd check here first.

Yep, glad you did, thanks.

 

One note, the 2nd and 3rd position parameters there above should be the ones for the bucket, or "default" and "" in the case of the default bucket.  It will work with admin permissions, but it's not necessary.

What you're saying doesn't really jive with the docs (http://www.couchbase.com/docs/couchbase-sdk-php-1.1/api-reference-connection.html). I only put "default" here for the example. In my actual code I have different buckets being accessed. (Maybe the docs are out of date for the "next" version, since this URL (http://www.couchbase.com/develop/php/next) has examples like what you're mentioning.)

We'll get that corrected.  Indeed, the couchbase.com/docs connection info needs to be fixed.  It won't affect what you're doing here, but it's better to have things using the correct arguments.

 

Could you also try something?  If you add a ,true as the last positional parameter? If you do then you'll be using persistent connections. This may make things better.

Yeah, it helps a bit, but it seems to make less connections (ie, a pooled connection is good for a little while) but even the pooled ones aren't being freed up, so it just grows slower.

Okay, should be straightforward to find in this case.  Sorry for the trouble.  

Thanks for the tips (the Apache tweak on MaxRequestsPerChild does also work to keep the number down but, as noted, is just a workaround.)

 

-- 

Matt Ingenthron

Couchbase, Inc.

[Jeff Minard]

For anybody following this, you can continue doing so over at the bug now:

http://www.couchbase.com/issues/browse/PCBC-140