(tcp 11211) failed with: Connection timed out (110)

[the_fonz]

Guys,

We are seeing tons of these messages reported;

Memcache::get(): Server 10.3.230.15 (tcp 11211) failed with:
Connection timed out (110)

They are reported on all six of our webservers. Memcache is working
but lots of connections are timing out as reported with the above
messages.

We are running memcached 1.2.5 with php-pecl-memcache-2.2.3 (we also
tried with memcache 1.4.5 but had the same errors). We have PHP 5.2.6
running on x86 Hardware with Red Hat 5.5.

We have six web servers all running Apache 2.2 on prefork mode.
Prefork MaxClients is set to 192.

Our memcached config looks like this;

PORT="11211"
USER="memcached"
MAXCONN="2048"
CACHESIZE="2048"
OPTIONS=""

We are running memcache between webservers on a LAN, no Firewall or
iptables being used.

Memcache is used to cache output of scripts most of the time. It could
be HTML or XML that is delivered to client.

Our tcp values look like this;

tcp_fin_timeout
60
tcp_max_orphans
65536
tcp_orphan_retries
0
tcp_keepalive_probes
9
tcp_keepalive_time
7200

I am totally out of ideas, I can't see any dropped packets on the
network, I just don't know what else to check.

Any ideas as I am pulling my hair out!

Thanks

[dormando]

Upgrade back to 1.4.5 and look for the 'listen_disabled_num' value in the
stats output. If the number is increasing, you're hitting maxconns.

If not, you're probably seeing packet loss, or have a firewall in the way
that's maxing out.

[Patrick Galbraith]

the_fonz wrote:
> Guys,
>
> We are seeing tons of these messages reported;
>
> Memcache::get(): Server 10.3.230.15 (tcp 11211) failed with:
> Connection timed out (110)
>
> They are reported on all six of our webservers. Memcache is working
> but lots of connections are timing out as reported with the above
> messages.
>
> We are running memcached 1.2.5 with php-pecl-memcache-2.2.3 (we also
> tried with memcache 1.4.5 but had the same errors). We have PHP 5.2.6
> running on x86 Hardware with Red Hat 5.5.
>
> We have six web servers all running Apache 2.2 on prefork mode.
> Prefork MaxClients is set to 192.
>

Fonz,

Ayye Cunningham!! (ok, I couldn't help it!)

Can you telnet port 11211 on each of your servers? If you write a simple
command-line script, can it connect?

Also, maybe try PECL/memcached?

--Patrick

[Brian Moon]

On 9/7/10 10:50 AM, the_fonz wrote:
> We are running memcached 1.2.5 with php-pecl-memcache-2.2.3 (we also
> tried with memcache 1.4.5 but had the same errors). We have PHP 5.2.6
> running on x86 Hardware with Red Hat 5.5.
>
> We have six web servers all running Apache 2.2 on prefork mode.
> Prefork MaxClients is set to 192.

Are you using persistent connections?

--

Brian.
--------
http://brian.moonspot.net/

[Tim Stockford]

Sorry, my memcache knowledge is about two weeks old (am learning though!)

Do you mean are we pooling connections? No we are not.

PHP should just drop the connection once it's finished with it. there is no "timeout" or "close connection" set within PHP.


> Date: Tue, 7 Sep 2010 11:19:18 -0500
> From: br...@moonspot.net
> To: memc...@googlegroups.com
> CC: tim_st...@hotmail.com
> Subject: Re: (tcp 11211) failed with: Connection timed out (110)

[the_fonz]

We reverted back to 1.4.5. Still get continuous timed out connections
with no increase in listen_disabled_num

I read http://brian.moonspot.net/php-memcached-issues

I see that PECL/memcached is buggy for persistent connections but am
wondering if I should enable persistent connections with
php-pecl-memcache-2.2.3, would this speed things up?

Is it worth trying PECL/memcached without persistent connections?

Thanks

On Sep 7, 5:40 pm, Tim Stockford <tim_stockf...@hotmail.com> wrote:
> Sorry, my memcache knowledge is about two weeks old (am learning though!)
>
> Do you mean are we pooling connections? No we are not.
>
> PHP should just drop the connection once it's finished with it. there is no "timeout" or "close connection" set within PHP.
>
> > Date: Tue, 7 Sep 2010 11:19:18 -0500
> > From: br...@moonspot.net
> > To: memc...@googlegroups.com

> > CC: tim_stockf...@hotmail.com

[Brian Moon]

On 9/9/10 10:20 AM, the_fonz wrote:
> We reverted back to 1.4.5. Still get continuous timed out connections
> with no increase in listen_disabled_num
>
> I read http://brian.moonspot.net/php-memcached-issues
>
> I see that PECL/memcached is buggy for persistent connections but am
> wondering if I should enable persistent connections with
> php-pecl-memcache-2.2.3, would this speed things up?
>
> Is it worth trying PECL/memcached without persistent connections?

I run PECL/memcache with persistent connections always.

If you dig through the comments on my blog post there is a work around
for the persistent connection issue in PECL/memcached. But, there are
other issues with that library that will hopefully be fixed soon.

[Patrick Galbraith]

Fonz,

I would certainly give PECL/memcached a test if you can. I'm in the
process (taking longer than I intended) to upgrade libmemcached that
PECL/memcached is built against, so that might end up fixing the buggy
issues.

--Patrick

[moses wejuli]

hi brian,

i notice this thread was last active over a year ago.. so i thought it was about time for a recap (in light of the fact that documentation for pecl/memcache and pecl/memcached is sometimes not always up-to-date) --

1) has the persistent connections bug in pecl/memcached been fixed , and if so, is it now stable (i.e. wud u recommend using it in a production environment)? what release/version?
2) in the past, you've recommended using pecl/memcache 2.2.x  for persistent connections with a workaround -- is this still the case now? is the workaround still necessary? does that workaround now address the maxing out of memory?
3) persistent connections are generally recommended but can usually cause problems with buggy client implementations as is evident with the php clients.. do u still recommend them?
4) from some of your tests, what is the performnce gap between using persistent and non-persistent connections in either client? is it large enough to warrant a trade in of a larger memory footprint by persistent connections for the performance gain? put another way: would you absolutely *not* recommend non-persistent connections from a performance standpoint?
5) lastly, and slightly off topic (apologies) but relevant, knowing as i do that you use mysql, wud you apply similar logic to connections in mysql using PDO? feel free not to bother with this as it's not the right forum :).. just though i wud drop it in..

cheers,

-m.