cannot assign requested address (99)

[Ilan Berkner]

We have a very busy high traffic site and recently started using memached to store frequently accessed queries.  We put memcached on each php application server so that the connection is local to the system.

Today, during one of our peek times, we started getting this error and had to disable the caching of queries:

[20-Oct-2010 12:05:57] PHP Warning: Memcache::connect() [<a href='memcache.connect'>memcache.connect</a>]: Can't connect to 127.0.0.1:11211, Cannot assign requested address (99) 

In searching online for this error, I came across several articles, but nothing that I though was relevant to our case.  I tried increasing the number of connections from 1024 (default) to 5000 but the error continued.

Does anyone have any suggestions / thoughts of where to look?

stats information is below:

STAT pid 13425

STAT uptime 1940

STAT time 1287591991

STAT version 1.4.5

STAT pointer_size 64

STAT rusage_user 6.443020

STAT rusage_system 25.109182

STAT curr_connections 15

STAT total_connections 340578

STAT connection_structures 118

STAT cmd_get 357224

STAT cmd_set 136334

STAT cmd_flush 0

STAT get_hits 326956

STAT get_misses 30268

STAT delete_misses 0

STAT delete_hits 0

STAT incr_misses 0

STAT incr_hits 0

STAT decr_misses 0

STAT decr_hits 0

STAT cas_misses 0

STAT cas_hits 0

STAT cas_badval 0

STAT auth_cmds 0

STAT auth_errors 0

STAT bytes_read 21641880

STAT bytes_written 41425437

STAT limit_maxbytes 1073741824

STAT accepting_conns 1

STAT listen_disabled_num 0

STAT threads 4

STAT conn_yields 0

STAT bytes 2925203

STAT curr_items 19070

STAT total_items 136334

STAT evictions 0

STAT reclaimed 12

END

[dormando]

> We have a very busy high traffic site and recently started using memached to store frequently accessed queries. �We put memcached on each
> php application server so that the connection is local to the system.
> Today, during one of our peek times, we started getting this error and had to disable the caching of queries:
>
> [20-Oct-2010 12:05:57] PHP Warning: Memcache::connect() [<a href='memcache.connect'>memcache.connect</a>]: Can't connect to

> 127.0.0.1:11211, Cannot assign requested address (99)�

You're probably running out of local ports. Google for that phrase and
you'll probably find help :)

Just as a side note; you're meant to set up memcached as a big cluster and
use all the cache space, as noted at http://memcached.org/about - sticking
them all to localhost only defeats half the purpose. yes I know that
"localhost is faster" but it's more than adequate over the network.

-Dormando

[Rohit Karlupia]

STAT curr_connections 15

STAT total_connections 340578

Looks like your client code is creating and closing connections, instead of reusing them. 

You could increase the port range and/or decrease the tcp close timeout, but much easier fix would be to use the client libraries connection reusing ability.

thanks!

Rohit Karlupia