Issue 122 in memcached: failed to write, and not due blocking: No error
memc...@googlecode.com
Owner: ----
Labels: Type-Defect Priority-Medium
New issue 122 by mr.sela: failed to write, and not due blocking: No error
http://code.google.com/p/memcached/issues/detail?id=122
Hi,
i am running memcached 1.4.4 win32 on windows server 64 bit,
i have on this server 4 instances of memcached 2G each.
few weeks ago we started to see that the instances from time to time are
freezing for 2 minutes and then get back to work, they are all freezing
almost at the same time when i ran it in -vv mode i saw the following:
<3564 new auto-negotiating client connection
<2440 new auto-negotiating client connection
240: Client using the ascii protocol
<240 get CheckUserCookie:1714451
> 240 END
<3408 connection closed.
3564: Client using the ascii protocol
<3564 get SortedLiveShows:3,0,1714273,0,1,0
> 3564 sending key SortedLiveShows:3,0,1714273,0,1,0
> 3564 END
Failed to write, and not due to blocking: No error
<3564 connection closed.
<240 connection closed.
2440: Client using the ascii protocol
<2440 get MyTags:4,1616578,1,1616578,3,1,50,0
> 2440 sending key MyTags:4,1616578,1,1616578,3,1,50,02 257 1309
> 2440 END
<2440 connection closed.
and here it is freezing for to minutes
let me just say that if i run only one instance of memcached on the
server it works fine with out any glitch
is some one have any idea what could be wrong?
thank you
Eyal Sela
--
You received this message because you are listed in the owner
or CC fields of this issue, or because you starred this issue.
You may adjust your issue notification preferences at:
http://code.google.com/hosting/settings
memc...@googlecode.com
Comment #1 on issue 122 by billyle...@suddenlink.net: failed to write, and
http://code.google.com/p/memcached/issues/detail?id=122
I have just installed win32-1.4.4-54 server along with
memcacheddotnet_clientlib-1.1.5 and am getting 'Failed to write, and not
due to blocking: No error' every time my client calls the server(Get, Set,
Stats, i.e.). I have tried tweaking the command line args with no
success. Any ideas. I've run memcached on linux for years without any
type of problems but as usual there is probably an issue with something in
windoze7.
thx.
Henrik Schröder
I have a vague memory of getting that error when the client didn't handle some commands correctly.
/Henrik
memc...@googlecode.com
Comment #2 on issue 122 by sean.y....@gmail.com: failed to write, and not
http://code.google.com/p/memcached/issues/detail?id=122
I have the same issue. Have anyone solve this problem?
Roberto Spadim
with arp table limit? maybe tcp/ip stack limit, get netstat -a and
check if tcp table is full
try using UDP client protocol instead TCP
2011/2/7 <memc...@googlecode.com>:
--
Roberto Spadim
Spadim Technology / SPAEmpresarial
memc...@googlecode.com
Comment #3 on issue 122 by rspa...@gmail.com: failed to write, and not due
http://code.google.com/p/memcached/issues/detail?id=122
maybe problem with tcp/ip or arp table?
try with udp client/server protocol
memc...@googlecode.com
Comment #4 on issue 122 by sean.y....@gmail.com: failed to write, and not
http://code.google.com/p/memcached/issues/detail?id=122
Thanks. Could you please explain what are the possible problems with arp
table? If I use UDP, how can I make sure it's working fine without the help
of the telnet interface?
I'm using Memcached 1.4.5 one W2K8 Server R2.
Thanks again.
memc...@googlecode.com
Comment #5 on issue 122 by sean.y....@gmail.com: failed to write, and not
I'm using Memcached 1.4.5 one W2K8 Server R2. And I am testing the
memcached server from the local machine using telnet. It's giving me the
following error when I send "STATS" in telnet:
<1352 server listening (auto-negotiate)
<1368 send buffer was 8192, now 268435456
<1368 server listening (udp)
<1368 server listening (udp)
<1368 server listening (udp)
<1368 server listening (udp)
<1380 new auto-negotiating client connection
1380: Client using the ascii protocol
<1380 stats
Failed to write, and not due to blocking: No error
<1380 connection closed.
Thanks again.
Roberto Spadim
Roberto Spadim
EARPFULL (104)
ARP table full. This error is reported by send when the ARP cache is
full. If this error occurs, increase CFG_ARPCLEN.
for linux you can see this message in /var/log/messages:
"21 messages suppressed Neighbour table overflow."
use this to get max values of arp table:
# echo 16384 > /proc/sys/net/ipv4/neigh/default/gc_thresh1
# echo 32768 > /proc/sys/net/ipv4/neigh/default/gc_thresh2
# echo 65535 > /proc/sys/net/ipv4/neigh/default/gc_thresh3
in windows i don´t know what happen... :/ sorry
but check system logs
2011/2/7 <memc...@googlecode.com>:
--
Roberto Spadim
check netstat -a when this problem happen
after 2 minutes (is you tcp with 2minutes timeout? )
2011/2/7 Roberto Spadim <rob...@spadim.com.br>:
Dustin
Please note that someone following a bug isn't necessarily going to
subscribe to this list to see if there's commentary on it here, too.
You've started dialogue over there and answered questions somewhere
else.
On Feb 7, 4:45 pm, Roberto Spadim <robe...@spadim.com.br> wrote:
> i think that you have a problem in arp, or tcp stack size
> check netstat -a when this problem happen
> after 2 minutes (is you tcp with 2minutes timeout? )
>
>
>
>
>
>
>
>
>
> > for rtos:
> > EARPFULL (104)
>
> > ARP table full. This error is reported by send when the ARP cache is
> > full. If this error occurs, increase CFG_ARPCLEN.
>
> > for linux you can see this message in /var/log/messages:
> > "21 messages suppressed Neighbour table overflow."
> > use this to get max values of arp table:
> > # echo 16384 > /proc/sys/net/ipv4/neigh/default/gc_thresh1
> > # echo 32768 > /proc/sys/net/ipv4/neigh/default/gc_thresh2
> > # echo 65535 > /proc/sys/net/ipv4/neigh/default/gc_thresh3
>
> > in windows i don´t know what happen... :/ sorry
> > but check system logs
>
memc...@googlecode.com
Comment #6 on issue 122 by dsalli...@gmail.com: failed to write, and not
http://code.google.com/p/memcached/issues/detail?id=122
I don't know why one might assume the problem is below layer 3 -- or even
outside of memcached itself.
It's likely to be related to the Windows port itself (which is not
something we built or planned to support in 1.4.4, though we've done a lot
of work since).
Roberto Spadim
many connections can cause this problem, try this command:
nmap -sP -T5 172.16.0.0/8
on linux you will get a table overflow (cache), wait some time and you
will get you network running again (i don´t know how to flush arp
cache)
2011/2/7 <memc...@googlecode.com>:
--
Dustin
Thank you for not responding in the bug. I would like to reduce
confusion over there.
On Feb 7, 5:53 pm, Roberto Spadim <robe...@spadim.com.br> wrote:
> no not below layer3, it´s a overfloow of table
I still don't see why you believe that an address lookup table is
causing this exact error on exactly one platform...
> on linux you will get a table overflow (cache), wait some time and you
> will get you network running again (i don´t know how to flush arp
> cache)
If you know why this exact error (which claims to not be a layer 3
write failure with no actual error code recorded) on Windows would be
caused by address resolution for three different users under
relatively no traffic, then that may lead us to a solution.
I don't actually know Windows enough to know why it would happen. I
also don't know where these builds came from. I'd blame the memcached
Windows code long, long before assuming Windows can't write data over
a socket due to ARP.
Roberto Spadim
problem (not a memcached problem)
i'm thinking about possible problems, if you don't want to try it...
on linux i had the same problem but with another software (network
file system, and apache) no error code was reported, but at
/var/log/messages i could get the arp table overflowed
maybe this could help you to solve the problem...
since my first problem (apache/nfs) wasn't at level3, it's at level 6,7
in my case, arp was running (it didn't stoped) but didn't accept a new
'record' in table (a new client)
after some time the cache get out (near 2 minutes!) and my program get
normal again
try with UDP... i think it can help, if not let's think about other problem
2011/2/8 Dustin <dsal...@gmail.com>:
--
memc...@googlecode.com
Comment #7 on issue 122 by sean.y....@gmail.com: failed to write, and not
http://code.google.com/p/memcached/issues/detail?id=122
This happened when the memcached was trying to send the response back to
the client. The socket error number is 10054, which generally means the
socket was forcibly closed by the remote host. However, I am
using "memcached.exe -l 127.0.0.1 -p 8199" When I telnet 127.0.0.1 8199, I
don't know why the socket will be closed. Could anybody shred some light on
this issue?
memc...@googlecode.com
Comment #8 on issue 122 by sean.y....@gmail.com: failed to write, and not
http://code.google.com/p/memcached/issues/detail?id=122
I agree. I don't think it's related to ARP, since I am using loopback
interface 127.0.0.1. The arp table is not full. I can stable repro this on
a few machines right after I start the memcache.exe. On some other boxes,
it works fine. I can't figure out the differences between this boxes. They
are of the same Windows version with all last patches.
memc...@googlecode.com
Comment #9 on issue 122 by sean.y....@gmail.com: failed to write, and not
http://code.google.com/p/memcached/issues/detail?id=122
I don't think it's related to ARP either, since I am using the loopback
interface 127.0.0.1. The arp table is not full. I can stably repro this
issue on a few machines right after I start the memcache.exe. On some other
machines, it works fine though. I can't figure out the differences between
these machines. They are of the same Windows version with all last patches.
Roberto Spadim
shared memory protocol?
for loop back (127.0.0.1) arp table have always 1 register (internal
on arp engine, or inside arp table), i think that's not a arp problem
too...
maybe windows virtual memory problem?
what's your today app size?
how many clients at same time?
how many keys?
what's the average key length?
2011/2/8 <memc...@googlecode.com>:
--
Sean
and 0 keys. On the same machine, if I run memcached.exe version 1.2.1,
it works fine.
On Feb 8, 2:02 pm, Roberto Spadim <robe...@spadim.com.br> wrote:
> i don't know, but can memcache use windows pipes?
> shared memory protocol?
> for loop back (127.0.0.1) arp table have always 1 register (internal
> on arp engine, or inside arp table), i think that's not a arp problem
> too...
> maybe windows virtual memory problem?
> what's your today app size?
> how many clients at same time?
> how many keys?
> what's the average key length?
>
Roberto Spadim
i don't know if you will get more speed with newer version
if you don't know, please continue trying in this mail list. i can't
help with windows :(
2011/2/8 Sean <sean....@gmail.com>:
dormando
trouble; I'm assuming it's a buggy libevent, or buggy interaction with
libevent. Given that it's an unhandled socket error :P
Where did you get the windows binary from?
Sean
http://blog.elijaa.org/index.php?post/2010/08/25/Memcached-1.4.5-for-Windows
I also tried build it myself using mingw. It turns out with the same
error.
On Feb 8, 2:46 pm, dormando <dorma...@rydia.net> wrote:
> I don't think anyone knows why that particular port of memcached has
> trouble; I'm assuming it's a buggy libevent, or buggy interaction with
> libevent. Given that it's an unhandled socket error :P
>
> Where did you get the windows binary from?
>
>
>
> On Tue, 8 Feb 2011, Roberto Spadim wrote:
> > =] can you use 1.2.1 without problems? use it ehheeheh
> > i don't know if you will get more speed with newer version
> > if you don't know, please continue trying in this mail list. i can't
> > help with windows :(
>
> > > and 0 keys. On the same machine, if I run memcached.exe version 1.2.1,
> > > it works fine.
>
> > >> i don't know, but can memcache use windows pipes?
> > >> shared memory protocol?
> > >> for loop back (127.0.0.1) arp table have always 1 register (internal
> > >> on arp engine, or inside arp table), i think that's not a arp problem
> > >> too...
> > >> maybe windows virtual memory problem?
> > >> what's your today app size?
> > >> how many clients at same time?
> > >> how many keys?
> > >> what's the average key length?
>
dormando
enough...
On Tue, 8 Feb 2011, Sean wrote:
> I am getting it from membase:
> http://blog.elijaa.org/index.php?post/2010/08/25/Memcached-1.4.5-for-Windows
>
> I also tried build it myself using mingw. It turns out with the same
> error.
>
>
>
> On Feb 8, 2:46�pm, dormando <dorma...@rydia.net> wrote:
> > I don't think anyone knows why that particular port of memcached has
> > trouble; I'm assuming it's a buggy libevent, or buggy interaction with
> > libevent. Given that it's an unhandled socket error :P
> >
> > Where did you get the windows binary from?
> >
> >
> >
> > On Tue, 8 Feb 2011, Roberto Spadim wrote:
> > > =] can you use 1.2.1 without problems? use it ehheeheh
> > > i don't know if you will get more speed with newer version
> > > if you don't know, please continue trying in this mail list. i can't
> > > help with windows :(
> >
> > > 2011/2/8 Sean <sean.y....@gmail.com>:
> > > > I am using only one telnet to test the memcache. So only one client
> > > > and 0 keys. On the same machine, if I run memcached.exe version 1.2.1,
> > > > it works fine.
> >
> > > > On Feb 8, 2:02�pm, Roberto Spadim <robe...@spadim.com.br> wrote:
> > > >> i don't know, but can memcache use windows pipes?
> > > >> shared memory protocol?
> > > >> for loop back (127.0.0.1) arp table have always 1 register (internal
> > > >> on arp engine, or inside arp table), i think that's not a arp problem
> > > >> too...
> > > >> maybe windows virtual memory problem?
> > > >> what's your today app size?
> > > >> how many clients at same time?
> > > >> how many keys?
> > > >> what's the average key length?
> >
> > > >> 2011/2/8 �<memcac...@googlecode.com>:
memc...@googlecode.com
Status: WontFix
Comment #10 on issue 122 by dorma...@rydia.net: failed to write, and not
due blocking: No error
http://code.google.com/p/memcached/issues/detail?id=122
I'm closing this, as 1.4 under windows isn't supported. If anyone on this
thread tries the 1.6 release under windows and has a similar issue, please
feel free to open a new bug.
memc...@googlecode.com
Comment #11 on issue 122 by pavlovma...@gmail.com: failed to write, and not
The same:
slab class 40: chunk size 573744 perslab 1
slab class 41: chunk size 717184 perslab 1
slab class 42: chunk size 1048576 perslab 1
<1564 server listening (auto-negotiate)
<1580 send buffer was 64512, now 268435456
<1580 server listening (udp)
<1580 server listening (udp)
<1580 server listening (udp)
<1580 server listening (udp)
<1592 new auto-negotiating client connection
1592: Client using the ascii protocol
<1592 set some_key 0 0 10
> 1592 STORED
Failed to write, and not due to blocking: No error
<1592 connection closed.
<1592 new auto-negotiating client connection
1592: Client using the ascii protocol
<1592 set some_key 0 0 10
> 1592 STORED
Failed to write, and not due to blocking: No error
<1592 connection closed.