Beanstalk Hangs Problem

[Al-Faisal El-Dajani]

Hello,

I'm using beanstalkd 1.4.2, compiled from source along with libevent
1.4.12. I am using it with activemessaging plugin for rails. For some
reason, beanstalkd keeps hanging after a while, and attempting to
connect to it using Beanstalk::Pool.new gives me the following
message: Errno::ECONNRESET: Connection reset by peer

In development it works fine and I do not face any problems, this only
happens in production mode. The only difference between development
and production modes is that in production mode I am using more tubes
(up to 18), and the beanstalkd server is on a separate machine
accessed by 3 beanstalk client instances. Other than that it is the
exact same environment and code base.

I can not find any logs for beanstalk that might contain information
about why this is happening. If somebody can offer some help or tell
me where to look to figure this out it would be deeply appreciated,
thanx.

[Keith Rarick]

On Tue, Nov 10, 2009 at 6:53 AM, Al-Faisal El-Dajani
<faisal...@gmail.com> wrote:
> ... attempting to

> connect to it using Beanstalk::Pool.new gives me the following
> message: Errno::ECONNRESET: Connection reset by peer

The most likely cause of this is that the clients are somehow leaking
connections. If beanstalkd runs out of open file descriptors (often
the OS limits this to 1024 by default) it will stop accepting new
connections until existing ones are closed.

You can use the netstat command to see how many connections are open
at any time.

kr

[Jaume Sabater]

On Tue, Nov 10, 2009 at 3:53 PM, Al-Faisal El-Dajani
<faisal...@gmail.com> wrote:

> I can not find any logs for beanstalk that might contain information
> about why this is happening. If somebody can offer some help or tell
> me where to look to figure this out it would be deeply appreciated,
> thanx.

Sorry for hijacking this thread. About a couple of weeks ago I sent a
mail to this list asking for the way beanstalkd logs information. I
think it was Keith that replied that he had not been able to find any
logger fast enough to support the way beanstalkd works. Just wanted to
note that this is the perfect example of logging put into good use.
Perhaps when beanstalkd is in live, heavily loaded environments what
you said to me is true, but if not under very heavy load, the use of
normal logging procedures would help "debugging".

Just my two cents anyway.

--
Jaume Sabater
http://linuxsilo.net/

"Ubi sapientas ibi libertas"

[Al-Faisal El-Dajani]

What Keith said was correct, the clients were leaking connections which was leading to my problem. What follows is an explanation of how the leak occurred, and how I fixed it.

The whole problem was in the way I reserved messages from the queue. I used to get messages like so:
message = tube.reserve(2) rescue nil

The entire problem was the that I specified a timeout for the reserve call. Tracing the code, you'll find that method reserve in class Pool calls method send_to_rand_conn which in turn calls method call_wrap.
Within method call_wrap, if an exception was encountered, It removes the address of the connection from the list of addresses WITHOUT closing the connection, and raises an exception. And that was the entire problem.

Whenever reserve(2) was called and did not find a message in the tube, it would raise a TimedOut exception, which leads to removing the connection address. Any future call to put or reserve would establish a new connection, because the address no longer exists. Since the old connection was not closed, and a new one was being created, this led to a leakage in connections which would soon prohibit new connections from being made.

There are 3 solutions to this problem:
1) Before removing the address, make sure that the connection is closed.
2) Create a different rescue stub to handle TimedOut exceptions (I went with this option as it was far simpler)
3) Do not use reserve with timeout.

I hope I made my problem clear now, and hope that a fix for this becomes part of the official gem soon.

Cheers,

--

You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beansta...@googlegroups.com.
To unsubscribe from this group, send email to beanstalk-tal...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=.

--
Al-Faisal El-Dajani
10/6

[Al-Faisal El-Dajani]

I should note that I am using the beanstalk-client ruby gem.

--
Al-Faisal El-Dajani
10/6

[Martyn Loughran]

I actually posted a patch for exactly this issue a couple of weeks ago
- this should solve your problem. Sorry I missed your message
yesterday.

http://groups.google.com/group/beanstalk-talk/browse_thread/thread/e8e1176c6dfefc80

2009/11/11 Al-Faisal El-Dajani <faisal...@gmail.com>:

[Alexander Kunz]

Hello,

i am searching a simple beanstalkd client for PHP. I try
BeanStalk.class.php from
http://sourceforge.net/projects/beanstalk/. But my script use 100% cpu power
when the queue is empty. I get this too when i use the inclued example,
it use also
100% of my Cpu power.

Are there any other simple beanstalkd classes on the net?

Kind regrads

[Michael Lang]

Sounds like you're using something other than $beanstalk->reserve();
in your loop. Does the examples from the PHP client (which do
demonstrate using "reserve") also use 100% CPU?

Michael

> --
>
> You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
> To post to this group, send email to beansta...@googlegroups.com.
> To unsubscribe from this group, send email to beanstalk-tal...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=.
>
>
>



--

http://codeconnoisseur.org

[Alexander Kunz]

Hello Miacheal,

thanks for your fast reponse. I use the following php code, this time
there is no loop...

<?php

require('BeanStalk.class.php');

$beanstalk = BeanStalk::open(array(
'servers' => array( '127.0.0.1:11300' ),
'select' => 'random peek'
));

$beanstalk->use_tube('test');

$job = $beanstalk->reserve();

echo $job->get();
Beanstalk::delete($job);

?>

[Michael Lang]

That code looks perfectly fine to me. I'm a Ruby developer and
haven't had any issues with CPU utilization in a similarly configured
loop. When the reserve method is called, it waits for beanstalk to
respond and CPU stays pretty much at 0% until the job comes in.

Michael

--
http://codeconnoisseur.org

[Alexander Kunz]

Yes, thats what i expect :) waiting for a new job with low or no CPU
utilization...
For testing i switch to beanstalkd 1.3, with the same result. 100% CPU
utilization,
after a short time 80% php 20% beanstalkd itself...