memory leak in 1.4.2?

[Ask Bjørn Hansen]

Hi everyone,

Today our monitoring system started sending messages that our primary
beanstalk box was upset. beanstalkd was using all available memory
(~12GB).

We have the binlog enabled. After restarting the box (and thus
beanstalkd) it's using a few hundred KB for 2400 jobs.

The memory usage had increased steadily over the last ~5-6 days since
we updated to 1.4.2 (plus patches: git = ee671f).

Anyone else seen something similar?

- ask

[ty...@monkeypox.org]

We've seen this behavior as well with 1.4.2 on our production beanstalkd
instance, unfortunately it's been practically unreproducible in a development
environment.

Running beanstalkd under valgrind doesn't give me the impression that
beanstalkd is leaking memory so much as it's being over-ambitious in its
allocation of memory for its internal job management.

Haven't gotten any good idea of what it might be, our load is on average quite
low but has periodic spikes, so I've not tracked down anything entirely useful
:(

Cheers,
-R. Tyler Ballance
--------------------------------------
GitHub: http://github.com/rtyler
Twitter: http://twitter.com/agentdero
Blog: http://unethicalblogger.com

[Carl]

Exactly the same problem here.

>  application_pgp-signature_part
> < 1KViewDownload

[Keith Rarick]

On Tue, Dec 8, 2009 at 12:57 PM, Carl <cmer...@gmail.com> wrote:
> Exactly the same problem here.

Have you tried 1.4.3? We fixed two memory leaks.

kr

[Carl]

I just realized that 1.4.3 was out. I'm still on 1.4.2. I'll update
and report back.


On Dec 8, 7:57 pm, Keith Rarick <k...@xph.us> wrote:

[Carl]

1.4.3 seems to have fixed my problem, thanks!

On Dec 8, 7:57 pm, Keith Rarick <k...@xph.us> wrote:

[Keith Rarick]

On Mon, Dec 14, 2009 at 5:28 AM, Carl <cmer...@gmail.com> wrote:
> 1.4.3 seems to have fixed my problem, thanks!

Excellent! Be sure to let me know if you notice any more problems
(memory or other).

kr

[Carl]

Yes, major problem with 1.4.3. It just crashes on me after a few hours
of use. Could it be because I tried to set the maximum job size to
1MB?

Is there a way to create a log file to see what happens, because right
now I have no clue what could be causing this.

Thanks!


On Dec 15, 4:39 am, Keith Rarick <k...@xph.us> wrote:

[Carl]

I've been seeing a few

beanstalkd: prot.c:693 in check_err: read(): Connection reset by peer

When not in daemon mode. Not sure if it's related. But it hasn't
crashed yet.

[Carl]

And now I'm seeing a lot of these

beanstalkd: prot.c:1781 in h_accept: accept(): Too many open files
beanstalkd: net.c:72 in brake: too many connections; putting on the
brakes
beanstalkd: net.c:88 in unbrake: releasing the brakes
beanstalkd: prot.c:1781 in h_accept: accept(): Too many open files
beanstalkd: net.c:72 in brake: too many connections; putting on the
brakes
beanstalkd: net.c:88 in unbrake: releasing the brakes


I think the Ruby gem leaves connections open, is that the case? I
found this fix http://www.mail-archive.com/beansta...@googlegroups.com/msg00372.html.
Has this been merged to your master branch?

/c

[Graham Barr]

On Tue, Dec 15, 2009 at 8:58 AM, Carl <cmer...@gmail.com> wrote:
> Yes, major problem with 1.4.3. It just crashes on me after a few hours
> of use. Could it be because I tried to set the maximum job size to
> 1MB?

Whoa!, thats a big payload. Have you considered putting the payload in
a database
and just a reference to it in the queue ?

If you have binlog turned on then you will eat disk space very quick, especially
if you have a long lived job while others come and got quickly.

Graham.

[Graham Barr]

On Tue, Dec 15, 2009 at 9:43 AM, Carl <cmer...@gmail.com> wrote:
> I think the Ruby gem leaves connections open, is that the case? I
> found this fix http://www.mail-archive.com/beansta...@googlegroups.com/msg00372.html.
> Has this been merged to your master branch?

Is your server running on Linux ?

If so, using the pid of the beanstalk server

ls -l /proc/<pid>/fd

will tell you what file descriptors are open to

netstat -pane | grep <pid>

will tell you where any net connections for the server process are connected to.

Graham.

[Keith Rarick]

That's good advice. Even so, beanstalkd should not *crash* with 1MB jobs.

kr

[Keith Rarick]

On Tue, Dec 15, 2009 at 6:58 AM, Carl <cmer...@gmail.com> wrote:
> Is there a way to create a log file to see what happens, because right
> now I have no clue what could be causing this.

You can run it without the "-d" flag. When something goes wrong,
beanstalkd will print out error messages. If the crash is from a
segfault, you'll have to rely on your operating system to tell you so.
(And, in that case, a stack trace or a core dump would be very helpful
to me.)

kr

[Carl]

I think I found/solve the issue. Somehow my Ruby app was leaking
connections, which was driving the C daemon crazy it seems. I've been
able to limit the connection leakage and it seems to be doing much
better now. I also use HEAD of the Ruby client. Would it be possible
to push a release soon? It's much easier to use a gem and it's been a
while.

I don't normally have 1MB jobs, but I set the limit high in case it
happens (very hard for us to predict the size of jobs). Jobs over 50k
are probably 1 in a 100,000, so I'm not too worried.

Thanks!



On Dec 15, 3:48 pm, Keith Rarick <k...@xph.us> wrote:

[Linan Wang]

i noticed that the ruby client installed via gem is different than the
one on git. the gem version still has connection leakage when use
reserve(timeout).

> --
>
> You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
> To post to this group, send email to beansta...@googlegroups.com.
> To unsubscribe from this group, send email to beanstalk-tal...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
>
>
>



--
Best regards

Linan Wang

[Carl]

Exactly. That's why I think it'd be nice to have a release :)

Anything can be done, Keith?

Thanks!

> > For more options, visit this group athttp://groups.google.com/group/beanstalk-talk?hl=en.

[Keith Rarick]

On Tue, Dec 15, 2009 at 5:03 PM, Carl <cmer...@gmail.com> wrote:
> I think I found/solve the issue. Somehow my Ruby app was leaking
> connections, which was driving the C daemon crazy it seems.

beanstalkd should handle that scenario reasonably. It will start
refusing new connections until the number of open connections drops.
If you're seeing different behavior, I'd like to fix it.

> I've been
> able to limit the connection leakage and it seems to be doing much
> better now. I also use HEAD of the Ruby client. Would it be possible
> to push a release soon? It's much easier to use a gem and it's been a
> while.

Yeah, I'm (slowly) working on it, with enormous help from several
other developers (as you can see in the commit logs).

kr