max job size?

[Ian Eyberg]

I have just delved into this recently but I noticed that when I changed my
max job size from the default of 65535 to 2097152 (around 2 meg) my performance
hit the toilet -- what are some max job sizes that you guys use?

Maybe someone can tell me whether or not beanstalk should even be considered
when it comes to larger job sizes of 2meg or so?

Previously I was using 1 - 2k for each job, when I started stuffing in larger
files it just wasn't working too well.

Thanks,
ian

[Keith Rarick]

On Tue, Aug 25, 2009 at 10:00 AM, Ian Eyberg<i...@telematter.com> wrote:
> I have just delved into this recently but I noticed that when I changed my
> max job size from the default of 65535 to 2097152 (around 2 meg) my performance
> hit the toilet -- what are some max job sizes that you guys use?

I always make small jobs (around 100 bytes), but there is no reason,
in principle, not to make big jobs.

> Maybe someone can tell me whether or not beanstalk should even be considered
> when it comes to larger job sizes of 2meg or so?

I simply haven't done any performance testing of large jobs. I'm sure
there are improvements to be made. Here's a ticket:

http://github.com/kr/beanstalkd/issues/#issue/18

I'll try to get to it soon after releasing 1.4.

kr

[Tim Gunter]

For large jobs, I've tended to put the actual job payload in storage and put just the ID of that storage bin in beanstalk.

Works well :)

--
I died in my dreams, what's that supposed to mean

[Dan Mayer]

My cofounder just did some research on the performance of adjusting
job size about a week ago. We pass long string data in beanstalk jobs,
and apparently went over the default job size limit. Before increasing
the limit we did some quick performance testing and found it to get
slower, but not that bad at least for our usage cases... Here is Ben's
analysis...


I did a bit of quick and dirty performance research on this (and then
played around in R for longer than I needed to, because I want to
learn it). For each job size (default=65535, 5x = 327675, 10x =
655350), I did 15 runs with a single test file with 5,000 tests. Here
is the statistical results (feel free to skip over, I generated them,
so I felt like including them)

summary(bean_std)
Min. 1st Qu. Median Mean 3rd Qu. Max.
34.42 37.85 38.83 37.97 38.97 39.54
sd(bean_std)
[1] 1.736959
summary(bean_5x)
Min. 1st Qu. Median Mean 3rd Qu. Max.
30.32 38.46 39.08 38.30 39.29 40.71
sd(bean_5x)
[1] 2.491613
summary(bean_10x)
Min. 1st Qu. Median Mean 3rd Qu. Max.
38.07 39.02 39.29 41.72 39.76 56.20
sd(bean_10x)
[1] 6.015544

This is a density graph ... basically like a histogram, but smoothed
out. Not really useful for seeing the exact values of the peaks, but
since that data is already included in the numbers, this just shows
that the majority of readings hover around 38 seconds for all values.

I took 15 readings for each value

Anyways hope that helps...

peace,
Dan Mayer
Co-founder, Devver.net (http://devver.net)
http://twitter.com/devver
http://mayerdan.com
http://twitter.com/danmayer

[Ian Eyberg]

in reply to everyone:

yes, I should have simply taken the time to do a stress test
of this myself; I ended up throwing a rabbitmq up that day cause
I was short on time....to everyone else I strongly encourage everyone
to follow Dan's example -- cause you won't know anything without benching..

although, Tim definitely had an extremely good idea which I almost implemented
myself

also... if I had taken 2 seconds to realize that -z would ѕet
the default size maybe I wouldn't have freaked out so much

anyways lessons to be learned here: bench, bench, bench... and read your docs...

thanks,
Ian