Does beanstalk keep every job in RAM?

[David Leimbach]

... and is there a way to prevent it from doing so?

I'm thinking about trying to use beanstalk in a situation where I'd like actually queue up some "event traffic" for eventual delivery to another system.  

After inserting 100,000 512 byte records, I'm seeing the resident memory size of beanstalkd go up to about 105MB on a 64bit platform.  

I've got durable logging on so I can kill and restart the server and it quickly gets back up to about 105MB fairly fast.

I'm actually more interested in the durability aspect of events rather than speed of retrieval and delivery.  I really want a persistent queue I guess.

Dave

[Chad Kouse]

Use the -b flag when starting beanstalkd. 

Example:

beanstalkd -b /var/spool/beanstalkd

--
Chad Kouse

--
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To view this discussion on the web visit https://groups.google.com/d/msg/beanstalk-talk/-/OmR4IHGk1SkJ.
To post to this group, send email to beansta...@googlegroups.com.
To unsubscribe from this group, send email to beanstalk-tal...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.

[David Leimbach]

Thanks, however, I was hoping that with "-b" that beanstalk would not still keep all the on-disk contents in RAM.  

Even with durability on, writing 100,000 512 byte jobs to beanstalk is consuming 109MB of RAM.  I was laboring under what appears to be a bad assumption that beanstalk would not keep every job in RAM when I'm using durable queues.

Dave

[Chad Kouse]

Is there a specific reason you don't want it to put stuff in RAM?

-- 

Chad Kouse

To view this discussion on the web visit https://groups.google.com/d/msg/beanstalk-talk/-/hlm0E1aXPLAJ.

[David Leimbach]

I don't need it to be in RAM if it's on disk, and RAM is precious on these nodes.  

It doesn't appear to be the correct resource utilization for my purposes.

[Chad Kouse]

Yeah. I don't see any way to make it read stuff from only disk.  I think the normal use case is to have consumers constantly draining the job queues. 

--
Chad Kouse

To view this discussion on the web visit https://groups.google.com/d/msg/beanstalk-talk/-/VIHnIHbbK68J.

[Keith Rarick]

On Fri, Aug 24, 2012 at 10:09 AM, David Leimbach <lei...@gmail.com> wrote:
> I don't need it to be in RAM if it's on disk, and RAM is precious on these
> nodes.

You should probably not be using beanstalkd for this.

Keeping all jobs in memory is pretty fundamental to
how beanstalkd is intended to work. If you have more
data than fits in memory, consider using a database
such as postgres instead.

[Chad Kouse]

We actually use beanstalkd to queue up jobs to be written to disk (event logs to eventually be processed with Hadoop) - sounds sorta similar to what you are describing. Maybe just writing to flat files (or in our case - queuing in beanstalkd to be written to a flat file every few seconds) will be sufficient. 

We use beanstalkd in between to effectively make writing to a flat file a non blocking call in our application.  

--
Chad Kouse

--

You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.

[Bharanee Rathna]

out of curiosity, do you think we can get this with mmap ? that will give the users flexibility to adjust how much they want to page into main memory.

[Keith Rarick]

On Fri, Aug 24, 2012 at 3:39 PM, Bharanee Rathna <deep...@gmail.com> wrote:
> out of curiosity, do you think we can get this with mmap ?

Nope. At least, not as easily as you seem to imply. Keeping
everything in memory is an enormously simplifying design choice.
Spilling data to disk would require beanstalkd to effectively
become a database (and probably to become slower, too).

There are plenty of databases out there that do roughly this,
even some that pretend to be queueing systems. (And some that
pretend to be queueing systems pretending to be memcacheds.)

[David Leimbach]

I'm not going to lie, I've got some strange needs for this project :-)