High-water mark and disk off-loading

[Jens Rantil]

Hi,

I am doing some inventory for a queue solution that can handle somewhat larger payloads 95% percentile maybe ~256 kilobytes. Two questions regarding this:

1. Is beanstalkd a good use case for larger payloads? I understand that I need to increase the max job size. Does it have any performance implications doing this?
2. For a lot of queued tasks, will beanstalkd off-load to disks if I happen to queue too much when using write-ahead log? I am thinking of high-water marking or similar. If beanstalkd does not handle this, how are you guys handling occasions when the queue might fill up? Regular polling of queue size?

Thanks,

Jens

[melon li]

why don't you save the payload into a file and pass the  file path by payload? That would be more efficient when your worker retrieves a job to do from the queue.
在 2015年8月17日星期一 UTC+8下午7:23:41，Jens Rantil写道：

[Lisias]

On Monday, August 17, 2015 at 8:23:41 AM UTC-3, Jens Rantil wrote:

Hi,

1. For a lot of queued tasks, will beanstalkd off-load to disks if I happen to queue too much when using write-ahead log? I am thinking of high-water marking or similar. If beanstalkd does not handle this, how are you guys handling occasions when the queue might fill up? Regular polling of queue size?

I made some stress tests on beanstalkd some years ago using Solaris as host. I just punched millions and millions of job until beanstalkd starts to refuses new jobs - on that server, the process memory allocated up to 4G before refusing jobs.

The jobs are actively refused, your producer(s) is(are) notified and should be able to cache his(their) jobs until the server starts to accept them again.

Once I enabled the consumers, the beanstalkd started to accept new jobs normally. I spawned some extra consumers in order to catch up, and then beanstalkd started to give memory back to the system. No memory leaks. No accepted job was lost.

I don't remember my job sizes, but for sure they didn't reached 256kb.

Since then, I never stressed up beanstalkd again - so perhaps the nowadays mileage may vary.

-- 
Lisias

[Lisias]

On Monday, October 12, 2015 at 5:56:51 PM UTC-3, melon li wrote:

why don't you save the payload into a file and pass the  file path by payload? That would be more efficient when your worker retrieves a job to do from the queue.

Not necessarily. Writing and reading to disk are at least two orders of magnitude slower than RAM. If you can spare the RAM, using RAM is always faster even if you are using SSDs - unless the data is already stored somewhere else, of course.

[Jens Rantil]

Hi,

Sorry for late reply;

On Monday, October 12, 2015, melon li <melon...@gmail.com> wrote:

why don't you save the payload into a file and pass the  file path by payload? That would be more efficient when your worker retrieves a job to do from the queue.

Because I have a distributed system and I would prefer to not

1. denormalize my data as that could introduce many more error scenarios such as missing files etc.; somewhat related
2. introduce a shared file system between my machines. I've battled shared file systems such as NFS enough to know that they occasionally fail and bring down a whole cluster of machines at a time...and a fully distributed file system would unfortunately introduce too much complexity for our small team to afford.

Cheers,

Jens

--

Want to communicate with me securely? You can find my PGP key here.

[Jens Rantil]

Lisias,

I apologize for late reply. Your reply answered exactly what I needed to know. Given your answer I also assume that disabling swap on machines running beanstalkd is a smart thing to do.

Once again, many thanks!

Jens

--
You received this message because you are subscribed to a topic in the Google Groups "beanstalk-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/beanstalk-talk/O-AK3Hy_TrU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to beanstalk-tal...@googlegroups.com.
To post to this group, send email to beansta...@googlegroups.com.
Visit this group at http://groups.google.com/group/beanstalk-talk.
For more options, visit https://groups.google.com/d/optout.