enabling threadsafe!

[Nahum Wild]

has anyone been doing this?

I've got a backend intranet app and unreleased app that are running with this enabled in combination with "thin --threaded". Works a charm so for, although it's neither are very complicated. I'm looking at trying a more complitade Rails 3.2 app with it and puma, and wondering if anyone has had experience with that setup?

Also, I've been having a very good experience with sidekiq instead of resque.

cheers,
nahum

[Krasimir Angelov]

Also, I've been having a very good experience with sidekiq instead of resque.

Is this in context of enabling threadsafe! or in general? What do you like in sidekiq that's not so good in resque?

Cheers,

K. 

[Nahum Wild]

the decision to move to sidekiq is separate from using threadsafe! but its in a similar domain. I'll quote from the sidekiq readme as to why I'm looking into sidekiq:

"Sidekiq uses multithreading so it is much more memory efficient than Resque (which forks a new process for every job). You'll find that you might need 50 200MB resque processes to peg your CPU whereas one 300MB Sidekiq process will peg the same CPU and perform the same amount of work. Please see my blog post on Resque's memory efficiency and how I was able to shrink a Carbon Five client's resque processing farm from 9 machines to 1 machine."

For further reading about threadsafe! I suggest this:
http://tenderlovemaking.com/2012/06/18/removing-config-threadsafe.html

Since my original post asking about this, I've now got my esdao staging site up and going with puma and sidekiq, still tweaking things to get zero downtime deploys working how I want (which is why I'm not using thin), but it's looking to be a winning combination.

I should probably blog about it...

--nahum

--
You received this message because you are subscribed to the Google Groups "WellRailed" group.
To view this discussion on the web visit https://groups.google.com/d/msg/wellrailed/-/XFh4w0aNA1YJ.
To post to this group, send email to wellr...@googlegroups.com.
To unsubscribe from this group, send email to wellrailed+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/wellrailed?hl=en.

[Malcolm Locke]

On Fri, Sep 07, 2012 at 09:14:27AM +1200, Nahum Wild wrote:
> "Sidekiq uses multithreading so it is much more memory efficient than
> Resque (which forks a new process for every job). You'll find that you
> might need 50 200MB resque processes to peg your CPU whereas one 300MB
> Sidekiq process will peg the same CPU and perform the same amount of work.
> Please see my blog post on Resque's memory

> efficiency<http://blog.carbonfive.com/2011/09/16/improving-resques-memory-efficiency/>and

> how I was able to shrink a Carbon Five client's resque processing farm
> from 9 machines to 1 machine."

"To scale to the levels they wanted to get to, they projected that
they would need to run hundreds of Resque processes, each consuming
512MB. Insanity!"

If this figure of 512MB has been gleaned by just reading the largest
number from top (usually VIRT), and maybe it wasn't, then it is pretty
meaningless.

See this stack overflow answer for some guidelines on how to interpret
top output, on Linux at least. If you ever need to do that kind of
forward planning for 'how much memory will I need for x workers' it
helps to be able to interpret what the figures mean.

http://stackoverflow.com/a/561450/223304

Malc

[Krasimir Angelov]

For further reading about threadsafe! I suggest this:
http://tenderlovemaking.com/2012/06/18/removing-config-threadsafe.htm

Yeah, thats pretty good post on the topic. 

 

I should probably blog about it...

Please do :) 

[Andy Newport]

Sidekiq was originally built to wrangle around the fact that jruby is java so forking is impossible (in principle) making resque a questionable choice in the first place. To sell it on it's performance in that context is pretty irrelevant for anyone not running a jruby app.

There's also ruby's horrible memory model to consider, if you're forking off before running a job your memory usage won't suddenly jump and never drop (ruby's GC doesn't reclaim the heap very well due to it's lack of compaction) when a job runs slightly beyond the current heap allocation. For anyone running a decent sized deploy that's a Good Thing (tm) because that job you wrote a year ago (that's super inefficient for that one edge case that will "never happen") doesn't randomly take out your job system on your weekend off...

Finally, and this is a big one, debugging multi-threaded code is shite when it's your own code you wrote yesterday so.....

[Jeremy Wells]

Fork'd jobs also elicit the use of standard tools to figure out and recover from situations such as when your year old super inefficient job suddenly takes 20 hours. You can kill just that process, tell what job caused the problem, and attach gdb to it to get a backtrace. I think all of that is harder with a threaded model, although I don't know the tools you have available if you're running JRuby.