Storage Mechanism Size and Recommendations

[Chad]

Hi,

We are building a service on ruote, and have encountered some performance problems.  We originally used Route::Sequel::Storage with the mysql2 adapter.  On deployment we noticed that ruote made massive number of sql requests, producing numerous time outs.

For comparison, we switched to Route::FsStorage.  (We do not want to use the filesystem for storage, in case we need to scale ruote to another server).  Anyway, I ran 200 requests (via thin sinatra web service) to a simple process definition that takes a string "Hello" and outputs via one participant.  The FsStorage directory grew to 4.1MB.   This seems like a lot for just inputting and outputting.   This is going to be problem for our larger jobs that we have been writing.

So I have 2 questions.  Are the massive amount of sql requests normal, and is the amount of file system storage usual?  If so, how do people work handle this in production environments.

Any advice or help would be greatly appreciated.

Thanks,

Chad

[John Mettraux]

On Wed, Jul 25, 2012 at 03:21:25AM -0700, Chad wrote:
>
> We are building a service on ruote, and have encountered some performance
> problems. We originally used Route::Sequel::Storage with the mysql2
> adapter. On deployment we noticed that ruote made massive number of sql
> requests, producing numerous time outs.

Hello Chad,

do you have details? I'd like to make ruote-sequel better.

> For comparison, we switched to Route::FsStorage. (We do not want to use
> the filesystem for storage, in case we need to scale ruote to another
> server). Anyway, I ran 200 requests (via thin sinatra web service) to a
> simple process definition that takes a string "Hello" and outputs via one
> participant. The FsStorage directory grew to 4.1MB. This seems like a
> lot for just inputting and outputting. This is going to be problem for
> our larger jobs that we have been writing.

I ran two tests:

---8<---
require 'rufus-json/automatic'
require 'ruote'
require 'ruote-fs'

FileUtils.rm_rf('work') rescue nil

dboard = Ruote::Dashboard.new(Ruote::Worker.new(Ruote::FsStorage.new('work')))
dboard.noisy = ENV['NOISY'].to_s == 'true'

dboard.register_participant 'toto' do |workitem|
print '.'
end

200.times do
wfid = dboard.launch(Ruote.define do
toto
end)
dboard.wait_for(wfid)
end

puts
puts `du -sh work/`
--->8---

Those 200 hundred terminated flows take up 12K.

Then I moved to 200 hundred un-terminated flows:

---8<---
require 'rufus-json/automatic'
require 'ruote'
require 'ruote-fs'

FileUtils.rm_rf('work') rescue nil

dboard = Ruote::Dashboard.new(Ruote::Worker.new(Ruote::FsStorage.new('work')))
dboard.noisy = ENV['NOISY'].to_s == 'true'

dboard.register_participant 'toto', Ruote::NullParticipant

200.times do
dboard.launch(Ruote.define do
toto
end)
end

sleep 14.0

puts
puts `du -sh work/`
--->8---

and it reached 1.6M.

> So I have 2 questions. Are the massive amount of sql requests normal, and

Yes, it's normal. Ruote needs to keep up with incoming msgs (orders) and
schedules so it polls its storage.

When activity slows down, the polling slows down a bit.

Some storage implementations (sorry, ruote-swf for example) use long polling
techniques.

If you need less polling I could adapt (or provide means to adapt the polling
frequency).

> is the amount of file system storage usual? If so, how do people work
> handle this in production environments.

Yes, it's normal. I chose to not compress at all files for the fs storage
implementation... Disk space is cheap (I guess you work with virtual
servers).

It's not too difficult to modify a storage implementation to have some degree
of compression.

Ruote keeps track of lots of redudant information (especially in
expressions), they are like many save points. It's uneconomical but easier to
implement and easier to fix.


Best regards,

--
John Mettraux - http://lambda.io/jmettraux

[Chad]

Hi John,

If manage to get this working, we owe you a dinner (and more) the next time you're in Tokyo.

So here's the story with Sequel.  We are using 1.9.2-p290 and Sequel 3.37.0 and mysql2 0.3.11.

Here's a sample of the SQL traces, if we add a logger to a Sequel connection:

https://gist.github.com/3175854

The trace is on a new and clean database, with no data in it, and simply instantiating the Route::Engine like so:

  db = Sequel.connect(db_settings)

  db.loggers << OurLogging.logger

  RuoteKit.engine = Ruote::Engine.new(

    Ruote::Worker.new(

      Ruote::Sequel::Storage.new(db)))

Even though we started no process definitions, Ruote's Engine immediately hits the DB.  But more importantly, it never stops.  The log just grows and grows and grows.

Regarding the FsStorage, your megabyte numbers are dramatically smaller than ours.  We are basically doing nothing as well, but I'll perform your tests..   I grant that disk-space is cheap, but there's not enough disk-space in the world available if we keep Ruote running for a year, without  deleting some of the stuff Ruote saves on the file system.  If we have to use FsStorage, we would need to run a cron job to clean things occassionally.  What files would we need to purge?  The structure of the FsStorage is rather daunting.

Also, when inspecting either the file system or the database, it appears that Ruote serializes any data - in a JSON object - any data passes through via input, or during the processes themselves.  It looks like it's not a good idea to be passing around large files inside of workitems, since they would either be persisted in the DB or FileSystem indefinitely.  Is that a correct conclusion?

So lots of questions, but I'm interested to hear about the Sequel storage issues.

Thanks in advance for helping us.  We planned on deploying on Friday, but this is looking like a deal breaker. :(

Chad

[Chad]

By the way, you're example is using Route::Dashboard.new and mine is using Route::Engine.  The Sequel problems still occurs, if I switch to the dashboard.  It also occurs if I don't assign the engine to RuoteKit.

Thanks again,

[John Mettraux]

On Wed, Jul 25, 2012 at 05:30:11AM -0700, Chad wrote:
>
> If manage to get this working, we owe you a dinner (and more) the next time
> you're in Tokyo.

Oh, you guys are in Tokyo? Where? I'm in Hiroshima.

> So here's the story with Sequel. We are using 1.9.2-p290 and Sequel 3.37.0
> and mysql2 0.3.11.
>
> Here's a sample of the SQL traces, if we add a logger to a Sequel
> connection:
>
> https://gist.github.com/3175854
>
> The trace is on a new and clean database, with no data in it, and simply
> instantiating the Route::Engine like so:
>
> db = Sequel.connect(db_settings)
> db.loggers << OurLogging.logger
>
> RuoteKit.engine = Ruote::Engine.new(
> Ruote::Worker.new(
> Ruote::Sequel::Storage.new(db)))

OK, I'll look at it in the coming days.

Seems like there are too many

SELECT * FROM `documents` WHERE ((`typ` = 'configurations') AND (`ide` = 'engine')) ORDER BY `rev` DESC LIMIT 1

If you have a stack trace for one of the timeouts you mentioned, that'd be
much appreciated.

> Even though we started no process definitions, Ruote's Engine immediately
> hits the DB. But more importantly, it never stops. The log just grows and
> grows and grows.

Yes, as said, it polls for msgs and schedules, for those two types of data it
considers the storage as a queue.

Maybe your logger could ignore selects for 'typ' "configurations", "msgs" and
"schedules". That'd dramatically reduce the log size.

> Regarding the FsStorage, your megabyte numbers are dramatically smaller
> than ours. We are basically doing nothing as well, but I'll perform your
> tests.. I grant that disk-space is cheap, but there's not enough
> disk-space in the world available if we keep Ruote running for a year,
> without deleting some of the stuff Ruote saves on the file system. If we
> have to use FsStorage, we would need to run a cron job to clean things
> occassionally. What files would we need to purge? The structure of the
> FsStorage is rather daunting.

Well, terminated flows weigh 0K... It seems that your test involves flows
that are alive and flows that have errors.

As said before, I tend to save lots of redundant info, not to lose anything.
The storage implementations are also quite naive (not many shortcuts for fear
of backfires).

> Also, when inspecting either the file system or the database, it appears
> that Ruote serializes any data - in a JSON object - any data passes through
> via input, or during the processes themselves. It looks like it's not a
> good idea to be passing around large files inside of workitems, since they
> would either be persisted in the DB or FileSystem indefinitely. Is that a
> correct conclusion?

Ah yes, you have to avoid passing large stuff inside of workitems. Nowadays,
it's much more efficient to pass pointers to stuff (URIs for example) than
stuff itself.

> So lots of questions, but I'm interested to hear about the Sequel storage
> issues.

OK, please don't forget to give me more details about the timeouts.

> Thanks in advance for helping us. We planned on deploying on Friday, but
> this is looking like a deal breaker. :(

:-(

[John Mettraux]

On Wed, Jul 25, 2012 at 05:35:08AM -0700, Chad wrote:
>
> By the way, you're example is using Route::Dashboard.new and mine is using
> Route::Engine. The Sequel problems still occurs, if I switch to the
> dashboard. It also occurs if I don't assign the engine to RuoteKit.

Hello again Chad,

Ruote::Dashboard and Ruote::Engine are interchangeable. Dashboard is newer,
it's meant to emphasize the "hey, the engine is the sum of workers +
storages, not the actual Ruote::Engine instance", but, well Ruote::Engine
sticks.

So it won't fix any storage issues.

[Steve Sexton]

I am also trying to bring up a new ruote implementation using Ruote::Sequel.  On my last post John recommended to use edge Ruote rather than official 2.2.0 release.  We still haven't done that, but we did find, when we were doing SQL performance tuning for the app, that Ruote::Sequel does a TON of queries on documents using (typ, ide, rev) as keys.  These showed up in the MySQL slow query log, too.  There is no index for that query!  Once we added the index we've had much better performance with ruote.

 

So, you might want to try adding an index on documents table and see if that helps.  We use rails, so we did it with a migration:

  add_index :documents, [:type, :ide, :rev] 

 

hth,

  Steve

[John Mettraux]

Hello Steve,

that's weird. Even the 2.2.0 version you're using has

primary_key [ :ide, :rev, :typ ]

https://github.com/jmettraux/ruote-sequel/blob/a815d93e55af59e82250614e0cddfd9c68334638/lib/ruote/sequel/storage.rb#L49

Playing with PostgreSQL and explain, the index associated with the primary
key makes it in.

Anyway, I changed a bit:

https://github.com/jmettraux/ruote-sequel/commit/61e09fe0c96c1dc25a17c9d0a74ae155851b8a0b

What database are you using? Any other tip that could benefit us?

(I'll anyway take some time to a) make sure ruote queries the storage less
(for example there are too many get('configuration', 'engine') calls b)
implement some simple optimization in the storages themselves).

[Steve Sexton]

 

This is with MySQL 5.5.22 and InnoDB table.  Agreed that the primary_key [ :ide, :rev, :typ ] is there, even on my old version.

 

My memory is a little fuzzy, but I think that the query was ORDER BY typ, ide, rev without any where clause.

I'm pretty sure that explain showed no index and file sort, which made total sense to me once I saw the table definition.

I had to add an index, changing the primary key just turned up a different set of queries in the slow query log.

At any rate, once I added the index, my ruote got a lot more stable, and faster.

 

Beats me why pg would say that the primary key is usable - I would think it would at least have to sort the index - but for MySQL, there are definitely some common queries where the primary key isn't used.

[Chad]

Hi John,

Thanks for your help again.  

Yes, we are in Tokyo, in the  'pleasure-district' of Gotanda. :)  And I'm serious about that dinner!

I'm not really worried about the logging, filling up our district space.  It's just the number of SQL requests that our hitting our databases causes disconnects, which results in 

Here's a trace of what happens when the SQL requests hose our server:

Sequel::DatabaseDisconnectError: Mysql2::Error: MySQL server has gone away

  /mnt/app/shared/bundle/ruby/1.9.1/gems/sequel-3.37.0/lib/sequel/adapters/mysql2.rb:90:in `query'

  /mnt/app/shared/bundle/ruby/1.9.1/gems/sequel-3.37.0/lib/sequel/adapters/mysql2.rb:90:in `block in _execute'

  /mnt/app/shared/bundle/ruby/1.9.1/gems/sequel-3.37.0/lib/sequel/database/logging.rb:33:in `log_yield'

  /mnt/app/shared/bundle/ruby/1.9.1/gems/sequel-3.37.0/lib/sequel/adapters/mysql2.rb:90:in `_execute'

  /mnt/app/shared/bundle/ruby/1.9.1/gems/sequel-3.37.0/lib/sequel/adapters/shared/mysql_prepared_statements.rb:23:in `block in execute'

  /mnt/app/shared/bundle/ruby/1.9.1/gems/sequel-3.37.0/lib/sequel/database/connecting.rb:229:in `block in synchronize'

  /mnt/app/shared/bundle/ruby/1.9.1/gems/sequel-3.37.0/lib/sequel/connection_pool/threaded.rb:105:in `hold'

  /mnt/app/shared/bundle/ruby/1.9.1/gems/sequel-3.37.0/lib/sequel/database/connecting.rb:229:in `synchronize'

  /mnt/app/shared/bundle/ruby/1.9.1/gems/sequel-3.37.0/lib/sequel/adapters/shared/mysql_prepared_statements.rb:23:in `execute'

  /mnt/app/shared/bundle/ruby/1.9.1/gems/sequel-3.37.0/lib/sequel/adapters/mysql2.rb:75:in `execute_insert'

  /mnt/app/shared/bundle/ruby/1.9.1/gems/sequel-3.37.0/lib/sequel/dataset/actions.rb:760:in `execute_insert'

  /mnt/app/shared/bundle/ruby/1.9.1/gems/sequel-3.37.0/lib/sequel/dataset/actions.rb:299:in `insert'

  /mnt/app/shared/bundle/ruby/1.9.1/bundler/gems/ruote-sequel-b67adc8ac809/lib/ruote/sequel/storage.rb:327:in `do_insert'

  /mnt/app/shared/bundle/ruby/1.9.1/bundler/gems/ruote-sequel-b67adc8ac809/lib/ruote/sequel/storage.rb:97:in `put_msg'

  /mnt/app/shared/bundle/ruby/1.9.1/bundler/gems/ruote-8eaa5c25b015/lib/ruote/receiver/base.rb:130:in `launch'

  /mnt/app/releases/20120725060913/my-app.rb:31:in `process'

  /mnt/app/releases/20120725060913/my-app-api.rb:41:in `block in <class:App>'

[Chad]

Hi John (again!).

So I've explored the FsStorage.  I ran the same scripts that you sent, and I get dramatically different results.  While your'e script produces 12k on finish, mine ends at 692k.

I would love to hear how I could add long-polling, that you suggested.

[John Mettraux]

On Wed, Jul 25, 2012 at 10:10:50PM -0700, Chad wrote:
>
> So I've explored the FsStorage. I ran the same scripts that you sent, and
> I get dramatically different results. While your'e script produces 12k on
> finish, mine ends at 692k.

Hello Chad,

I ran my test on Snow Leopard, vanilla Mac OS Extended (Journaled).

How about you?

[John Mettraux]

On Wed, Jul 25, 2012 at 10:03:53PM -0700, Chad wrote:
>
> I'm not really worried about the logging, filling up our district space.
> It's just the number of SQL requests that our hitting our databases causes
> disconnects, which results in
>
>
> Here's a trace of what happens when the SQL requests hose our server:
> Sequel::DatabaseDisconnectError: Mysql2::Error: MySQL server has gone away
>
> /mnt/app/shared/bundle/ruby/1.9.1/gems/sequel-3.37.0/lib/sequel/adapters/mysql2.rb:90:in

Hello again Chad,

may I have more info about your setup? Are you experiencing those errors with
a forking server (passenger, unicorn, ...)?

Thanks in advance,

[John Mettraux]

On Wed, Jul 25, 2012 at 10:03:53PM -0700, Chad wrote:
>

> Yes, we are in Tokyo, in the 'pleasure-district' of Gotanda. :) And I'm
> serious about that dinner!

That'd be great!

> I'm not really worried about the logging, filling up our district space.
> It's just the number of SQL requests that our hitting our databases causes
> disconnects, which results in
>
> Here's a trace of what happens when the SQL requests hose our server:
> Sequel::DatabaseDisconnectError: Mysql2::Error: MySQL server has gone away

After reading https://github.com/jeremyevans/sequel/issues/368

I'm also wondering: do those timeout errors appear if you use the [slower]
mysql driver instead of mysql2?

Would you have time to check?


Thanks in advance,

[Chad]

I'm running Debian Unstable (pretty much the latest ubuntu) with ext4 file system.  Our deployment environments are Ubuntu with ext4 too, so I expect the same result.

Chad

[Chad]

We are using a forking passenger.  And a Amazon RDS (mysql).  Any other details that you need?  I can provide as much as I know.

Chad

[John Mettraux]

On Wed, Jul 25, 2012 at 10:44:54PM -0700, Chad wrote:
>
> We are using a forking passenger. And a Amazon RDS (mysql). Any other
> details that you need? I can provide as much as I know.

I was looking at

https://groups.google.com/forum/?fromgroups#!topic/sequel-talk/wNoFul3dmEg

The third message is showing an after fork reconnect technique. I wonder if
that could help you guys.

[Chad]

Cool, I'll email you offline with my company email address.  Looking forward to it.

I have all the time in the world, since the current state of my ruote app is unreleasable.  :)    I'll test the old mysql gem.  

Chad

[Chad]

Hi John,

It appears that PhusionPassenger forking technique seems to prevent the SQL disconnects.  It's sort of scary, but we are going to test it against it.

Ruote's SQL behaviors seems to be this.  On start up, make a massive about of SQL requests.  Then it settles down.  I start creating jobs, and the SQL requests seem to consistently stop with the job stops.  

Thanks for the pointers,

Chad

[John Mettraux]

On Thu, Jul 26, 2012 at 01:12:28AM -0700, Chad wrote:
>
> It appears that PhusionPassenger forking technique seems to prevent the SQL
> disconnects. It's sort of scary, but we are going to test it against it.

Hello Chad,

it's not scary, it's something to be aware of when using those forking
servers. It's painful to learn.

> Ruote's SQL behaviors seems to be this. On start up, make a massive about
> of SQL requests. Then it settles down. I start creating jobs, and the SQL
> requests seem to consistently stop with the job stops.

As should be.

That's great. Please keep me posted. Cheers,

[Chad]

Hi John,

I regret to say that the Passenger solution is not holding.  We continue to experience the disconnects.  It just takes alot longer.  Once ruote (or ruote sequel) encounters one disconnect, this appears to be the behavior: Ruote continues to accept incoming requests, but process_def/participants are in limbo.  They are added to the database, and issued an id.  However, they are unknown if we attempt to find them with engine.process(the-id).  

When then restart our webservice built around Ruote, and, of course, a new engine is created, connecting to the database.  Now, the engine starts doing its work, and engine.process(the-id) finds the process.

Does this sound familiar?  Any advice?

Chad

[John Mettraux]

On Thu, Jul 26, 2012 at 02:14:29AM -0700, Chad wrote:
>
> I regret to say that the Passenger solution is not holding. We continue to
> experience the disconnects. It just takes alot longer. Once ruote (or
> ruote sequel) encounters one disconnect, this appears to be the behavior:
> Ruote continues to accept incoming requests, but process_def/participants
> are in limbo. They are added to the database, and issued an id. However,
> they are unknown if we attempt to find them with engine.process(the-id).

Sounds right, once the connection is borked, there is nothing much
ruote-sequel can do...

But wait, what is strange is that Engine#process(wfid) should raise that same
disconnect error when being called... I will investigate that.

> When then restart our webservice built around Ruote, and, of course, a new
> engine is created, connecting to the database. Now, the engine starts
> doing its work, and engine.process(the-id) finds the process.
>
> Does this sound familiar? Any advice?

No, it doesn't sound familiar (unlike the post fork disconnects).

Would you still have time to test with "mysql" (instead of "mysql2")?

Their sequel drivers seem to have different reconnection code, so I'd love to
know how mysql holds. Maybe mysql2 and sequel are not in sync (I'm probably
wrong since Mysql2 0.3.11 is from December 2011, while Sequel 3.37.0 is from
July 2012, but I need to know anyway).

Meanwhile I'll investigate those silent Engine#process(wfid) failures you
mentioned.

[Chad]

Hi John,

Thanks for looking into this.  We have time to test mysql.  At the moment, though, we are going to try setting up Redis.  We have it up locally, but we are reverse-engineering how our DevOps team deploys redis - (No DevOps here in Japan, but in the US).

Chad

[John Mettraux]

On Thu, Jul 26, 2012 at 03:06:32AM -0700, Chad wrote:
>
> Thanks for looking into this. We have time to test mysql. At the moment,
> though, we are going to try setting up Redis. We have it up locally, but
> we are reverse-engineering how our DevOps team deploys redis - (No DevOps
> here in Japan, but in the US).

Hello Chad,

beware testing too many storages and skipping at the first shallow issue. At
some point, nails have to be nailed, 'til the head.

[Chad]

Hi John,

Thanks for the encouragement.  I agree.  It's sort of also hard to believe that Ruote can generate so much SQL traffic that an Amazon RDS instance shuts down the connection.  

It seems, though, that Ruote is not happy after it encounters such an error, since it accepts new jobs, but doesn't run them. :(

Tonight, we are going to work with DevOps, to see what's going on.  Meanwhile, the Redis Storage is not an option for us, on our platform, but Dynamo DB is.  Until we can figure out, with our DevOps teams, why RDS would respond as it does, I'm probably going to write a proof of concept Dynamo storage.  Our DevOps team is sleeping, so not much I can do in the meantime.

By the way, did you get my email that I sent to your gmail account last night?

Chad

[John Mettraux]

On Thu, Jul 26, 2012 at 08:46:21PM -0700, Chad wrote:
>
> Thanks for the encouragement. I agree. It's sort of also hard to believe
> that Ruote can generate so much SQL traffic that an Amazon RDS instance
> shuts down the connection.

Hello,

that's a new piece of information. So far the only concrete thing that came
out of the thread was that some connections were lost because of the forking.

How do you reach the conclusion that RDS shuts down the connection? Are we
sure it's a corner case of the forking problem?

> It seems, though, that Ruote is not happy after it encounters such an
> error, since it accepts new jobs, but doesn't run them. :(

I did some tests (with the mysql2 driver), they look like this:

https://gist.github.com/3186470

sequel reconnects each time easily.

> Tonight, we are going to work with DevOps, to see what's going on.
> Meanwhile, the Redis Storage is not an option for us, on our platform, but
> Dynamo DB is. Until we can figure out, with our DevOps teams, why RDS
> would respond as it does, I'm probably going to write a proof of concept
> Dynamo storage. Our DevOps team is sleeping, so not much I can do in the
> meantime.

So how does RDS respond? Anything in its log?

If you want me to be useful you have to tell me all those details that matter
(for example, you guys told me about Passenger and RDS in message 16/25 of
this thread ;-) )

> By the way, did you get my email that I sent to your gmail account last
> night?

Hello, yes, thanks a lot. You guys'll owe me a dinner after one year of
stability in production ;-)

I'd be glad to work on a sequel storage that polls less, but that's not
doable as a 3 hours hack.


Best regards,

[John Mettraux]

Hello,

as a side note, I've setup ruote's CI to test with mysql2 and I'm getting two

Sequel::DatabaseDisconnectError: Mysql2::Error: This connection is still
waiting for a result, try again once you have the result

The "mysql" has no such problem.

I don't know if you experienced such disconnects. It seems the "mysql" driver
is better.

Best regards,

[John Mettraux]

Hello,

I'm currently cooking up an enhanced version of ruote-sequel (2.3.0) that
should query significantly less. I hope to be done before Friday.

Best regards,

[Chad]

That's awesome.  We've been watching the github activity. 

We still haven't detected why we are experiencing disconnects.  I think it's passenger related.  If we start our app on amazon, using rackup (hitting thin), it seems like the issue goes away.  It's just unpredictable, though.

I've also been writing a dynamo db storage too.  Probably 75% done.  Now, I'm in debugging mode, bug hunting.

Chad

[Hartog De Mik]

When it is a passenger problem; Passenger has hooks for reconnecting
after a fork - here is one example (of many) :
http://ryreitsma.blogspot.de/2011/07/redis-and-phusion-passenger-reconnect.html

Hih,
Grtz,
Hartog.

2012/7/30 Chad <cal...@mdsol.com>:

> --
> you received this message because you are subscribed to the "ruote users"
> group.
> to post : send email to openwfe...@googlegroups.com
> to unsubscribe : send email to openwferu-use...@googlegroups.com
> more options : http://groups.google.com/group/openwferu-users?hl=en

[John Mettraux]

> On Monday, July 30, 2012 2:44:40 PM UTC+9, John Mettraux wrote:
> >
> > I'm currently cooking up an enhanced version of ruote-sequel (2.3.0) that
> > should query significantly less. I hope to be done before Friday.

Hello Chad and list,

this "query less" version is at

https://github.com/jmettraux/ruote-sequel/tree/begin_step

when running a set of 4 sequence tests it cuts the number of selects from
~274 to ~154. It does it by having the worker do a single select for all the
info it needs.

I have only tried it with mysql (not yet tried with mysql2).

I still have to try a few other enhancements (less process in the storage,
more in the db, enhanced #reserve implementation, etc).

If I could have some kind of feedback about it. I'd like to merge it into
master at some point.


Thanks in advance,

[John Mettraux]

2012/8/7 John Mettraux <jmet...@gmail.com>:

>
> I have only tried it with mysql (not yet tried with mysql2).

Actually, it seems to work fine with mysql2.

--
John Mettraux - http://lambda.io/processi