Change storage implementations in production and other questions :)
Nathan
We had an experience this weekend where we attempted to roll out our
ruote-powered application to a new segment of users. However, we had
to roll back our efforts pretty quickly because our work item
processing started taking up to 10 minutes, particularly when creating
new workflows (as opposed to advancing living workflows, even though
that slowed to a crawl as well). Our application serves work items in
real time via a UI to users based on their actions and the workflow
definitions, so we are hoping for response times of a few seconds at
most.
On Monday we are going to start picking things apart, trying to figure
out what is wrong with our setup. We threw together the MongoDB
storage late last year and have been using it since, but we haven't
really load tested it, updated it for the latest version of Ruote, or
tried to make it work with multiple workers. We have noticed however
that it has a pretty high CPU utilization which has been growing over
time and now rests at over 50%.
Anyway, the first thing I want to try when troubleshooting this is
swapping out the MongoDB storage for another storage, preferably Redis
based on speed. If that works well, then I know the culprit is our
storage adapter. Otherwise, I'll have to dig deeper. My main question
is this:
* is there a reasonable way to migrate ruote from one storage to
another? I'd like to do our test on a copy of the production database.
My next question is pretty broad, so I apologize for that, but
* Are there any known performance bottlenecks or hot spots we should
be looking at? We will profile of course, but if there are some
obvious places to put the tip of the chisel that would be great to
know.
Also,
* I am guessing we have a number of workflows in the database that are
"dead" or "orphaned" - workflows and processes that, due to exceptions
or un-clean resets were never completed or cancelled. Could this
affect performance in a significant way? Should we routinely attempt
to clean out orphans?
Currently our ruote database (in MondoDB) is 1.4GB with about 3K
schedules and 190K expressions. Our workflows are pretty big - so each
expression is fairly large in size. Maybe much of this is cruft, not
sure - but I'm curious how our setup compares to others? Is this
large, average, very small? Do any of you have experience with DB
sizes this big or bigger? How long should it take to launch a workflow
of substantial size?
Thanks for your time and insight,
Nathan
John Mettraux
On Sat, Nov 12, 2011 at 03:32:31PM -0800, Nathan wrote:
>
> We had an experience this weekend where we attempted to roll out our
> ruote-powered application to a new segment of users. However, we had
> to roll back our efforts pretty quickly because our work item
> processing started taking up to 10 minutes, particularly when creating
> new workflows (as opposed to advancing living workflows, even though
> that slowed to a crawl as well). Our application serves work items in
> real time via a UI to users based on their actions and the workflow
> definitions, so we are hoping for response times of a few seconds at
> most.
>
> On Monday we are going to start picking things apart, trying to figure
> out what is wrong with our setup. We threw together the MongoDB
> storage late last year and have been using it since, but we haven't
> really load tested it, updated it for the latest version of Ruote, or
> tried to make it work with multiple workers. We have noticed however
> that it has a pretty high CPU utilization which has been growing over
> time and now rests at over 50%.
Hello Nathan,
here are a few questions/ideas.
Ruote is polling for msgs and schedules. Could this be the cause ? An engine with a FsStorage polls every at least twice per second, it typically uses 0.3% of the CPU (mac osx).
You should try to determine why it's so busy. Try to stash together a single file script involving the engine, the MongoDB storage and two or three workflow runs and measure.
Maybe we could work together on the MongoDB storage (it could give me an occasion to learn about MongoDB).
> Anyway, the first thing I want to try when troubleshooting this is
> swapping out the MongoDB storage for another storage, preferably Redis
> based on speed. If that works well, then I know the culprit is our
> storage adapter. Otherwise, I'll have to dig deeper. My main question
> is this:
>
> * is there a reasonable way to migrate ruote from one storage to
> another? I'd like to do our test on a copy of the production database.
All the storages have a #copy_to method whose default implementation is
https://github.com/jmettraux/ruote/blob/master/lib/ruote/storage/base.rb#L242-267
Here is an example usage:
https://github.com/jmettraux/ruote/blob/master/test/functional/ft_42_storage_copy.rb
> My next question is pretty broad, so I apologize for that, but
>
> * Are there any known performance bottlenecks or hot spots we should
> be looking at? We will profile of course, but if there are some
> obvious places to put the tip of the chisel that would be great to
> know.
Well, one worker only processes one workflow operation at a time, so when there is one worker only one workflow is alive at one time, with the exception of the participant work (see http://groups.google.com/group/openwferu-users/browse_thread/thread/e72368cf72954cd9)
> Also,
>
> * I am guessing we have a number of workflows in the database that are
> "dead" or "orphaned" - workflows and processes that, due to exceptions
> or un-clean resets were never completed or cancelled. Could this
> affect performance in a significant way? Should we routinely attempt
> to clean out orphans?
If you use Engine#processes a lot then yes, the default implementation lists all expressions and errors to build ProcessStatus instances. It could be losing time on listing dead stuff.
> Currently our ruote database (in MongoDB) is 1.4GB with about 3K
> schedules and 190K expressions.
3K schedules, It could be the problem. The current default implementation of #get_schedules is
https://github.com/jmettraux/ruote/blob/master/lib/ruote/storage/base.rb#L177-195
As you can see, a potential optimization is commented out. This dumb implementation goes "let's fetch all the schedules in memory and then filter the ones that have to get triggered now".
The solution would be, in ruote-mongodb, to override #get_schedules and do the filtering in MongoDB itself so that those 3K schedules don't get copied twice per second.
ruote-couch (sorry not the fastest) for example, has a dedicated #get_schedules implementation.
I just checked ruote-redis and it hasn't any optimization (I have to do something about that).
> Our workflows are pretty big - so each
> expression is fairly large in size. Maybe much of this is cruft, not
> sure - but I'm curious how our setup compares to others? Is this
> large, average, very small? Do any of you have experience with DB
> sizes this big or bigger?
Which JSON library are you using ? I'm happy with yajl-ruby, it's faster than all the others.
I personally only have experience with much smaller deployments. I'd rate your shop as big. I wonder how David's Meego team compares.
> How long should it take to launch a workflow
> of substantial size?
The launch itself is only about copying the whole tree and the initial workitem fields, it shouldn't be that time-consuming.
So, I'd investigate #get_schedules, an easy step is to measure how long the call to get_schedules in lib/ruote/worker.rb is taking, and then do something like "get_schedules = get all the schedules where at < now"
I hope this helps,
--
John Mettraux - http://lambda.io/processi
Nathan Stults
environment to take measurements and apply some realistic loads and take
a closer look at all the points you mentioned. One question on the
schedules - if the behavior of a worker is to pull all schedules and
fire triggered ones, how does this work in a multi-worker environment?
Is that what "reserve" is used for in the storage? (We haven't
implemented reserve in MongoDB, but probably should)
Once we get this sorted out I would love to work with you on the MongoDB
driver, that is a great suggestion.
Thanks again,
Nathan
Hello Nathan,
https://github.com/jmettraux/ruote/blob/master/lib/ruote/storage/base.rb
#L242-267
https://github.com/jmettraux/ruote/blob/master/test/functional/ft_42_sto
rage_copy.rb
https://github.com/jmettraux/ruote/blob/master/lib/ruote/storage/base.rb
#L177-195
I hope this helps,
--
you received this message because you are subscribed to the "ruote
users" group.
to post : send email to openwfe...@googlegroups.com to unsubscribe
: send email to openwferu-use...@googlegroups.com
more options : http://groups.google.com/group/openwferu-users?hl=en
John Mettraux
On Mon, Nov 14, 2011 at 09:14:00AM -0800, Nathan Stults wrote:
>
> John, thank you for all the pointers. Today we will set up a test
> environment to take measurements and apply some realistic loads and take
> a closer look at all the points you mentioned. One question on the
> schedules - if the behavior of a worker is to pull all schedules and
> fire triggered ones, how does this work in a multi-worker environment?
> Is that what "reserve" is used for in the storage? (We haven't
> implemented reserve in MongoDB, but probably should)
Hello,
yes Storage#reserve(doc) is meant to return true if the worker has successfully reserved the document for its own use. It's very important for multi-worker storages to implement this method correctly. If it returns true twice for the same doc (msg or schedule) you'll end up with a workflow operation being performed twice (branches popping out of nowhere) and schedules triggering twice.
Maybe simply fixing #get_schedules will yield sufficient gain so that you can stick with one worker. We'll see.
Best regards,
Nathan
adapter to fix the schedule loop, made some adjustments to make it a
bit faster and introduced a locking scheme for multi-worker
concurrency. We tried the Redis storage but for some reason it wasn't
processing all of our messages during our load tests, could be user
error, but in any case we're sticking with the Mongo one for now even
though I think it is probably somewhat slower.
When loaded up with a number of simultaneous, large workflow launches
that produce a number (8-10) additional work items things are still
pretty slow. I noticed that the slowest workflows to go from launch to
equilibrium have a large number of "set" expressions to set variables
and fields. We also have a lot of participant (and other) expressions
that are conditional using "if" and are usually skipped. We have used
these pretty liberally in our workflow code.
In profiling, it turns out that nearly each variable set *appears* to
cause the process to persist via a "put". I think I can mitigate this
to some extent by combining evented IO via event machine with writing
a worker implementation that puts message dispatch into a push-fed EM
event loop (instead of the standard polling loop), but I get the
feeling the JSON serialization / de-serialization cost is adding up,
and that of course is CPU bound. If I modify the ruote code to force
'should_persist' to false in 'un_set_variable' the difference in
performance is dramatic, but I bet my tests wouldn't pass that way,
although I'm unsure of the ramifications actually.
My question is about when ruote decides it needs persist? My guess
would have been that persistence only occurs just prior to unloading a
workflow process because all paths have led to a dead end requiring
external stimulus, but that doesn't seem to be the case. We have a lot
of business rules modeled using flow expressions and variable sets, as
well as a lot of conditional participant expressions, and I figured
these were probably nearly free from a performance perspective. If
this is not the case though, for instance if these branches are
setting of variables are actually causing ruote to save the document
and put a continuation on the message queue, we may need to refactor
our workflows to put all those business rule calculations into
external helpers.
I will say this though: digging through ruote's code and tests is
teaching me a lot. Reading good code is always such a rewarding
experience.
Thank you so much for your time,
Nathan
John Mettraux
On Wed, Nov 16, 2011 at 06:07:13PM -0800, Nathan wrote:
>
> Hi John. We've been hammering at this all week. We updated our MongoDB
> adapter to fix the schedule loop, made some adjustments to make it a
> bit faster and introduced a locking scheme for multi-worker
> concurrency. We tried the Redis storage but for some reason it wasn't
> processing all of our messages during our load tests, could be user
> error, but in any case we're sticking with the Mongo one for now even
> though I think it is probably somewhat slower.
>
> When loaded up with a number of simultaneous, large workflow launches
> that produce a number (8-10) additional work items things are still
> pretty slow. I noticed that the slowest workflows to go from launch to
> equilibrium have a large number of "set" expressions to set variables
> and fields.
Hello Nathan,
you could cut down the number of those initial "sets", by passing the initial
workitem fields and variables when launching:
fields = { 'a' => 'b' }
variables = { 'c' => 'd' }
wfid = dashboard.launch(pdef, fields, variables)
There is also the set_fields expression which lets you set a batch a fields
with one expression:
set_fields :val => { 'customer' => { 'name' => 'Fred', 'age' => 40 } }
It's documented at:
http://ruote.rubyforge.org/exp/restore.html
There is no "set_variables" expression, should you need one, it's very easy
to implement.
> We also have a lot of participant (and other) expressions
> that are conditional using "if" and are usually skipped. �We have used
> these pretty liberally in our workflow code.
You're right, 'if' and 'unless' are evaluated right after the expression got
persisted initially. IIRC I do it this way in order to have simpler code (and
also in order to have the expression at hand in case of error in the
if/unless (a target for replay_at_error).
I want people to use if/unless liberally, it makes code much more readable.
Eventually, if it proves a big perf sink, we could optimize that and forgo
the initial persist, eval and reply immediately.
> In profiling, it turns out that nearly each variable set *appears* to
> cause the process to persist via a "put".
Yes, each time you save a variable that results in having the expression
holding the variable getting persisted.
> I think I can mitigate this
> to some extent by combining evented IO via event machine with writing
> a worker implementation that puts message dispatch into a push-fed EM
> event loop (instead of the standard polling loop), but I get the
> feeling the JSON serialization / de-serialization cost is adding up,
> and that of course is CPU bound.
Yes, this is costly, I'm using YAJL, it's even faster than Marshal for some
version of Rubies (I have to test with 1.9.2+).
> If I modify the ruote code to force
> 'should_persist' to false in 'un_set_variable' the difference in
> performance is dramatic, but I bet my tests wouldn't pass that way,
> although I'm unsure of the ramifications actually.
Let's look at that code
---8<---
01 def un_set_variable(op, var, val, should_persist)
02
03 if op == :set
04 Ruote.set(h.variables, var, val)
05 else # op == :unset
06 Ruote.unset(h.variables, var)
07 end
08
09 if should_persist && r = try_persist # persist failed, have to retry
10
11 @h = r
12 un_set_variable(op, var, val, true)
13
14 else # success (even when should_persist == false)
15
16 @context.storage.put_msg("variable_#{op}", 'var' => var, 'fei' => h.fei)
17 end
18 end
--->8---
If should_persist is false, it means the variable/value binding is not saved.
If it yields a great increase in performance, it probably means #try_persist
tends to be unsucessful and triggers a recursion. You can verify that by
placing some logging output at line 10 and observe what happens.
should_persist is set to false, by some expressions that set series of
variables (or other expression attributes) and then persist.
> My question is about when ruote decides it needs persist? My guess
> would have been that persistence only occurs just prior to unloading a
> workflow process because all paths have led to a dead end requiring
> external stimulus, but that doesn't seem to be the case.
Ruote "implements" concurrency by splitting workflow instances into
expressions. A concurrence expression places an "apply" message for each of
its children on the work queue, and if you have multiple workers, each of
those children may end up getting applied by a different worker.
When an external answer comes back for a participant, only the
required participant expression is fetched back from the storage
There is no concept of loading all the expessions of a workflow instance and
then saving them all. Although one could imagine implementing a storage that
does that (in fact I have already done that, but it's proprietary software),
but it requires some warranties that only one worker is processing the
messages for a give workflow instance at any time.
I've been striving for the simplest possible concepts and you're probably
hitting a limitiation of my design or there is something wrong with your
storage implementation (most likely both + some inefficiencies in ruote
certainly). Fortunately, we can measure.
(It'd be interesting to know the cost of persisting one big expression, maybe
you could log that (with exp size) from the storage implementation).
(Could we forgo JSON and use BSON hashes directly ? Crazy question, don't
know if it makes sense)
> We have a lot of business rules modeled using flow expressions and variable
> sets, as well as a lot of conditional participant expressions, and I figured
> these were probably nearly free from a performance perspective. If
> this is not the case though, for instance if these branches and
> setting of variables are actually causing ruote to save the document
> and put a continuation on the message queue, we may need to refactor
> our workflows to put all those business rule calculations into
> external helpers.
Ruote tends to save as soon as possible so that it doesn't get caught with
inconsistent workflow instances. Persistence always has priority over
performance (hence the escape to multiple workers).
> I will say this though: digging through ruote's code and tests is
> teaching me a lot. Reading good code is always such a rewarding
> experience.
Thanks :-) Maybe I could have a look at the latest version of ruote-mongodb
with you.
Please don't hesitate to hammer the list with questions, the replies I just
wrote are probably not sufficient. I'm intrigued by your should_persist
finding.
I'd be happy if we/you found inconsistencies/inefficiencies in ruote with
this work.
Cheers,
Nathan
with performance goals, so I'm going to deploy things as I have them
now and see how it runs in production. Our load tests were simulating
substantially more concurrent users than we really have, so I think
we'll be ok for some time (crossing my fingers anyway).
I do agree that the design of ruote is heavily focused on reliability
and consistency over speed, which is definitely what you want in the
kind of system ruote was designed for. While in the future it would be
nice if ruote could be optionally tuned for speed at the cost of
consistency in the face of any kind of failure specifically for real
time user facing types of systems where, and if that day ever comes I
have some thoughts, but I think we can get by just fine with a couple
more worker processes and the tweaks I've already made to the mongo
storage.
You're comment about BSON vs JSON did make me realize that I was
needlessly serializing documents to JSON, or using Rufus::Json.dup,
even though they would then be serialized by the mongo driver into
BSON, so I quit doing that. It caused a few minor problems, which you
can see from this workaround method:
https://gist.github.com/1378141
Also, one unit test is failing for the mongo driver, but I'm not the
rest must be so string as the functional tests pass:
test/unit/storage.rb:167
<"z"> expected but was <:z>
BSON will pull out a symbol if it puts in a symbol. For document
values (not keys) is this really an issue for ruote?
Finally, as a side note, I did adapt the mongo driver to use em-mongo
and evented IO, and I wrote an asynchronous worker to take advantage
of that, but it performed much much worse than serial IO. I believe
this is because of the vast number of writes that require locking, so
with the high speed loop I'm thinking lock contention combined with
the high CPU load was too much.
As for the mongo storage, I would love to go through the code with
you. I'm sure it could be made much better. I'm not sure how we would
do it, but I'm very open to it any time you like.
Thanks again for your help. I may have said this before, but Ruote is
the single best supported open source project I've used. I don't know
how you find the time, but your responsiveness, detail and patience
are really a very strong feature of Ruote.
Nathan
> > that are conditional using "if" and are usually skipped. �We have used
John Mettraux
On Fri, Nov 18, 2011 at 04:56:39PM -0800, Nathan wrote:
>
> Thank you for the detailed response. I think I'm being too aggressive
> with performance goals, so I'm going to deploy things as I have them
> now and see how it runs in production. Our load tests were simulating
> substantially more concurrent users than we really have, so I think
> we'll be ok for some time (crossing my fingers anyway).
>
> I do agree that the design of ruote is heavily focused on reliability
> and consistency over speed, which is definitely what you want in the
> kind of system ruote was designed for. While in the future it would be
> nice if ruote could be optionally tuned for speed at the cost of
> consistency in the face of any kind of failure specifically for real
> time user facing types of systems where, and if that day ever comes I
> have some thoughts, but I think we can get by just fine with a couple
> more worker processes and the tweaks I've already made to the mongo
> storage.
Hello Nathan,
at one hand of the speed spectrum there is the in-memory storage. It complicates the architecture, but one could imagine having a dedicated in-memory backed engine for transient processes and a more persistent one for more important processes.
Ruote-redis should perform well (especially the latest version).
I'm currently working on a [proprietary] storage that only saves the various expressions when all the work for them is done. It's nice, but it has its own issues (when is work "done", and potential loops that make it never reach "done")
The dumb "save and be safe" model followed by ruote has its advantages and shouldn't feel too weak when there are humans in the loop.
> (...)
>
> As for the mongo storage, I would love to go through the code with
> you. I'm sure it could be made much better. I'm not sure how we would
> do it, but I'm very open to it any time you like.
You don't mind if I give a try at it from scratch ? (So I can learn MongoDB as I go).
> Thanks again for your help. I may have said this before, but Ruote is
> the single best supported open source project I've used. I don't know
> how you find the time, but your responsiveness, detail and patience
> are really a very strong feature of Ruote.
I simply try to respond as quickly as possible and to have a ruote lean enough to immediately come up with test cases to validate feedback or continue the discussion.
Thanks for helping other users as well like you did, much appreciated !
Nathan Stults
my tests our MongoDB storage was almost exactly 2x slower than Redis,
with loads of varying sizes, but there isn't a good reason for this to
be the case that I can think of, so we are doing something wrong I
think. (Ruote doesn't call "get_many" for expressions during the normal
course of executing a flow does it? )
I spent a few more hours reading the code for the expressions this
weekend, and I finally think I have a complete grasp of how everything
fits together at a high level. I believe there is a way to dramatically
improve the performance and scalability characteristics of Ruote without
squeezing more performance out of the storage or tinkering with when an
expression chooses to persist itself just by changing the way Ruote
processes expressions a little bit. I'm going to play with this in my
fork and if it bears fruit, I'll ask you to take a peek and see what you
think.
-----Original Message-----
From: openwfe...@googlegroups.com
[mailto:openwfe...@googlegroups.com] On Behalf Of John Mettraux
Sent: Sunday, November 20, 2011 4:23 PM
To: openwfe...@googlegroups.com
Subject: Re: [ruote:3307] Re: Change storage implementations in
production and other questions :)
Hello Nathan,
Cheers,
--
John Mettraux
On Mon, Nov 21, 2011 at 03:06:41PM -0800, Nathan Stults wrote:
> Probably a good idea to try from scratch with the MongoDB adapter. In
> my tests our MongoDB storage was almost exactly 2x slower than Redis,
> with loads of varying sizes, but there isn't a good reason for this to
> be the case that I can think of, so we are doing something wrong I
> think.
Hello Nathan,
isn't Redis faster than MongoDB (having less features and working mostly in memory) ? They're really two different beasts.
> (Ruote doesn't call "get_many" for expressions during the normal
> course of executing a flow does it? )
No, it doesn't.
> I spent a few more hours reading the code for the expressions this
> weekend, and I finally think I have a complete grasp of how everything
> fits together at a high level. I believe there is a way to dramatically
> improve the performance and scalability characteristics of Ruote without
> squeezing more performance out of the storage or tinkering with when an
> expression chooses to persist itself just by changing the way Ruote
> processes expressions a little bit. I'm going to play with this in my
> fork and if it bears fruit, I'll ask you to take a peek and see what you
> think.
Looking forward to it ! Thanks.
Nathan Stults
>> Probably a good idea to try from scratch with the MongoDB adapter.
In
>> my tests our MongoDB storage was almost exactly 2x slower than Redis,
>> with loads of varying sizes, but there isn't a good reason for this
to
>> be the case that I can think of, so we are doing something wrong I
>> think.
>Hello Nathan,
>isn't Redis faster than MongoDB (having less features and working
mostly in memory) ? They're really two different beasts.
Yep. I thought the gap was too much, even given that, but I guess not:
http://stackoverflow.com/questions/5252577/how-much-faster-is-redis-than
-mongodb
-----Original Message-----
From: openwfe...@googlegroups.com
[mailto:openwfe...@googlegroups.com] On Behalf Of John Mettraux
Sent: Monday, November 21, 2011 3:40 PM
To: openwfe...@googlegroups.com
Subject: Re: [ruote:3309] Re: Change storage implementations in
production and other questions :)
Hello Nathan,
No, it doesn't.
--
John Mettraux
On Mon, Nov 21, 2011 at 03:06:41PM -0800, Nathan Stults wrote:
>
> >
> > As for the mongo storage, I would love to go through the code with
> > you. I'm sure it could be made much better. I'm not sure how we would
> > do it, but I'm very open to it any time you like.
>
> You don't mind if I give a try at it from scratch ? (So I can learn
> MongoDB as I go).
Hello Nathan,
here's my take on the ruote + MongoDB:
https://github.com/jmettraux/ruote-mon
It's using :safe => true and all the Mongoodness available. It's multi-worker safe.
I wanted to have an idea of its relative speed, so I ran the whole functional test suite of the latest ruote (b66b34f) on Ruby 1.9.2p290. 2.4 GHz Intel Core 2 Duo, with 4GB (Macbook 2008), (3 tries each).
Ruote::HashStorage 303 / 300 / 302 seconds
Ruote::FsStorage 362 / 363 / 353 s
Ruote::Redis::Storage 388 / 389 / 389 s (redis 2.2.12)
Ruote::Mon::Storage 410 / 413 / 413 s (mongod 2.0.1)
Ruote::Sequel::Storage 468 / 469 / 477 s (sequel 3.20.0 + pg 8.4.2)
I don't know how it would fare in your test harness.
I'm sorry, I have no result for ruote-mongodb, I've tried to run it (current master) two or three times, each time it got stuck after five or six tests.
The code is minimal, I did some tiny adaptations to the ruote test/unit/storage.rb to accomodate for BSON hashes, but not much more. Maybe the #put method can be simplified.
I didn't know MongoDB was so pleasant to work with.
What do you think ?
Nathan Stults
blindly trying to copy from the Redis adapter, and used your pure
optimistic concurrency approach instead for a noticeable speed boost.
I finished my proof of concept on an optimized storage implementation
and was partially successful. My approach was to create a storage that
would fully execute an incoming message and any messages resulting from
the application of that message persisting to in memory storage in a
sort of transaction or "unit of work" and then persist to the DB at the
end of the transaction. Concurrency is handled by locking on the WFID
rather than the expression id, and a transaction is processed without
stopping until it is complete. "Complete" means that if the expression
initially processed caused any other messages to be placed on the queue,
those are all processed as well, recursively, until no more messages are
generated (i.e. no more expressions will reply).
This approach was effective, but not dramatically so. My load test
launches 10 simultaneous workflows, replies a work item, and waits for
five more items from a concurrence. I was able to improve the
performance of this environment of the Mongo storage by about 50%, and
the Redis storage by about 30%. These percentages were consistent (even
a little better) when the load was 100 simultaneous workflows instead of
ten.
One interesting side effect of this work is that the characteristics of
Ruote's response to load are different. Originally, due to the strictly
round robin nature of putting fine grained, atomic components of a
single logical method call into a fifo queue, all work would always be
completed at roughly the same time. If the load was high, the work would
take a long time to complete, but complete pretty much all at once. With
the alternative techniques, work is completed in a first come first
serve fashion, and the workflows tended to complete one at a time
throughout the life of the test.
I was expecting more striking results, but the result is good enough for
our operations. This technique will allow us to comfortably stay with
Mongo DB for quite some time, keeping Redis in reserve if we need
another boost. Redis processed 10 workflows using 2 workers in about 10
seconds using my hybrid storage, and Mongo DB took an average of 14
seconds. By contrast, without the hybrid storage Redis averaged 15
seconds and Mongo DB 30 seconds). All mongo tests used your improved
approach to concurrency.
The experiment wasn't effective enough that I intend to package this
stuff up or make it generally available, but if anyone else is ever in
our shoes I wouldn't mind doing so upon request.
-----Original Message-----
From: openwfe...@googlegroups.com
[mailto:openwfe...@googlegroups.com] On Behalf Of John Mettraux
Sent: Wednesday, November 23, 2011 12:53 AM
To: openwfe...@googlegroups.com
Subject: Re: [ruote:3311] Re: Change storage implementations in
production and other questions :)
Hello Nathan,
https://github.com/jmettraux/ruote-mon
--
John Mettraux
On Wed, Nov 23, 2011 at 05:10:37PM -0800, Nathan Stults wrote:
>
> Awesome work. I removed all my elaborate locking, which I had been
> blindly trying to copy from the Redis adapter, and used your pure
> optimistic concurrency approach instead for a noticeable speed boost.
Hello Nathan,
glad it's useful.
After this work on ruote-mon, I'm tempted to make it the official MongoDB
ruote storage (by official I mean, I would package/ci/release/support it). I
have to say that since I worked on it, I'm interested in learning more about
MongoDB.
What do you think ?
> I finished my proof of concept on an optimized storage implementation
> and was partially successful. My approach was to create a storage that
> would fully execute an incoming message and any messages resulting from
> the application of that message persisting to in memory storage in a
> sort of transaction or "unit of work" and then persist to the DB at the
> end of the transaction. Concurrency is handled by locking on the WFID
> rather than the expression id, and a transaction is processed without
> stopping until it is complete. "Complete" means that if the expression
> initially processed caused any other messages to be placed on the queue,
> those are all processed as well, recursively, until no more messages are
> generated (i.e. no more expressions will reply).
As I wrote earlier on the thread I use this approach for a proprietary
storage implementation. It keeps executing the msgs. It's "complete" (I used
the term "done"), when all the msgs have executed or all msgs are "dispatch"
calls.
There is a weak point: if you run into a loop, "complete" won't be reached.
In our proprietary storage we have to peg the number of messages processes to
prevent the worker from dedicating itself fully to the looping flow (thus
becoming somehow a zombie worker).
I wonder if your approach is really concurrent: it's always the same worker
processing all the msgs until "complete".
I'm really happy someone worked so deeply on ruote, it's been a while.
Great work !
Nathan Stults
much oriented around our specific needs.
> There is a weak point: if you run into a loop, "complete" won't be
reached.
> In our proprietary storage we have to peg the number of messages
processes to prevent the worker from dedicating itself fully to the
looping flow (thus becoming somehow a zombie worker).
Do you mean you have workflows with an infinite loop and no dispatches
in the loop? Or do you just have huge iterators iterating very large
lists? Could you show an example? Our human-centric workflows don't tend
to have infinite loops without some kind of dispatch or timeout to
release the worker, but I might not be understanding what you are
saying.
> I wonder if your approach is really concurrent: it's always the same
worker processing all the msgs until "complete".
Well, true, the worker itself isn't concurrent, concurrency is achieved
through multiple workers, but this is more or less the way it was before
except a little more efficient for our use cases. However, this assumes
our style of workflows which are always "human provides input; ruote
calculates the next set of work and dispatches work items to AMQP;
rinse; repeat". Because the work ruote has to do in a single transaction
is always finite and usually pretty small we didn't really lose any
practical concurrency, even with a single worker. It sounds like your
use cases involve much longer running, fully automated workflows, which
wouldn't work nearly as well using the "unit of work" style for the
reasons you already mentioned.
-----Original Message-----
From: openwfe...@googlegroups.com
[mailto:openwfe...@googlegroups.com] On Behalf Of John Mettraux
Sent: Wednesday, November 23, 2011 5:50 PM
To: openwfe...@googlegroups.com
Subject: Re: [ruote:3313] Re: Change storage implementations in
production and other questions :)
Hello Nathan,
glad it's useful.
Great work !
--
John Mettraux
On Mon, Nov 28, 2011 at 10:33:04AM -0800, Nathan Stults wrote:
>
> Please do use your new Mongo storage as the official one. Ours is very
> much oriented around our specific needs.
Hello Nathan,
OK, great. I will still point to ruote-mongodb until you instruct me to stop
doing it.
> > There is a weak point: if you run into a loop, "complete" won't be
> > reached.
> > In our proprietary storage we have to peg the number of messages
> > processes to prevent the worker from dedicating itself fully to the
> > looping flow (thus becoming somehow a zombie worker).
>
> Do you mean you have workflows with an infinite loop and no dispatches
> in the loop? Or do you just have huge iterators iterating very large
> lists? Could you show an example? Our human-centric workflows don't tend
> to have infinite loops without some kind of dispatch or timeout to
> release the worker, but I might not be understanding what you are
> saying.
It mostly looks like:
repeat do
participant 'x', :if => '${y}'
end
So yes, a loop with potentially no dispatching (depending on y).
Many thanks,