Bottlenecks in engine queue
Matt Nichols
I'm running ruote 0.9.19. My engine is a CachedDbPersistedEngine
configured with an Sqs Listener and Sqs Participants. My work
typically comes in large bursts (like 20,000 long-running processes
involving human interaction). When I launch a large batch, two things
cause problems:
1). There is quite a long period where the engine queue is backed up
with the launch items
2). Even though I have some concurrent processes at the beginning,
they seem to be running in sequence
These issues combined are keeping the human interaction processes from
being initiated quickly, extending mean time to completion.
Here is my process definition (with some annotation in the trouble
areas):
class MasterWorkflow0 < OpenWFE::ProcessDefinition
def make
param :field => "image_file_id"
sequence do
concurrence do
# This sequence doesn't seem to start until the image_status
participant has responded to the engine.
sequence :tag => "celebrity_tagging" do
# Human Process
participant "tagging", :type => "celebrity"
# More sequencial expressions here ...
end
# This can be done in parallel with the "celebrity_tagging"
sequence, but is going before the celebrity_tagging sequence.
# The responses to the engine from this participant have to
"wait in line" behind the launch items clogging the queue.
participant(:image_status, :status => :inprocess)
end
# More sequencial expressions here ...
forget { participant(:image_status, :status => :complete) }
end
end
end
(The Sqs listener, just poles a single work engine queue. All of the
participants are Sqs participants.)
Any idea how I can configure the workflow or the listeners so that
there is more concurrency in the workflow or in the engine? I'm open
to architecture or system component changes as well (e.g. different
listener or participant types that offer better performance).
Hope this is clear.
Thanks for your help.
John Mettraux
>
> I'm running ruote 0.9.19. My engine is a CachedDbPersistedEngine
> configured with an Sqs Listener and Sqs Participants. My work
> typically comes in large bursts (like 20,000 long-running processes
> involving human interaction). When I launch a large batch, two things
> cause problems:
>
> 1). There is quite a long period where the engine queue is backed up
> with the launch items
> 2). Even though I have some concurrent processes at the beginning,
> they seem to be running in sequence
Hello Matt,
I've noticed you have sequences in your concurrence. What prompts you
to say "they seem to be running in sequence", does nothing happen in a
sequence while the "first" sequence is not completely over ?
Ruote has no magic in it, there is only one thing running at one time,
even more since it's Ruby. The engine piles work on its queue and only
one apply / reply (launch) is handled at a time.
Maybe I'd suggest trying with file persistence with Marshal (not
YAML). It's the fastest. The problem with it is when changing Ruby
version, the Marshal format changes. But Ruote 0.9.x has a tool for
switching persistence (use with care anyway) :
http://openwferu.rubyforge.org/persistence.html#pooltool
But I guess you'd be starting with brand new process instances.
> These issues combined are keeping the human interaction processes from
> being initiated quickly, extending mean time to completion.
OK, makes sense. By how much time are processes extended at the worse load ?
What about some load balancing ? You could run 2+ engines (with
distinct storages) and balance launchitems among them. The trick
afterward would be to distinguish workitems.
You can start an engine by passing a :engine_name parameter. By
default, it's set to "engine". With different values, you could easily
let what stands behind your SQS participants reply to different listen
queues. Each distinct engine (with distinct engine_name values) would
listen to a different [reply] queue.
This disadvantage with this is when you want to query the engine for
info about running processes, you now have N reports to aggregate.
This is one suggestion, thanks for using ruote, best regards,
--
John Mettraux - http://jmettraux.wordpress.com