(F)AQ from an outsider
Kenneth Kalmer
Unlike Yaro, I'm here to learn more about ruote, and its eco-system,
so please bear with my newbie questions and scenarios.
I'm a senior Ruby developer for a small Ruby shop in South Africa,
thats taking on bigger and bigger projects. Part of my current
research is to identify commonalities in our current projects and
consolidate them into our own internal gems that can be shared and
easily maintained across the projects. Reasons for this should be
self-explanatory, so I'm onto two common scenarios we're seeing across
the projects.
1) Rails -> Queue Service -> Background Process
2) Background Process -> DB/REST -> Rails
Phew, that was hectic. To very briefly sum it up as follows:
1) Rails -> Queue Service -> Background Process
This is not unique to us, so hopefully this is useful to someone else
too. We have a process where the user takes certain actions in a Rails
app that starts to set of a fortunate series of events. We constantly
find our self writing "intelligent" queue clients (that sit on top of
SQS or a ActiveRecord queue). The main problem we have here is that
the queue clients become so intelligent that they cannot be removed
from the project as a common library, but they're not part of the
project since they only abstract whatever implementation we use. The
code is neither here or there.
The processes are not business processes per se, but still have to be
triggered by certain events and then delegated to backend processes
for "running". Our current queue implementations work, but we end up
writing several different daemons for processing the work, and the
skeleton code of the daemons are almost identical apart from the
require's, pidfiles and output/work directories. In the end they call
a class that runs our actual code. Rough explanation, sure you get the
just of it.
If I understand ruote correctly, I can implement a single daemon that
runs a specific engine (based on parameters) and spawn the daemon as
many times as I need. I can then trigger the workflow from inside
Rails and have the engine pick it up. That engine, can if needs be,
pass it on to another and so forth, depending on the complexity of the
process.
2) Background Process -> DB/REST -> Rails
Almost the invert, but not really. Its more well defined background
processes that feed Rails (or its datastore) with data. As an example,
we're working on several internal projects that mash up external data
sources into an easier experience for the user. One live example is
www.loadwatch.co.za, that aggregates various government and parastatal
website information, as well as news feeds, together into a single
site for use for by South Africans. We're working on similar projects
that focus on other fields where information is not readily accessible
without complicated Google terms. This is kinda a hot topic for us,
and we're excited about the prospects in this field.
In this scenario I'm almost thinking having multiple crawlers feeding
a central engine, which in turn delegates the information off to the
specific applications for presentation by users. The central engine
would do all the language processing, filtering, stemming,
translations,etc, before deciding where to send it to for further
mashing and end-user consumption.
-- This line left blank intentionally --
So its processes, they're well defined, and work. The code that runs
them have gone through so many permutations I don't even want to think
about it, and we still haven't reached "process nirvana". I'm hoping a
foray into ruote will prove promising for these and other issues we're
facing as a young dev team. Given the number of actors (and perceived
ease of adding new ones) we should be able to cover all our existing
angles (SQS, ActiveRecord, DRb, REST) as well as some neat
refactoring. Scaling and distribution should also be a breeze due to
the ability to make an "engine housing" that can just be spawned
wherever to help take on some of the work.
Sidenote, we're using the rufus-scheduler in several of our projects
with great success, thanks for the hard work on that baby.
Looking forward to an interesting conversation on the topic, and
please pardon me if I mess up the terminology for certain things. I'm
still digesting most of the lacking documentation/examples on the site
:)
Chow
--
Kenneth Kalmer
kenneth...@gmail.com
Folding@home stats
http://fah-web.stanford.edu/cgi-bin/main.py?qtype=userpage&username=kenneth%2Ekalmer
John Mettraux
>
> So its processes, they're well defined, and work. The code that runs
> them have gone through so many permutations I don't even want to think
> about it, and we still haven't reached "process nirvana". I'm hoping a
> foray into ruote will prove promising for these and other issues we're
> facing as a young dev team. Given the number of actors (and perceived
> ease of adding new ones) we should be able to cover all our existing
> angles (SQS, ActiveRecord, DRb, REST) as well as some neat
> refactoring. Scaling and distribution should also be a breeze due to
> the ability to make an "engine housing" that can just be spawned
> wherever to help take on some of the work.
Hello Kenneth,
I hope I got most of your [real world / front line] scenarii, looks
like you're having challenges and fun.
Maybe a first rule of thumb would be to avoid Ruote if the business
process is transient/instantaneous, Ruote is a long-running capable
interpreter somehow. So for business processes that can be reduced to
"def bp (a, b) a + b; end" it's overkill.
(BTW, thanks for using the new nickname "ruote", so much easier to type).
(Warning, long and boring prose ahead, not sure if I reply to any of the [F]AQ)
It seems that the queue(s) are central/common to your architectures.
So a ruote engine would have to listen to queues / post to them.
You also mention your idea of having a queue client that spawns
engines for "processing" of certain messages. Would be an interesting
piece of software (and not only limited to Ruote I guess).
The Ruote engine generally "listens" to two kind of "messages" :
launchitems and workitems. Launchitems are request for launching an
instance of a process definition (running a program), they come with a
payload (aka attributes aka fields). Workitems (generally) were
emitted out of the engine before coming back with their payload
modified. Workitems are tokens moving inside the execution tree and
they sometimes "exit" the engine to be handled by a participant.
Participants are expecting two kind of messages from a ruote engine :
workitems and cancelitems. Workitems got explained (a bit) in the
previous paragraph. Cancelitems are emitted by an engine when the
branch of a process instance gets cancelled (maybe the whole process)
and that branch contains a participant that emitted a workitem, the
real participant has to be notified (it potentially has to stop its
work / discard the workitem).
That's for the basics.
A Ruote engine provides you a language for passing around hashes
(workitem payloads) and patiently wait for replies (you can define
timeouts).
Somehow I guess, the question is, do you really need Ruote ? You've
already got a loosely coupled architecture based on queues and it
works. Maybe you'd like to benefit from having "process definitions"
instead of "the code is neither here or there".
The way I see your architecture, you could have a unique
[implementation of] a queue consumer with a rule table, which would
trigger a piece of code depending on the content of the message being
consumed. Some pieces of code would push messages back on the queue.
(Erlangish flavour). Each rule + consequence could be regarded as a
method signature + method body in an open interpreter.
The disadvantage would be that all the business processes are
intertwined together (a set of top level methods called via a message
queue). Ruote could help having a focus on a [business] process A vs
[business] process B, via process definitions.
Could an organization have a mix of the All Message Queue interpreter
with a bit of Ruote BP execution ? Why not. That would deserve writing
down a set of rule of thumbs / rule of engagement / policy.
(I'm really not replying to your questions, just barely establishing a
parallel with the reflection you detailed).
One sure thing : Ruote should be easy to try and to discard if not
necessary or simply not fitting an architect / team / organization
mindset. No worries.
> Sidenote, we're using the rufus-scheduler in several of our projects
> with great success, thanks for the hard work on that baby.
Oh thanks :)
> Looking forward to an interesting conversation on the topic, and
> please pardon me if I mess up the terminology for certain things. I'm
> still digesting most of the lacking documentation/examples on the site
> :)
Sorry for the prose, not even a decent answer, and for borrowing your
terminology. Your questions, feedback, critique, reports and patches
are welcome. Thanks for your interest and for sharing details of your
architecture. Looking forward to this conversation.
Ciao,
--
John Mettraux - http://jmettraux.wordpress.com
Kenneth Kalmer
I never had a chance to thank you for the lengthy reply back in May,
and I have some more questions with a better defined example in this
case. For the sake of clarity I've truncated this entire email,
readers can go back to the old messages at
http://groups.google.com/group/openwferu-users/browse_thread/thread/345490e4bb9aad22/a4130f0956e267b4
Since we last spoke we've been using XMPP with great success in our
background processes, we also continue to use queuing services with as
much success. However things have started to become clearer to me in
terms of separating the business processes from the underlying code
implementations. It's become more clear that we need something like
ruote for one specific project, where XMPP already plays a massive
role in the underlying infrastructure (but thats not key to the
discussion).
Some background first. The system runs the biggest wholesale ISP in
South Africa. We have various levels of users and provision services
from various providers. We've identified workflow management as the
key to making the current iteration a success. With workflows used
loosely, I mean something more than just acts_as_state_machine. We
already have well defined states in place, since they're also key for
us. States are current used in two ways, the first to affect service
availability, and the second to indicate it's step in the workflow.
The latter bothers me still, feels like we're bending AASM to do
something it's not really meant to. But first a sample workflow.
Who is involved?
* Staff (of our clients)
* Client
* Support (our own staff, various tiers and areas of responsibility)
* Bots (XMPP consumers that perform said actions)
Process of provisioning generic automated service:
1. Staff provisions service
2. Client approves provisioning based on whatever criteria (internal
to their business, we don't care), might be auto-approved based on
Staff member
3. Automation: XMPP blasts out to specific bot who performs actions
and reports back
4. Support member reviews results from the bot, intervenes manually if
required (upstream API failed, servers down, whatever reason)
Process of provisioning manual service
1 & 2 as above
3. Support member validates documentation
4. Documentation might flow up and down between Support and Client
(would loop unless all requirements as met, or would be cancelled)
5. Process applied for with third party, waiting for third-part
providers (might take several days)
6. Process finishes (successfully or failed)
7. Communicated to client the result of above
8. Client might need to confirm receiving equipment or testing
functionality before we close the worklow, might jump back into
another part of the service if something is not satisfactory.
Process of transferring services from our competitors (yes it happens often :) )
1 & 2 as per first example
3. Automation will happen, but services would on hold
4. Documentation and technical conversations up and down (think domain
transfer here with EPP codes, etc...)
5. Process completes or fails based on whatever reason
(non-technological reason)
6. Service provisioned if successful
Key to how we build our systems is using several independent process
scattered over a couple of servers. All Ruby at this stage, but as I
pointed out in my blog series, any component can ultimately be
replaced by another piece of technology if needed (not foreseen, but
future proofed). The core is a single Rails application from where
humans would interact with all the processes involved.
What I can't seem to figure out (be it a mental block or whatever
other reason), is how to use ruote to manage these processes. Would
ruote take over the communication between the processes, or do we
stick to XMPP/AMQP and have WorkItems in the message paylods? Are the
process definitions kept in a single repo and shared among each
utility involved in the process? Are the actors actually related to
how we pass WorkItems around (ie, ActiveRecord, or XMPP, or AMQP/SQS)?
I've been reading the other newbie mails as well, but seems everyone
has solved that part of the problem for themselves and moved on.
I guess what I'm asking for is the names of classes that would by
inserted in the various point above, considering it is a vastly
distributed system with independent components running all over the
show.
Thanks a lot, I know its a mouthful.
--
Kenneth Kalmer
kenneth...@gmail.com
http://opensourcery.co.za
John Mettraux
>
> What I can't seem to figure out (be it a mental block or whatever
> other reason), is how to use ruote to manage these processes. Would
> ruote take over the communication between the processes, or do we
> stick to XMPP/AMQP and have WorkItems in the message paylods? Are the
> process definitions kept in a single repo and shared among each
> utility involved in the process? Are the actors actually related to
> how we pass WorkItems around (ie, ActiveRecord, or XMPP, or AMQP/SQS)?
> I've been reading the other newbie mails as well, but seems everyone
> has solved that part of the problem for themselves and moved on.
>
> I guess what I'm asking for is the names of classes that would by
> inserted in the various point above, considering it is a vastly
> distributed system with independent components running all over the
> show.
Hi Kenneth,
at first, a few remarks about aasm vs ruote.
act_as_a_state_machine (aasm) is great for tracking the state of
resources, and I feel sometimes like proposing ruote for doing the
coordination of resource states (higher level).
You could have a virtual resource named Process and describe its
states and transitions in aasm-speak. I'm unfortunately not an aasm
user, I've just studied it a bit in order to know when to apply it and
especially to know what developers are leveraging it for.
One particular thing I don't know about aasm is how you do the
management of (states, transitions) sets with it. Ruote being an
interpreter for multiple processes is OK with having various versions
of a process definition running at the same time.
(I have an impression : ruote == stored proc && aasm == triggers, but
this is vague and probably inadequate)
Now you have an architecture where XMPP and Queues play an important
role and that feels sound (btw, thanks for open sourcing your work :
http://github.com/kennethkalmer/powerdns-on-rails that's cool).
If I take a look at your first process, in ruote speak it would look like :
---8<---
class GenericProvisioning0 < OpenWFE::ProcessDefinition
sequence do
participant ref='staff' activity='provision service'
participant ref='client' activity='approve provisioning'
unless='${f:approved_by_staff}'
participant ref='automation' activity='emit order'
participant ref='support' activity='review result'
end
end
--->8---
Seems like the participant 'staff', 'client' and 'support' are
classical "human task" participants, could be covered by the
ActiveParticiant
(http://github.com/jmettraux/ruote/tree/r0.9.19/lib/openwfe/extras/participants/activeparticipants.rb)
The participant 'automation' would require some work... Maybe something like :
---8<---
require 'yaml'
require 'jabber-simple' # gem ?
module OpenWFE
module Extras
class JabberParticipant
include OpenWFE::LocalParticipant
def initialize (options={})
@block = block
@im = Jabber::Simple.new(options[:account], options[:password])
@im.received_messages do |msg|
workitem = decode_workitem(msg)
reply_to_engine(workitem)
end
end
def consume (workitem)
target = workitem.attributes['target']
@im.deliver(target, encode_workitem(workitem))
end
protected
def decode_workitem (msg)
YAML.load(msg.body)
end
def encode_workitem (wi)
YAML.dump(wi)
end
end
end
end
--->8---
Then you'd have to bind that participant in the engine :
---8<---
engine.register_participant(
"automation",
OpenWFE::Extras::JabberParticipant.new(
:account => "autom...@jabber.dns.example.co.za", :password =>
"joburg222"))
--->8---
That would be one way of doing it. Note that the participant code
contains the dispatching and the receiving implementation.
That's just for XMPP (well Jabber), I guess it's easy to derive code
for other asynchronous or synchronous mechanisms.
In the end, you could use Ruote to interpret your high-level process
definitions and "orchestrate" your services/agents/participants. Could
be interesting to separate tactical stuff (business processes) from
technical stuff (local states and triggers).
Just a bunch of suggestions. Your remarks and critiques are welcome.
Best regards,
John Mettraux
>
> ---8<---
> class JabberParticipant
> include OpenWFE::LocalParticipant
>
> def initialize (options={})
>
>
> @im = Jabber::Simple.new(options[:account], options[:password])
>
> @im.received_messages do |msg|
> workitem = decode_workitem(msg)
> reply_to_engine(workitem)
> end
> end
> --->8---
oops, the part about the block has to get commented out... Crumb from
the initial draft...
Kenneth Kalmer
>
> On Sat, Nov 8, 2008 at 5:56 PM, Kenneth Kalmer <kenneth...@gmail.com> wrote:
>>
>> What I can't seem to figure out (be it a mental block or whatever
>> other reason), is how to use ruote to manage these processes. Would
>> ruote take over the communication between the processes, or do we
>> stick to XMPP/AMQP and have WorkItems in the message paylods? Are the
>> process definitions kept in a single repo and shared among each
>> utility involved in the process? Are the actors actually related to
>> how we pass WorkItems around (ie, ActiveRecord, or XMPP, or AMQP/SQS)?
>> I've been reading the other newbie mails as well, but seems everyone
>> has solved that part of the problem for themselves and moved on.
>>
>> I guess what I'm asking for is the names of classes that would by
>> inserted in the various point above, considering it is a vastly
>> distributed system with independent components running all over the
>> show.
>
> Hi Kenneth,
>
> at first, a few remarks about aasm vs ruote.
>
> act_as_a_state_machine (aasm) is great for tracking the state of
> resources, and I feel sometimes like proposing ruote for doing the
> coordination of resource states (higher level).
Makes sense, exactly what I had in mind.
> You could have a virtual resource named Process and describe its
> states and transitions in aasm-speak. I'm unfortunately not an aasm
> user, I've just studied it a bit in order to know when to apply it and
> especially to know what developers are leveraging it for.
>
> One particular thing I don't know about aasm is how you do the
> management of (states, transitions) sets with it. Ruote being an
> interpreter for multiple processes is OK with having various versions
> of a process definition running at the same time.
>
> (I have an impression : ruote == stored proc && aasm == triggers, but
> this is vague and probably inadequate)
AASM transitions are handled via events define with the class/model.
Quick example would be:
class GenericServer < ActiveRecord::Base
acts_as_state_machine :initial => :pending
state :pending, :enter => :trigger_automation
state :active, :enter => :send_welcome_mail
event :activate, :from => :pending, :to => :active
def trigger_automation
end
def send_welcome_mail
end
end
Then you'll simply call the banged event to transition from 'pending'
to 'active', ie. GenericService.new.activate! Callbacks are defined
when states are entered and left.
I think I have an idea what you mean with the route = stored procs &&
assm = triggers idea, but I'll play with it a bit more...
> Now you have an architecture where XMPP and Queues play an important
> role and that feels sound (btw, thanks for open sourcing your work :
> http://github.com/kennethkalmer/powerdns-on-rails that's cool).
Likewise :)
Orchestration is exactly what I was looking for ultimately. Finite
state machines are wonderful when used in a small setting, but 100's
of states (albeit finite), is not worth the headache. I've seen us get
caught in the trap of using states to define process steps, and then
we quickly wipe the board clean and start again. By this time you can
imagine the board begin spotless, hence looking outside the AASM box.
> Just a bunch of suggestions. Your remarks and critiques are welcome.
> Best regards,
As always John, you're kindness and patience towards newbies is
nothing short of inspirational. I'll help people out with my own
projects in a similar way. Thanks for the help, I'll get cracking on a
fictional workflow to see if I can wrap my head around this and will
pass feedback along.
Regards
Kenneth Kalmer
<kenneth...@gmail.com> wrote:
> On Sun, Nov 9, 2008 at 7:59 AM, John Mettraux <jmet...@openwfe.org> wrote:
>>
>> On Sat, Nov 8, 2008 at 5:56 PM, Kenneth Kalmer <kenneth...@gmail.com> wrote:
>>>
>>> What I can't seem to figure out (be it a mental block or whatever
>>> other reason), is how to use ruote to manage these processes. Would
>>> ruote take over the communication between the processes, or do we
>>> stick to XMPP/AMQP and have WorkItems in the message paylods? Are the
>>> process definitions kept in a single repo and shared among each
>>> utility involved in the process? Are the actors actually related to
>>> how we pass WorkItems around (ie, ActiveRecord, or XMPP, or AMQP/SQS)?
>>> I've been reading the other newbie mails as well, but seems everyone
>>> has solved that part of the problem for themselves and moved on.
>>>
>>> I guess what I'm asking for is the names of classes that would by
>>> inserted in the various point above, considering it is a vastly
>>> distributed system with independent components running all over the
>>> show.
OK, so I've figured most of this out spending a couple of hours inside
route-web and following the Tea Testing Team example. I think the
barrier initially was understanding where Rails stops, and ruote
begins. I'll spend some time on route-web2 and route-rest, just to get
more up to date and make sure I can navigate around the concepts.
I'll basically be persisting the engine and workitems in the database
with DbPersistedEngine & ActiveParticipant, and will need to build a
custom Jabber participant just like you said (more on this below).
>> Now you have an architecture where XMPP and Queues play an important
>> role and that feels sound...
Yes, and no. What confused the living daylights out of me here was
this: Where does the engine run? Inside each participant? Only a
couple of minutes ago did I put this part of the puzzle together
(reading http://jmettraux.wordpress.com/2007/03/13/openwferu-over-amazon-sqs/).
In my case I would have the engine running in each instance of Rails
(no problem), and I would need the instance running inside a custom
Jabber bot (which would act the same way as SqsListener). Trying to
hook a background Jabber thread in Rails is going to be suicide, but I
can see how anyone would try and put a background thread in for
polling SQS.
Then just to make sure I've gotten my head around the separation of
the processes where the engine lives, and the participants live, I
could go as far running a copy of the engine inside every Jabber
participant. This would not be advisible since it would leave the DB
with legs wide open, but just to make sure I've got the concept right.
(What do I fork so I cant get this participant baked right in?)
>> In the end, you could use Ruote to interpret your high-level process
>> definitions and "orchestrate" your services/agents/participants. Could
>> be interesting to separate tactical stuff (business processes) from
>> technical stuff (local states and triggers).
>
> Orchestration is exactly what I was looking for ultimately. Finite
> state machines are wonderful when used in a small setting, but 100's
> of states (albeit finite), is not worth the headache. I've seen us get
> caught in the trap of using states to define process steps, and then
> we quickly wipe the board clean and start again. By this time you can
> imagine the board begin spotless, hence looking outside the AASM box.
Yip, this is still pretty much be the case still. Anything we
provision would be in a "pending" state by default, and remain in this
state as it traverses the business processes involved (no matter how
many steps or the duration of said process). The marriage between AASM
and ruote is a beautiful one once the lines have been drawn and the
responsibilities of each clearly defined. Only thing to note is that
some things that would have traditionally lived inside AASM
transitions would now become participants in the ruote process (like
email notifications).
I'll be building some actual code from all my messy notes today, will
shoot with more questions as I progress.
Best
John Mettraux
<kenneth...@gmail.com> wrote:
>
> Then just to make sure I've gotten my head around the separation of
> the processes where the engine lives, and the participants live, I
> could go as far running a copy of the engine inside every Jabber
> participant. This would not be advisible since it would leave the DB
> with legs wide open, but just to make sure I've got the concept right.
Hi Kenneth,
there is also the model where the engine lives outside of the "front
applications" and is queried over HTTP (ruote-rest or ruote-web2).
Participants could then live in/within this backoffice engine. Many
possibilities.
Kenneth Kalmer
And with that you answered the next question. Was thinking about the
possibility using route-rest to keep our main application "clean", and
since we're already connecting to various specialized internal REST
services it would be a natural fit.
Thanks again for your time and patience.
John Mettraux
<kenneth...@gmail.com> wrote:
>
> And with that you answered the next question. Was thinking about the
> possibility using route-rest to keep our main application "clean", and
> since we're already connecting to various specialized internal REST
> services it would be a natural fit.
Hi Kenneth,
thanks for the patch, it's in
(http://github.com/jmettraux/ruote-rest/commit/e12a01f4b04eafd09fc045e5a2e5e184eefd9dca)
Kenneth Kalmer
>
> On Mon, Nov 24, 2008 at 2:37 PM, Kenneth Kalmer
> <kenneth...@gmail.com> wrote:
>>
>> And with that you answered the next question. Was thinking about the
>> possibility using route-rest to keep our main application "clean", and
>> since we're already connecting to various specialized internal REST
>> services it would be a natural fit.
>
> (http://github.com/jmettraux/ruote-rest/commit/e12a01f4b04eafd09fc045e5a2e5e184eefd9dca)
Pleasure, will keep them coming in the following weeks :)
I'm no longer an outsider it seems, but I'll keep the thread running
as a resource for others in the future, so on to the next set of
questions (with some background to my current progress).
ruote-rest is working great, and I'm currently writing a thin HTTParty
wrapper to launch processes and manipulate work items. All
participants at this stage of the match is still the default
ActiveParticipant setup by ruote-rest. I'll get to the XMPP
participant at the office.
Now since we're catering for a large number of users, couple hundred
direct and several thousand indirect, and most of them would somehow
be affected by the workflows, I'm wondering how to best use
participants and search for work items. Currently I'm toying with
having participants that match the roles in our system, but loading
all the work items and filtering based on a specific attribute seems
overkill. A thought that krept into my head over breakfast was pass
our process definitions through ERB and dynamically build the
participant names each time we launch a process.
Is this sane? Or is there a better way to let ruote-rest/ruote search
for work items matching a specific participant AND attributes. I've
found the RDOC's and will continue checking out the
OpenWFE::Extras::WorkItem code at the office.
John Mettraux
<kenneth...@gmail.com> wrote:
>
> ruote-rest is working great, and I'm currently writing a thin HTTParty
> wrapper to launch processes and manipulate work items. All
> participants at this stage of the match is still the default
> ActiveParticipant setup by ruote-rest. I'll get to the XMPP
> participant at the office.
>
> Now since we're catering for a large number of users, couple hundred
> direct and several thousand indirect, and most of them would somehow
> be affected by the workflows, I'm wondering how to best use
> participants and search for work items.
Neat, I'm looking forward to your feedback.
> Currently I'm toying with
> having participants that match the roles in our system, but loading
> all the work items and filtering based on a specific attribute seems
> overkill. A thought that krept into my head over breakfast was pass
> our process definitions through ERB and dynamically build the
> participant names each time we launch a process.
>
> Is this sane? Or is there a better way to let ruote-rest/ruote search
> for work items matching a specific participant AND attributes. I've
> found the RDOC's and will continue checking out the
> OpenWFE::Extras::WorkItem code at the office.
The ERB idea is great. You usually launch a process by feeding an
initial payload (launchitem) containing the process definition or
pointing to the process definition. The engine doesn't care if you
point to a static document or a generated one, so it's OK to generate
process definitions on the fly, just point to the right URL.
There is some code in OpenWFE::Extras::WorkItem for "searching"
workitems, maybe it will be sufficient for you, you could even go with
participant_name / store_name only.
Cheers,