Storage Participant Best Practices

[Gordon Hempton]

Hi,

I have a process that has several steps involving storage participants. These storage participants are consumed by multiple humans, essentially acting as a work queue.

Problems arise when there are lots of humans acting on these storage participants. What is the best way to manage reservation and assignment of these tasks? Two strategies that I can think of:

1) Call `participant.reserve` and assign the owner of the task whenever a human starts work on an item. In this case, I would simply query the storage participant for workitems that don't already have an owner when a human requests a new task? This is further complicated by the fact that I want to reserve several workitems at a time so that the interface can be more responsive. If the task is never completed (e.g. that owners leaves the company etc.) what is the best way to unreserve the workitem?

2) Have an assignment step in the workflow that randomly picks an owner for each workitem. The downside to this is that it would require that all humans are working at the same efficiency and some humans would exhaust their queue more quickly than others (at which point I would want them to pull from other owner's items).

Thoughts?

As always, thanks for the great project/support.

Cheers,

Gordon

[John Mettraux]

Hello Gordon,

I'm not a fan of "best practice" questions, it's hard to answer, one can't
guess anything about the context where the "practice" has to apply.

Feel free to implement your own storage participant with the reservation
routines you need.

You've listed two techniques, implement those as your reservation routines.
Make sure you can switch at runtime. "Oh, technique A is not the right one in
that can, let's switch that participant to technique B" (on the fly). Make it
so that you can add technique C (and maybe deprecate technique A) later on.

Make sure to add diagnostic/monitoring tools to your participant (or your
frontend to a vanilla ruote storage participant), so you can assess and then
switch technique and/or develop better techniques.


Sorry for playing it meta.

Best regards,

--
John Mettraux - http://lambda.io/jmettraux

[Gordon Leland Hempton]

No worries about keeping it meta. Part of this question was just to establish that what I was doing was sane and that there wasn't some obviously better way of doing things.

I feel as though reserving workitems from a "queue-like" storage participant would be an extremely common scenario and am surprised there aren't more storage participant methods to this effect (e.g. something as simple as a method to return unreserved workitems).

Cheers,

Gordon

--
--
you received this message because you are subscribed to the "ruote users" group.
to post : send email to openwfe...@googlegroups.com
to unsubscribe : send email to openwferu-use...@googlegroups.com
more options : http://groups.google.com/group/openwferu-users?hl=en
---
You received this message because you are subscribed to the Google Groups "ruote" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openwferu-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Gordon L. Hempton
http://codebrief.com

360.460.8098

[John Mettraux]

On Thu, Oct 10, 2013 at 03:01:33PM -0700, Gordon Leland Hempton wrote:
>
> No worries about keeping it meta. Part of this question was just to
> establish that what I was doing was sane and that there wasn't some
> obviously better way of doing things.
>
> I feel as though reserving workitems from a "queue-like" storage
> participant would be an extremely common scenario and am surprised there
> aren't more storage participant methods to this effect (e.g. something as
> simple as a method to return unreserved workitems).

I just went for the most common scenario (imho) "engine assignes work to
participant A". The reserve/owner thinggies came afterward. Since they got
introduced, you're the first to request such a method. I guess it would be
simple to add it.

So your best practice question boils down to "please add a
list_workitem_whose_owner_is_nil" request?

[Gordon Leland Hempton]

Forgive the poor title choice. My experience with ruote is limited and I am just looking for some directional guidance. I guess my question boils down to "What is the best way to implement a workitem queue?". An alternative to using a storage participant would be to use amqp with rabbit mq or something.

Is there a canonical use case for the reserve/owner thingy?

Thanks for your help,

Gordon

--
--
you received this message because you are subscribed to the "ruote users" group.
to post : send email to openwfe...@googlegroups.com
to unsubscribe : send email to openwferu-use...@googlegroups.com
more options : http://groups.google.com/group/openwferu-users?hl=en
---
You received this message because you are subscribed to the Google Groups "ruote" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openwferu-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[John Mettraux]

On Thu, Oct 10, 2013 at 03:19:07PM -0700, Gordon Leland Hempton wrote:
> Forgive the poor title choice. My experience with ruote is limited and I am
> just looking for some directional guidance. I guess my question boils down
> to "What is the best way to implement a workitem queue?". An alternative to
> using a storage participant would be to use amqp with rabbit mq or
> something.

Hello,

yes, that is a possibility. I've seen it used for automated participants
(rather than human participants).

Never heard of people grafting a reservation system on top of it though.

> Is there a canonical use case for the reserve/owner thingy?

It's somehow explained in:

https://github.com/jmettraux/ruote/blob/master/lib/ruote/part/storage_participant.rb#L361-L377

If field "owner" is set, then the workitem is reserved.


Best regards,

John

[Jonathan Locker]

Gentlemen,

I am involved in using Ruote as a workflow service for our systems, creating both manual and automated tasks.

When a new process is created, the workflow participants (manual or automated) create a message sent to Rabbit MQ. The appropriate system picks up the queue, and does whatever the specific participant requires (create a manual task, update certain fields).

The short of it is that Manual tasks (and their reservation-assignment) are handled by that external system. When the task is complete, it puts a message in the "done" queue, which Ruote uses to update the workflow participants, and kick of the next step in the process.

But for Gordon's purposes, this would require building some type of task assignment/completion system and using ruote as a backend service...may not be what he had in mind.

Just two cents,

Jonathan Locker

[Nando Sola]

Hi folks! Here goes our 0,02€

In our case, we've got a special "PubSubStorage" that extends "StorageParticipant". It serializes workitem to an external XMPP PubSub queue and decides wether or not to procced based on a config keyword. The user interaction with this participant comes in two flavours:

- - ^fill_form

  - Abstra::RuoteProject::Participants::PubSubStorageParticipant

  - {}

- - ^wait_for_action

  - Abstra::RuoteProject::Participants::PubSubStorageParticipant

  - {proceed_on: pick}

Once the participant sends the payload to the external queue, the user can either "pick" the waiting workitem (triggering the next workflow step, ie: filling up a forrm) or directly fill the form without the wait. The logic resides on StorageParticipant#on_workitem, which already has information on the actions that will either keep it holding on or directly proceed (as seen above). The PubSub server is configured in a way that automatically persists the queues where workitems are sent. All further communication inbetween the user and engine can be done either fully async or via HTTP (à la RuoteKit). Just implement a Command Pattern in your endpoint and retrieve the workitem from the storage. To decide the recipients of the workitems (ie. user or groups), an additional AuthorizationService can be used outside the engine itself. Just define your payload well.

The main goal is to rely on Ruote just for the essential. Keeping it as simple as possible. Let the enclosing application decide on all authorization/authentication stuff and the "task assignment logic". Moreover, any modern PubSub infrastructure (AMQQ, XMPP, JMS) has already automatic persistence in place, which can be used and abused, instead of implementing a new queue-specific storage. We've been working like this for years without further issue. Again, keep it simple :-)

Kind regards,

//nando