Question on engine dying and coming up again
Kenneth Kalmer
Since my ruote presentation at the Ruby on Beer, Andrew Timberlake (now a list member) has been investigating ruote quite extensively and has started playing with ruote-rest. We've spent the morning in IRC, and most of last night at Ruby on Beer going through some ruote(-rest) topics and exchanging ideas.
He brought up an excellent point in #ruote just a couple of minutes ago.
Given the scenario in this gist -> http://gist.github.com/107469
If the engine is killed while sleepy is sleeping, and the engine is restarted, the expression just sits idle. How would we get the engine to 'retry' the participant?
I haven't thought of this, since the engine "Doesn't Go Down (tm)", but Andrew makes a good point in that the entegrity of the process is at stake...
Best
--
Kenneth Kalmer
kenneth...@gmail.com
http://opensourcery.co.za
@kennethkalmer
John Mettraux
>
> Given the scenario in this gist -> http://gist.github.com/107469
>
> If the engine is killed while sleepy is sleeping, and the engine is
> restarted, the expression just sits idle. How would we get the engine to
> 'retry' the participant?
>
> I haven't thought of this, since the engine "Doesn't Go Down (tm)", but
> Andrew makes a good point in that the entegrity of the process is at
> stake...
Hi Kenneth and Andrew,
excellent point.
Maybe at first some context : http://openwferu.rubyforge.org/part.html
There are two kind of participants, local and remote. The distinction
is not really intra- / extra- runtime but this one : local participant
respond to the engine by themselves, while remote participants fire
the workitem to some other system which will reply to the engine.
I have to say I focused until now on participants that either handled
the workitem very quickly and replied, either, the majority of cases,
where the participants hand the workitem to a worklist of some sort.
Two those cases have a vulnerability window narrow as that of
classical apps.
Now this sleep, symbolizing a long running operation, opens a wide
vulnerability window.
1)
you can cancel the stalled expression. It will be as if the engine had
never seen it, the process instance will immediately resume.
2)
you can re-apply the expression. This isn't yet packaged in ruote-rest
and ruote-web2 (nor ruote-kit). Programmatically, it looks like :
---8<---
expression = engine.process_status(wfid).expressions.last # assuming
the participant expression is the last one
wi = expression.applied_workitem
wi.attributes['__reapplied__'] = true # as discussed on IRC
engine.apply(wi)
--->8---
my tasks :
http://rubyforge.org/tracker/index.php?func=detail&aid=25799&group_id=2609&atid=10195
http://rubyforge.org/tracker/index.php?func=detail&aid=25800&group_id=2609&atid=10195
3)
I will explore the "re-apply on restart" idea discussed on IRC
(transcript attached)
my task :
http://rubyforge.org/tracker/index.php?func=detail&aid=25801&group_id=2609&atid=10195
Thanks a lot, excellent discussion,
--
John Mettraux - http://jmettraux.wordpress.com
-.-.-.-.-.-.-.-.-.-
IRC transcript : (#ruote on freenode.net)
20:35 kennethkalmer: john, meet Andrew Timberlake
20:35 kennethkalmer: Andrew, meet John Mettraux
20:35 jmettraux: hi Andrew !
20:36 kennethkalmer: Andrew is part of our Ruby on Beer brigade and is
learning ruote
20:36 kennethkalmer: (and possibly afk)
20:36 jmettraux: aah, nice, I saw you tweeted about ruote and beer @gauteng
20:36 kennethkalmer: i'm blasting off a mail to the list with a
question andrew brought up
20:37 kennethkalmer: mail sent
20:40 andrewtimberlake: Hi John, great project you've got here. I
believe its going to make a project I'm working on much simpler to
control.
20:40 jmettraux: andrewtimberlake: Hi, thanks
20:41 jmettraux: about ruote and sleepy : when a SleepExpression is
applied, is registers a schedule (in the engine's rufus-scheduler) to
get triggered after/at the given time
20:43 jmettraux: sleepy ruote : when the engine restarts, the
engine.reload() method is called (ruote-rest and ruote-web2), this
method checks quickly every expression that owns a schedule and
recreates that schedule
20:43 jmettraux: sleepy ruote : if at reschedule/reload/restart a
"due" schedule is encountered, it is triggered immediately
20:45 andrewtimberlake: My 'sleepy' example is not so much about
sleeping as about a participant who could be killed if the server
crashes (someone trips over the power cord) before returning to the
engine. When ruote-rest is restarted. The process remains at that
participant indefinitely
20:46 andrewtimberlake: I don't want to begin working with ruote if it
is possible that processes will be left idle after a restart
20:46 jmettraux: andrew: ok, that's a possibility. You can fetch the
expression (participant expression) and reply to it with the
"appliedWorkitem" to resume the process instance
20:47 jmettraux: andrew: I will "package" that action, just in case
21:08 jmettraux preparing lengthy reply on ruote ml
21:41 andrewtimberlake: jmettraux: "You can fetch the expression
(participant expression) and reply to it with the "appliedWorkitem" to
resume the process instance" <- How do I do this?
21:41 jmettraux: kennethkalmer: I'm wondering if we should track
"should re-apply on restart" in the participant map for each
participant
21:42 jmettraux: exp = engine.process_status(wfid).expressions.last
21:42 kennethkalmer: that sounds like a better plan
21:42 jmettraux: exp.reply(exp.applied_workitem)
21:43 jmettraux: kennethkalmer: the actual plan is to let the
"administrator" re-ignite stalled process instances
21:43 andrewtimberlake: How do I do that over REST?
21:43 jmettraux: kennethkalmer: by default, participant would get
stuck (current behaviour)
21:44 jmettraux: andrewtimberlake: over REST, you cannot for now
21:44 andrewtimberlake: OK
21:44 jmettraux: andrewtimberlake: have to implement that
21:44 kennethkalmer: ruote-kit is starting this weekend, period
21:44 jmettraux: :)
21:44 kennethkalmer: really need to get to grip with the internals of
the engine :)
21:45 jmettraux: you guys have a good grasp of it, judging from the question ;)
21:45 kennethkalmer: andrewtimberlake cornered me a couple of times
this morning :)
21:46 kennethkalmer: guys it's my learning "from the outside in" that
differs to his "inside out" approach with ruote
21:46 jmettraux: great team
21:46 kennethkalmer: and great bear ;)
21:46 kennethkalmer: arg
21:46 kennethkalmer: beer
21:46 jmettraux: there is actually a way to re-ignite
21:47 jmettraux: cancelling the expression
21:47 jmettraux: it's equivalent to the manoeuver I just described
21:47 jmettraux: DELETE /expressions/xyz/0.1.2
21:47 jmettraux: what's not implemented is re-apply
21:48 jmettraux: you currently can only re-apply [replay] at errors
21:51 jmettraux: so the participant-map idea would be that, at
participant registration time, we can flag participant as "re-apply on
reload"
21:52 kennethkalmer: i really think that is a sane setting
21:53 jmettraux: the engine.reload would enumerate participant
expressions, check if they are applied and linked to a participant
that is "re-apply on reload" and do the reapply if yes
21:53 kennethkalmer: participants imho should be small and 'single
tasked' so in theory it should work
21:53 jmettraux: yes, our classical participant just hands over the
workitem to a "real participant"
21:54 kennethkalmer: exactly
21:54 jmettraux: and don't reply by themselves to the engine
21:54 jmettraux: this participant map setting is for the next version then
21:55 jmettraux: the re-apply packaging via REST is a must for now
21:55 jmettraux: wdyt ?
21:55 kennethkalmer: yip, adding it the kit rambings
21:55 andrewtimberlake: What about a method like do_not_thread so each
participant can identify itself as re-applyable
21:55 jmettraux: I have to add to rr and rw2
21:55 jmettraux: good idea
21:56 kennethkalmer: also asked andrew to also make a list of his
experiences and annoyances as he goes along so we can address them
21:56 jmettraux: it will be a long one *)
21:56 kennethkalmer: but needed :)
21:56 andrewtimberlake: A participant should probably also be notified
that it is being retried so it can perform necessary checks to not
duplicate work
21:57 kennethkalmer: i second that
21:57 jmettraux: well, if it's that smart, it can recognize behind
handed the workitem for a second time
21:57 jmettraux: the fei will be identical
21:57 andrewtimberlake: Good point
21:59 kennethkalmer: would that mean that the participant would
somehow have to maintain state?
21:59 jmettraux: I think that yes
21:59 jmettraux: not all of them
21:59 kennethkalmer: as a stupid example, the jabber "send and forget"
style usage
22:00 jmettraux: the classical db backed participant will override an
old workitem with the new workitem with the same fei
22:00 kennethkalmer: it has no way to keep state, unless it can access
its own chat logs somehow
22:01 jmettraux: the engine should not know anything about the detail
of the participant
22:01 kennethkalmer: agreed
22:01 kennethkalmer: but it's the participant's right to know it's
being re-applied
22:01 jmettraux: it's up to the participant impl to deal with the
particularities of the medium used if any
22:01 jmettraux: true
22:01 kennethkalmer: for the engine it's still decoupled
22:02 kennethkalmer: for the participant it's optional, but extremely useful
22:03 jmettraux pondering how to do this notification the right way
22:04 kennethkalmer: jmettraux: how about a __reapplied__ attribute in
the workitem?
22:04 kennethkalmer: ruote already uses __timed_out__ and some others,
it would keep with the conventions
22:04 jmettraux: kennethkalmer: good one
22:05 kennethkalmer: instead of a second arg to #consume
22:11 andrewtimberlake: kennethkalmer: I'm having a look at your
presentation again and you use joe :activity => 'Review Quotes'. All
the examples I've seed are participant :ref => :xyz. What's the
difference?
22:12 jmettraux: the engine looks up for expression names first among
subprocesses and participants
22:12 andrewtimberlake: How do I do the former, and where does
:activity => 'Review Quotes' go?
22:12 jmettraux: so you can write xyz instead of participant :ref => :xyz
22:12 andrewtimberlake: K, and the :activity, is that put in the attributes?
22:13 jmettraux: but if there is a subprocess named xyz, it will have
priority over the participant named xyz
22:13 jmettraux: yes, the :activity is an attribute
22:13 jmettraux: it's more a convention
22:14 jmettraux: the engine will add a :params => { :activity =>
'review quotes' } field to the workitem emitted to joe
22:14 andrewtimberlake: Thanks
22:14 jmettraux: you're welcome
22:14 andrewtimberlake: PS. I really like the graph output, it makes
it really easy to see if I'm writing my process definitions correctly
22:15 jmettraux: feedback is welcome on that, some expressions have no
proper representation yet
22:15 jmettraux: are you using ruote-fluo ?
22:15 jmettraux: http://difference.openwfe.org:4567/?pdef=pdef_ttt.xml ?
22:17 andrewtimberlake: I'm just looking at the representation in
ruote-rest when I load up an process
22:18 jmettraux: ah ok, the link shows an ongoing effort at some web
based process editor, you can edit the process in the left pane
22:18 andrewtimberlake: I'm still working with very simple processes
while I figure everything out.
22:18 jmettraux: cool
John Mettraux
>
> 2)
>
> you can re-apply the expression. This isn't yet packaged in ruote-rest
> and ruote-web2 (nor ruote-kit). Programmatically, it looks like :
>
> ---8<---
> expression = engine.process_status(wfid).expressions.last # assuming
> the participant expression is the last one
> wi = expression.applied_workitem
> wi.attributes['__reapplied__'] = true # as discussed on IRC
> engine.apply(wi)
> --->8---
>
> my tasks :
>
> http://rubyforge.org/tracker/index.php?func=detail&aid=25799&group_id=2609&atid=10195
> http://rubyforge.org/tracker/index.php?func=detail&aid=25800&group_id=2609&atid=10195
OK,
engine.reapply() is in :
http://github.com/jmettraux/ruote/commit/8ab020f2e929d5a38819e1460f860007c6d75f43
http://github.com/jmettraux/ruote-rest/commit/a07de21b220c8ddd5e8b71064751bc4c23b58af3
> 3)
>
> I will explore the "re-apply on restart" idea discussed on IRC
> (transcript attached)
>
> my task :
>
> http://rubyforge.org/tracker/index.php?func=detail&aid=25801&group_id=2609&atid=10195
Still pondering this one, have to think about "re-apply loops".
Best regards,