Workflow Project daily progress report [GSOC 2013]

[hardik juneja]

Hello Dominic --

Today i implemented the approach you suggested tomorrow.
Its working smooth.

Tomorrow, I will try to extend the functionality of the model developed till now.
This include taking create/view/search/update as inputs and generating the workflow from them.

Roadblocks : For update we need an id of the particular record that user want to update, but in our menu we will just show option to update
so, we can show them a list view first and then they can open the record to update it.

But, what if there are no record? 
Then i think we should check the db for that and with notice can move further.

One more thing, I think the list views will be stops.
How to move further once a list view is showed? how long will we wait on a list view?
I still will have to think on this.

Regards,
Hardik Juneja

[Dominic König]

tisdagen den 25 juni 2013 11.31.27 skrev hardik juneja:
> Roadblocks : For update we need an id of the particular record that user
> want to update, but in our menu we will just show option to update
> so, we can show them a list view first and then they can open the record to
> update it.

A menu item to update records doesn't make sense.

Typically, we have a table or list view of records where each entry has a
button/link/icon to edit it - /that/ would go to update.

IMHO, the menu item would be the workflow itself.

> One more thing, I think the list views will be stops.
> How to move further once a list view is showed? how long will we wait on a
> list view?

Hmm, interaction is a two-player game. If the system has completed a step
(whether it provided a form or a list doesn't really make a difference), then
it's the user's turn to initiate the next request - and that's what the system
is waiting for.

So the next step is initiated by the user clicking an action item on the page.
That could be

- a form-action (=submit, if the page contains a form)
- a record-action (e.g. read/update/delete, if the page contains records)
- a workflow action
- implicit (e.g. back/skip/cancel/continue/checkout)
- explicit (menu action)

For each of these actions, the page provides the necessary action items:
buttons, links, menus...whatever - even a list view.

As soon as the user picks an action - the workflow continues. If he does never
pick an action - then it does not continue: no request - no response. That's
how servers work.

I mean, yes - an asynchronous task could run a timer, and if the user takes
too long to respond, automatically alert his neighbour to check whether he has
died of a heart attack...but hey: that's probably beyond GSoC. For now, we
accept the user's sudden death and do nothing until they serve us a new one.

Dominic

[Dominic König]

tisdagen den 25 juni 2013 11.31.27 skrev hardik juneja:

> One more thing, I think the list views will be stops.
> How to move further once a list view is showed? how long will we wait on a
> list view?
> I still will have to think on this.

What about a popup box "DO SOMETHING!!!" accompanied with a nasty siren sound?

Dominic

[Dominic König]

I mean - it's called a "work"flow, not a "sit-and-think-"flow

:D

Yeah - ok ok, I'm going to get more serious again soon, I promise.

Dominic

[Dominic König]

tisdagen den 25 juni 2013 11.31.27 skrev hardik juneja:

> One more thing, I think the list views will be stops.
> How to move further once a list view is showed?

From my brother's handbook for interaction design:

"Stay calm and wait for someone to resuscitate the user."

So, not just my idea :D

Dominic

[hardik juneja]

Hmm, interaction is a two-player game. If the system has completed a step

(whether it provided a form or a list doesn't really make a difference), then
it's the user's turn to initiate the next request - and that's what the system
is waiting for.

So the next step is initiated by the user clicking an action item on the page.
That could be

- a form-action (=submit, if the page contains a form)
- a record-action (e.g. read/update/delete, if the page contains records)
- a workflow action
    - implicit (e.g. back/skip/cancel/continue/checkout)
    - explicit (menu action)

For each of these actions, the page provides the necessary action items:
buttons, links, menus...whatever - even a list view.

As soon as the user picks an action - the workflow continues. If he does never
pick an action - then it does not continue: no request - no response. That's
how servers work.

My point was -

If user is going to define route for the workflow.

So, lets say he chooses

1) req/req/create

2) org/org/create

3) org/org/list

4) pr/pr/create

now after completing two steps 
The third one is a list view.

So now he should be given freedom to do anything 

which include following tasks-

- a form-action (=submit, if the page contains a form)
- a record-action (e.g. read/update/delete, if the page contains records)
- a workflow action
    - implicit (e.g. back/skip/cancel/continue/checkout)
    - explicit (menu action)

and then workflow continue with the forth step

okhay :)

 

I mean, yes - an asynchronous task could run a timer, and if the user takes
too long to respond, automatically alert his neighbour to check whether he has
died of a heart attack...but hey: that's probably beyond GSoC. For now, we
accept the user's sudden death and do nothing until they serve us a new one.

Right, thats beyond GSOC ;)

[hardik juneja]

I know that was a foolish question but i wanted to say that once we mare on a list view and when user perform certain actions and if that includes say updating a record as he reaches the update view then get_vars (?workflow = ) is gone.

So there must be some way to tell the workflow request handler, that the user has not yet left the workflow and will come back. :)

--

Hardik Juneja

[Dominic König]

Correct,

a page that is rendered by a workflow must include the workflow ID in any
possible next action that is initiated /from/ this page.

One option is to store the current workflow in the session, and handle any
subsequent request in this session as step within this workflow until the user
explicitly exits the workflow.

But this isn't quite right: if the user navigates away, e.g. by choosing a
menu item outside of the workflow, or from his browser bookmarks, then you may
connect a request to the workflow that doesn't belong to it.

The better option is to store the workflow ID in the page, and have any
subsequent request /from that page/ that belongs to the workflow re-submit it -
whilst other requests which do not belong to the workflow, do not re-submit it.

For example:

If a workflow renders a datatable, then it renders all action buttons in the
datatable with the workflow ID in the action URL.

When the user then clicks on an action button for a record, then the workflow
ID is automatically re-submitted (in the request URL, i.e. get_vars).

If he instead navigates away to another page outside of the workflow, then the
action buttons on that page would not include the workflow ID, hence requests
from that page are not handled by the workflow anymore.

---

The question wasn't really "stupid" - it comes back to what you said about
where to store the "workflow information" (i.e. the identity of the workflow
instance): and in fact, the best option is to store it in the page, and that
is, in any action item on that page that belongs to the workflow.

Usually, there aren't really many different action items - for a start it seems
sufficient to modify the action buttons in datatables according to the workflow
settings (because a "list view" is actually not a stop, but a "choose record
and next action" step), and in all POST forms to add a hidden INPUT that
carries the workflow identity.

You may need additional elements for workflow navigation: "back", "cancel",
"skip", "continue", "cancel" and the like, probably as additional header or
footer elements (e.g. "buttons") - and of course (optional? configurable!) info
elements to display the workflow status (could be a header, for example) and
instructions for the user.

Furthermore, you may need menu items to return to pending workflows after
navigating away. This isn't always required, though.

---

Sorry for having made fun of it - but it was very hard to resist... :S

Dominic

[Dominic König]

tisdagen den 25 juni 2013 22.48.41 skrev Dominic König:
> But this isn't quite right: if the user navigates away, e.g. by choosing a
> menu item outside of the workflow, or from his browser bookmarks, then you
> may connect a request to the workflow that doesn't belong to it.

Typical catch:

User starts a workflow in one browser tab, and an additional (non-workflow) page
in another tab. This typical habit of users with modern browsers speaks
strongly against session.

> Sorry for having made fun of it - but it was very hard to resist... :S

It could be worse: it could be my brother typing :D these customer service
guys compensate their constant hyper-friendliness with quite a dark kind of
humor.

Dominic

[hardik juneja]

If a workflow renders a datatable, then it renders all action buttons in the
datatable with the workflow ID in the action URL.

When the user then clicks on an action button for a record, then the workflow
ID is automatically re-submitted (in the request URL, i.e. get_vars).

If he instead navigates away to another page outside of the workflow, then the
action buttons on that page would not include the workflow ID, hence requests
from that page are not handled by the workflow anymore.

Appending the workflow id in the action button looks sufficient for now.

 

You may need additional elements for workflow navigation: "back", "cancel",
"skip", "continue", "cancel" and the like, probably as additional header or
footer elements (e.g. "buttons") - and of course (optional? configurable!) info
elements to display the workflow status (could be a header, for example) and
instructions for the user.

I have all of these in my mind. :)

do you want this to be done earlier?

 

Furthermore, you may need menu items to return to pending workflows after
navigating away. This isn't always required, though.

So if we are going to have a menu then it will be good to save the workflow id and its name, with its status (current and completed) in the DB too.

Sorry for having made fun of it - but it was very hard to resist... :S

Its okay, I too found it funny :)

what's the name of the handbook that you mentioned earlier?

[Dominic König]

Can we have our meeting on IRC please?

Dominic

[hardik juneja]

yes!!

On Wed, Jun 26, 2013 at 1:21 PM, Dominic König <dom...@nursix.org> wrote:

Can we have our meeting on IRC please?

Dominic

--

Hardik Juneja

[hardik juneja]

H

ello Dominic,

Sorry for the delay

Things got delayed due to git conflicts other git issues.

please review the pull request 

https://github.com/flavour/eden/pull/522

--

Hardik Juneja

[Dominic König]

Already done.

[Dominic König]

Hardik--

let me re-iterate my "Shopping List" example:

In this (imaginary) workflow, we start with a request to GET a list of supply
items:

/supply/item/wf_id=reqitems

This initiates a new workflow "reqitems" - S3Request.__call__ will detect the
"wf_id" and hand over to the workflow engine, which executes the request in the
context of the workflow.

Normally, /supply/item just gives you a list view of supply items - it's not
leading anywhere.

But with the workflow identifier in the URL, this presents a list view of supply
items *plus* additional action items:

1) an action button for each record to add it to the "shopping list"
2) a menu to see the currently "selected" items with option to "deselect" them
again
3) a button "Request these items"

So, if the user clicks on a "Add to shopping list" button, then this initiates
a request to

supply/item/4?wf_id=reqitems:1&wf_action=select

(This time it has an instance-ID for the workflow)

Again: the S3Request detects the workflow qualifier and hands this request over
to the workflow engine for processing.

If the user clicks on "Request these items", then this button links to

GET req/req/create?wf_id=reqitems:1

And again, S3Request detects the workflow qualifier and hands the request over
to the workflow engine. This will return a req/req create-form that is
prepopulated with the previously selected items. The user can fill in this form
with the request details, and then submit to

POST req/req/create?wf_id=reqitems:1

And again, this goes to the workflow engine which now creates the new request.

None of the requests ever goes to a dedicated workflow-controller - but through
regular RESTful controllers. That is, the opposite of what you did:

Instead of:
/workflow/index?c=supply&f=item&wf_id=req_items

we call:
/supply/item?wf_id=reqitems

This has a lot of advantages:

a) controller settings can be applied even by the workflow engine, e.g. you
have the prep and postp of that controller available
b) controller-based authorization rules apply automatically
c) you get the right S3Request/S3Resource combo - no need to "fake" anything
d) you get the right menu/breadcrumbs status

The tricky part is to connect the steps by rendering the action items into the
page, and to realize the interim data storage (e.g. the "list of selected
items").

What we need right now is:

a) tweak of S3Request.__call__() to call the workflow engine if the request is
addressed at a workflow
b) the workflow engine to instantiate the respective workflow (or return to the
regular S3Request if no such workflow can be found)
c) a configuration pattern for workflows: this can be a Python-class pattern for
now like workflow_x = S3WorkflowNode(...) & S3WorkflowNode(...) &
(S3WorkflowNode(...) | S3WorkflowNode(...))

If we have that, we have to discuss how we match a request to the right
workflow node, and how we store the current workflow status.

And then we discuss a framework how we render workflow action items into the
page.

Once all this works, we can discuss how to serialize workflows to store them in
the DB (or export/import them via XML) - but for now we really need the engine
first.

The key people I know who are keen to implement workflows in Eden are all
Python-capable - and only when they are happy, we extend it beyond that
audience.

Dominic

torsdagen den 27 juni 2013 00.47.29 skrev hardik juneja:

[hardik juneja]

Hmmm,

This is getting interesting :)

I almost got every thing from the start to the end.

Instead of:
/workflow/index?c=supply&f=item&wf_id=req_items

No, my url pattern was same as yours.

Something like 

so /workflow/index was a place which shows you a form to enter the route so if you enter

-> req

-> create

-> commit

-> list

and then click submit button

[ But now we will use define a configuration pattern for workflows: this can be a Python-class pattern for

now like workflow_x = S3WorkflowNode(...) & S3WorkflowNode(...) &

(S3WorkflowNode(...) | S3WorkflowNode(...))

 

]

then it first takes you to req/req/create, as you click submit then to req/commit/list
so the url will for above was - 

req/req/commit/create?workflow=req_items:1Ab34e:w3sd:234asd

then, 

req/req/commit?workflow=req_items:1Ab34e:w3sd:234asd


I have got what we are trying to do
Thanks a lot
for such a detailed example :)

[Fran Boon]

On 26 June 2013 22:41, Dominic König <dom...@nursix.org> wrote:
> let me re-iterate my "Shopping List" example:
> In this (imaginary) workflow

I don't think this is imaginary at all - I think it's an excellent
example which would be well worth coding :)

F

[hardik juneja]

Hi Dominic --

I have cleaned up the pull request and moved the condition check into Request.__Call__()
its working fine.

And, I will like to discuss the design for defining configurations for the workflow engine.

You suggested something like -

 workflow_x = S3WorkflowNode(...) & S3WorkflowNode(...) &

(S3WorkflowNode(...) | S3WorkflowNode(...))

​can we this be -

workflow_1 = [ S3WorkflowNode("req", "req", "create"), 

[S3WorkflowNode("req", "req", "list"), S3WorkflowNode("supply", "item", "create")], 

S3WorkflowNode("req", "commit", "list") ]​

or 

workfow_1 = [ ("req", "req", "create"),

[("supply", "items", "create"), ("req", "req", "list")] ,

("req", "commit", "list") ] 

​First one would be better​, I think.

​After defining this we will have to save it somewhere.

And then our S3Workflow class will acess it and will pop each step and will execute it.
and after popping if it found that its is a list then it will check for the status and will execute the required status.

​

​Regards,

Hardik juneja​

[Dominic König]

Hmm --

that's your call, actually.

I'm not so much a friend of "Can we..." questions - of course we "can".
Instead, I'd like to see a good reasoning behind such design decisions, i.e.
an answer to the question "why should we"?

I'm not so convinced that a nested list structure will be sufficient - there may
be more operators required to specify connections between steps than one
(=comma is the only available operator in nested lists).

But as I say - your decision.

Dominic

[hardik juneja]

I'm not so much a friend of "Can we..." questions - of course we "can".
Instead, I'd like to see a good reasoning behind such design decisions, i.e.
an answer to the question "why should we"?

First of all i will like to know how what we are going to do with the configuration variable, i mean in this case "workflow_x"

workflow_x = S3WorkflowNode(...) & S3WorkflowNode(...) &
(S3WorkflowNode(...) | S3WorkflowNode(...))

what i had in my mind was if we use a nested list pattern something like -

workflow_1 = [ S3WorkflowNode("req", "req", "create"), 

[S3WorkflowNode("req", "req", "list"), S3WorkflowNode("supply", "item", "create")], 

S3WorkflowNode("req", "commit", "list") ]

then we can store this variable somewhere and we make workflow class to access this list and and then our workflow class should be intelligent enough to decide the order of the node to call.

so it takes each elements from the list and execute it

and if it found a sublists then it check for the workflow data and look for correct element in the sublist and execute it.

but, the problem is how will it come to know what actually a condition is.
i mean the condition on the basis of which the element in the sublist will be chosen.

And,
if we consider this design 

workflow_x = S3WorkflowNode(...) & S3WorkflowNode(...) &
(S3WorkflowNode(...) | S3WorkflowNode(...))

Then i will like to know how what we will do next
are we going to store this somewhere?
or pass it to a class?
if yes, then is that class going to break it on the basis of operators?

Sorry, i am taking time on the design but i think this is important before i can move further

Looking forward to hear from you,
Hardik Juneja

[Fran Boon]

I would have thought nested dicts was a little more flexible than lists

F

[Fran Boon]

On 27 June 2013 21:34, Fran Boon <fr...@sahanafoundation.org> wrote:
> I would have thought nested dicts was a little more flexible than lists

& as I mentioned before, JSON, seems a good storage mechanism - both
for data in page (either jquery.data or a hidden field) & later into
the DB

You definitely need to be able to handle conditional logic...lists
just aren't flexible enough IMHO
With a dict you have a lot more options...

F

[hardik juneja]

You definitely need to be able to handle conditional logic...lists

just aren't flexible enough IMHO
With a dict you have a lot more options...

Nested Dictionary would be much better, I agree.

Are you suggesting json to define the configuration ?
We are discussing about the way to define the configuration for the workflow engine.

Hardik Juneja

[Fran Boon]

On 27 June 2013 22:27, hardik juneja <hardikj...@gmail.com> wrote:
> Nested Dictionary would be much better, I agree.

:)

> Are you suggesting json to define the configuration ?
> We are discussing about the way to define the configuration for the workflow
> engine.

Way to store it

F

[Dominic König]

torsdagen den 27 juni 2013 21.35.50 skrev Fran Boon:
> On 27 June 2013 21:34, Fran Boon <fr...@sahanafoundation.org> wrote:
> > I would have thought nested dicts was a little more flexible than lists
>
> & as I mentioned before, JSON, seems a good storage mechanism - both
> for data in page (either jquery.data or a hidden field) & later into
> the DB

Hmm, right - but storing workflow status/data isn't really what I'm after here.

This is primarily about the workflow definition itself, i.e. how to implement a
workflow. This will need serialization some day, but for now we've agreed to
hardcode it in Python.

> You definitely need to be able to handle conditional logic...lists
> just aren't flexible enough IMHO

This is definitely more important.

But let me become a little more generic here:

Typically, a workflow defines data objects, and certain events for these data
objects which trigger actions.

The workflow engine is to identify the event and the data objects it involves,
determine the status of the objects and from that determine and invoke an
event handler (from a pre-defined set of event handlers) to handle the event.

The event handler would then introspect the data object and determine which
actions to perform and how, then perform the actions and perhaps update the
object's status (if the action is complete), and exit.

---

Or, from a different perspective:

The workflow engine is controlled by events - and if a certain event and a
certain object status come together, a certain event handler is called for the
data object.

If the event handler is successful, it would update the object's status, so
when a next event for the same data object occurs, it would have a different
status, thus be handled by another event handler.

With this general principle, the workflow engine moves a data object from one
event handler (node) to another, until a final status is reached.

---

The first class of events we're focussing on are RESTful requests. The event
hook is what Hardik put into S3Request.__call__ - this detects the event and
hands it over to the workflow engine (=event manager). Additionally, the
S3Request also identifies the data object the event is about(=S3Resource).

We may later get other event classes connected (e.g. status transitions of
data objects can be events in themselves = internal events).

---

The data we need to store in addition to the mere data object is the data
object's current status (within a certain workflow). This is ideally a JSON
structure which can easily be serialized both into the page and the DB.

However, /this/ discussion thread is about how to implement a workflow, which
we agreed to have hardcoded in Python for now (serialization to come later).

The focus for that should be to make it /easy/ for developers to define a
workflow, yet flexible.

Dominic

[Dominic König]

torsdagen den 27 juni 2013 23.47.26 skrev du:
> The first class of events we're focussing on are RESTful requests. The
> event hook is what Hardik put into S3Request.__call__ - this detects the
> event and hands it over to the workflow engine (=event manager).
> Additionally, the S3Request also identifies the data object the event is
> about(=S3Resource).

Well...and the type of event: the request with all it's parameters.

Dominic

[Dominic König]

But in the standard case, S3Request does not take the workflow /status/ of the
resource into account - i.e. it does not connect any previous actions
performed on the same object to the current event.

This is a basic principle of RESTful API's: the interface is stateless, it
forgets what you (or anyone) did before and handles every request as if it was
the first ever.

However, the data resources are not "stateless" - they can be changed by
requests so subsequent requests find a modified resource. This does not violate
the principles of the API.

Therefore (and this comes with an exclamation mark): workflow is focussed
around the "status" of the /resource/ (and not the status of the interface!).

It does not really matter what your last request was - the critical point is
what the current status of the resource is, and what "next" status is to be
achieved within a certain workflow.

Additionally, events in a workflow don't have to involve always one and the
same user - in fact, it is quite common that it wouldn't be the same user.

E.g., in our shopping list example, after sending the request the workflow
isn't really complete yet ;)

Dominic

[hardik juneja]

Dominic --

What about my question :(

https://groups.google.com/d/msg/sahana-eden/uWXty9xQiqg/oVIg4nzOGTIJ

[Dominic König]

So,

the workflow "nodes" are not really requests, but statuses of the data object
(in our example the "request"):

new => prepare => requested => committed => fulfilled

Alternative routes are possible:

new => prepare => cancel
new => prepare => requested => cancel

Statuses could also be more complex, of course.

Whenever a node renders a page, it offers the action items to achieve a status
transition. That is the whole point of an interactive workflow node.

E.g., in our example, for the "new => prepare" transition that would be a
button that says "Request the selected items". Clicking this button moves the
workflow data object from the "new" status to the "prepare" status, thereby
moving it forward.

Of course, the same user could have multiple "shopping list" workflow instances
in process at the same time, each at a different status (e.g. one new, another
requested).

Dominic

[Dominic König]

Where to put the workflow configuration?

Good question - I don't know yet.

Since they need to be accessible cross-controller, it can't be in controllers.
And since they need to be template-specific, it can't be models either.

Seems they are best placed in a separate module á la "menus.py" or
"layouts.py" (workflows.py?).

Another option is of course to put them into separate models.

Whichever option you choose - ideally, a workflow doesn't produce any overheads
whatsoever unless/until it's needed. It was therefore I suggested a class-
pattern, because that only happens when the object really gets requested (like
method handlers, which are mere class handles until they have to process a
request).

I would really like to hear your suggestions first before I answer this,
otherwise I'm developing this instead of you - and that's not really what you
want is it?

Dominic

[Fran Boon]

On 27 June 2013 23:53, Dominic König <dom...@nursix.org> wrote:
> Where to put the workflow configuration?

> Since they need to be accessible cross-controller, it can't be in controllers.
> And since they need to be template-specific, it can't be models either.
> Seems they are best placed in a separate module á la "menus.py" or
> "layouts.py" (workflows.py?).

+1

Just like we have parser.py too...

F

[hardik juneja]

Sorry i slept yesterday

Whichever option you choose - ideally, a workflow doesn't produce any overheads
whatsoever unless/until it's needed. It was therefore I suggested a class-
pattern, because that only happens when the object really gets requested (like
method handlers, which are mere class handles until they have to process a
request).

I would really like to hear your suggestions first before I answer this,
otherwise I'm developing this instead of you - and that's not really what you
want is it?

Storing the configuration in modules say workflow.py like we do in menu.py is what i had in mind too