Storage History Behavior

[Doug Bryant]

Starting a new thread to reflect the current topic...

So for part of our evaluation of Ruote, we wanted to be able to look at the history.  The below solution was provided and works great for tests, but *sometimes* errors out when asking outside the context of a test.

I added this code to the Ruote::StorageHistory in our app - it just adds a method to StorageHistory which filters on dispatched and returns the participant_name sorted by original_put_at.  https://gist.github.com/2207156

So when we run this as part of tests, it works exactly like we expect it to - it returns an array of executed steps.

However, when we put it behind a url to ask what the steps for a given process are, I inconsistently get the following error - sometimes it works and sometimes it doesn't.

Rufus::Json::ParserError - lexical error: invalid char in json text.

          ce'])\\n  workitem.fields[\\"case_result\\"] = \\"Completed 

                     (right here) ------^

:

The process definition looks like this:

  STAGES_PDEF = Ruote.define do
    sequence do
      stage1
      stage2
      stage3
    end
  end

And the participant looks like this:

RuoteKit.engine.register_participant :stage1 do |workitem|
  Case.run_stage1(workitem.fields['reference'])
  workitem.fields["case_result"] = "Completed Stage 1"
end

The process definition works fine and completes.  It's only when asking about the history that it fails with this error.  It fails on line 6 of the above pasted code:  all_expressions = self.by_process(wfid)

Any ideas as to why this might occur?  Is it breaking because the process has already completed by the time the history is requested?  

Thanks, Doug

On Friday, March 23, 2012 3:46:44 PM UTC-5, John Mettraux wrote:

On Fri, Mar 23, 2012 at 09:38:11AM -0500, Doug Bryant wrote:
>
> One other question.  Is there a way to access the history of a workflow
> process. I saw Ruote::StorageHistory but not quite sure if that will work.
>  If on the previous example I append the two lines
>
> history = dashboard.context.history.by_process(wfid)
> > history.each{|h| puts h["participant_name"]}
>
> (...)
>
> It looks like the execution path through ruote, but includes lines such as
> concurrence (blank line - no participant name).  Looking more for a result
> like %w{con1 con2 print_messages}

Hello Doug,

yes, this history implementation records every event in the engine.

You could write:

---8<---
history = dashboard.history.by_process(wfid)
p history.select { |h|
  h['action'] == 'dispatched'
}.collect { |h|
  h['participant_name']
}
--->8---

That'll only display the participant for 'dispatched' messages (emitted by
the engine we it has successfully handed a workitem to a participant).

You could also replace the history by your own implementation only keeping
track of the events you care for.

> Thanks for your work on Ruote.

You're welcome. Best regards,

--
John Mettraux - http://lambda.io/processi

[John Mettraux]

Hello Doug,

what version of Ruby (and patchlevel) on which platform?
What JSON library are you using (YAJL, json or pure-json)?
What does the Rufus::Json::ParserError backtrace look like?

Your gist looks good, a more economic alternative would be to do the
filtering in the StorageParticipant#on_msg so that messages that don't
interest you are discarded and don't take up storage space. I need to provide
a hook to simplify such an approach...

In your gist: those aren't "expressions" but "messages", the history stores
messages.

Thanks in advance,

[Doug Bryant]

ruby 1.9.2p290 (2011-07-09 revision 32553) [x86_64-darwin11.3.0] vía RVM on Lion.

For json, we are using YAJL; Rufus::Json.backend = :yajl

Will update our code from the gist to reflect message rather than expression (thanks for that feedback).

Since you indicated looking at the json parser, I tried the json one also. Here are the results from each

YAJL:

> 2012-03-26 14:58:58] INFO WEBrick 1.3.1
> [2012-03-26 14:58:58] INFO ruby 1.9.2 (2011-07-09) [x86_64-darwin11.3.0]
> [2012-03-26 14:58:58] INFO WEBrick::HTTPServer#start: pid=46914 port=9292

> Rufus::Json::ParserError - lexical error: invalid char in json text.
> ce'])\\n workitem.fields[\\"case_result\\"] = \\"Completed
> (right here) ------^
> :

> /Users/doug/devroot/apps/bpm_reference/vendor/ruby/1.9.1/gems/rufus-json-1.0.1/lib/rufus/json.rb:185:in `rescue in decode'
> /Users/doug/devroot/apps/bpm_reference/vendor/ruby/1.9.1/gems/rufus-json-1.0.1/lib/rufus/json.rb:182:in `decode'
> /Users/doug/devroot/apps/bpm_reference/vendor/ruby/1.9.1/gems/ruote-sequel-2.2.0/lib/ruote/sequel/storage.rb:180:in `block in get_many'
> /Users/doug/devroot/apps/bpm_reference/vendor/ruby/1.9.1/gems/ruote-sequel-2.2.0/lib/ruote/sequel/storage.rb:180:in `collect'
> /Users/doug/devroot/apps/bpm_reference/vendor/ruby/1.9.1/gems/ruote-sequel-2.2.0/lib/ruote/sequel/storage.rb:180:in `get_many'
> /Users/doug/devroot/apps/bpm_reference/vendor/ruby/1.9.1/gems/ruote-2.3.0/lib/ruote/log/storage_history.rb:79:in `by_process'
> /Users/doug/devroot/apps/bpm_reference/lib/ruote_history_ext.rb:6:in `steps_by_process'

JSON

> Rufus::Json::ParserError - 399: unexpected token at '{"on_workitem":"proc do |workitem|\\n Case.run_stage3(workitem.fields['reference'])\\n workitem.fields[\\"case_result\\"] = \\"Completed Stage 3\\"\\nend"}],"workitem":{"fields":{"reference":"81","case_result":"Completed Stage 2","params":{"ref":"stage3"}},"wf_name":null,"wf_revision":null,"fei":{"engine_id":"engine","wfid":"20120326-2003-kabaroma-gijiruno","subid":"2e810d141687fef23c228ea005607b7a","expid":"0_0_2"},"participant_name":"stage3","re_dispatch_count":0},"type":"history","_id":"47068!70119173172880!2012-03-26!1332792219.243796!018!0_0_2!2e810d141687fef23c228ea005607b7a!20120326-2003-kabaroma-gijiruno","action":"dispatch","put_at":"2012-03-26 20:03:39.347652 UTC","original_id":"47068!70119173172880!2012-03-26!1332792219.243796!018","original_put_at":"2012-03-26 20:03:39.244019 UTC","_rev":1}]':
> /Users/doug/devroot/apps/bpm_reference/vendor/ruby/1.9.1/gems/rufus-json-1.0.1/lib/rufus/json.rb:185:in `rescue in decode'
> /Users/doug/devroot/apps/bpm_reference/vendor/ruby/1.9.1/gems/rufus-json-1.0.1/lib/rufus/json.rb:182:in `decode'
> /Users/doug/devroot/apps/bpm_reference/vendor/ruby/1.9.1/gems/ruote-sequel-2.2.0/lib/ruote/sequel/storage.rb:180:in `block in get_many'
> /Users/doug/devroot/apps/bpm_reference/vendor/ruby/1.9.1/gems/ruote-sequel-2.2.0/lib/ruote/sequel/storage.rb:180:in `collect'
> /Users/doug/devroot/apps/bpm_reference/vendor/ruby/1.9.1/gems/ruote-sequel-2.2.0/lib/ruote/sequel/storage.rb:180:in `get_many'
> /Users/doug/devroot/apps/bpm_reference/vendor/ruby/1.9.1/gems/ruote-2.3.0/lib/ruote/log/storage_history.rb:79:in `by_process'
> /Users/doug/devroot/apps/bpm_reference/lib/ruote_history_ext.rb:6:in `steps_by_process'

The process completes successfully, it just errors out when accessing the history

Doug

> --
> you received this message because you are subscribed to the "ruote users" group.
> to post : send email to openwfe...@googlegroups.com
> to unsubscribe : send email to openwferu-use...@googlegroups.com
> more options : http://groups.google.com/group/openwferu-users?hl=en

[John Mettraux]

2012/3/27 John Mettraux <jmet...@gmail.com>:

>
> Your gist looks good, a more economic alternative would be to do the
> filtering in the StorageParticipant#on_msg so that messages that don't
> interest you are discarded and don't take up storage space. I need to provide
> a hook to simplify such an approach...

https://github.com/jmettraux/ruote/commit/c04d7ee7d9f8703cd00fa6f453627dd67ae764c6

I hope it can useful to someone.

Best regards,

[John Mettraux]

On Mon, Mar 26, 2012 at 03:12:07PM -0500, Doug Bryant wrote:
>
> > /Users/doug/devroot/apps/bpm_reference/vendor/ruby/1.9.1/gems/rufus-json-1.0.1/lib/rufus/json.rb:185:in `rescue in decode'
> > /Users/doug/devroot/apps/bpm_reference/vendor/ruby/1.9.1/gems/rufus-json-1.0.1/lib/rufus/json.rb:182:in `decode'
> > /Users/doug/devroot/apps/bpm_reference/vendor/ruby/1.9.1/gems/ruote-sequel-2.2.0/lib/ruote/sequel/storage.rb:180:in `block in get_many'
> > /Users/doug/devroot/apps/bpm_reference/vendor/ruby/1.9.1/gems/ruote-sequel-2.2.0/lib/ruote/sequel/storage.rb:180:in `collect'
> > /Users/doug/devroot/apps/bpm_reference/vendor/ruby/1.9.1/gems/ruote-sequel-2.2.0/lib/ruote/sequel/storage.rb:180:in `get_many'

Hello Doug,

thanks for the detailed information. I see that you're using ruote-sequel :
What database are you using?

I'm trying to set up a test case.

[Doug Bryant]

We're using postgresql as the backend to route-sequel.

I'm also trying to setup a test to be able to show off the described behavior.

Doug

[Doug Bryant]

This gist demonstrates the behavior we are seeing.  https://gist.github.com/2209952

It's a heavily modified version of the barley.rb in examples.

It's a little bit manual and thrown together in a hurry. You need to set the db connection variables and manually create the cases table (sql above Case object). 

Thanks for taking a look at this.

Doug

[John Mettraux]

On Mon, Mar 26, 2012 at 02:44:17PM -0700, Doug Bryant wrote:
>
> This gist demonstrates the behavior we are seeing.
> https://gist.github.com/2209952
>
> It's a heavily modified version of the barley.rb in examples.
>
> It's a little bit manual and thrown together in a hurry. You need to set
> the db connection variables and manually create the cases table (sql above
> Case object).

Sorry, what does the Gemfile[.lock] look like?

[Doug Bryant]

Here is our Gemfile.  We had to build a gem from the route repository last week and put it into our internal repo because we couldn't get the app deployed correctly with a bundler :git reference.  Just remove the unnecessary bits like amqp and point route to the git repo.

https://gist.github.com/2210082

Doug

[John Mettraux]

On Mon, Mar 26, 2012 at 05:01:11PM -0500, Doug Bryant wrote:
>
> Here is our Gemfile. We had to build a gem from the route repository last week and put it into our internal repo because we couldn't get the app deployed correctly with a bundler :git reference. Just remove the unnecessary bits like amqp and point route to the git repo.
>
> https://gist.github.com/2210082

Hello Doug,

thanks, may I suggest using ruote-sequel from master? It has evolved since
the 2.2.0 gem you seem to be using. ruote-sequel master is aligned on ruote
master.

I'll try by myself in a few hours.

Thanks again,

[Doug Bryant]

That wasn't it. The app experienced the same behavior.

The one thing I did notice is that if you remove the line(s) where the model is updated from the process definitions, i.e. "Case.update_case_stage(workitem.fields['reference'], "stage3")", the error goes away.

Doug

[John Mettraux]

On Mon, Mar 26, 2012 at 06:00:08PM -0500, Doug Bryant wrote:
>
> That wasn't it. The app experienced the same behavior.
>
> The one thing I did notice is that if you remove the line(s) where the model is updated from the process definitions, i.e. "Case.update_case_stage(workitem.fields['reference'], "stage3")", the error goes away.

Hello Doug,

I reworked your gist, trimming it:

https://gist.github.com/2219926

I couldn't reproduce the error.

May I suggest you try this gist in your environment (you'd have to adapt the
sequel connection bit) and tell me how it goes?

You should also compare the gist's Gemfile.lock with your Gemfile.lock.

Best regards,

[Doug Bryant]

It must have been the specific versions of gems we were using. I updated your revised example to do the same thing as yesterday (connect to Active record model object & save data) and it works fine.

Thanks for the help with questions and implementing the StorageHistory#accept? method for filtering storage items.

Doug