RailsConf

[bokmann]

My railsconf talk on Workflow has been accepted:

http://en.oreilly.com/rails2009/public/schedule/detail/7535

Hope to see you guys there!

-db

[John Mettraux]

This is great. I especially like the "whole span" approach you're
taking. Looking forward to see the slides and the video.

If you need any help, please ask.

One major drawback of ruote "on Rails" is that, since it requires
threads for its workqueue, it can't work on passenger for instance.
Multi process mongrel packs require tweaking. It's not a big problem
for small company deployments, but the tendency in the Rails world (as
you know) is to "scale somehow", so :(

One option is ruote-rest behind the Rails application. We were
discussing that with Kenneth (he's using ruote[-rest] to drive DNS
provisioning flows), that's his solution : ruote-rest behind.

There'll be a convergence between ruote and ruote-rest in the future.
ruote-web2 will then work with ruote embedded (single process) or with
an 'external' ruote[-rest].

OK, just wanted to underline those issues (especially since you'll be
mentioning ruote in your talk).

Congratulations !!

--
John Mettraux - http://jmettraux.wordpress.com

[J B]

On Sat, Feb 28, 2009 at 10:21 AM, John Mettraux <jmet...@openwfe.org> wrote:

On Sat, Feb 28, 2009 at 11:54 PM, bokmann <bok...@gmail.com> wrote:
>
> My railsconf talk on Workflow has been accepted:
>
> http://en.oreilly.com/rails2009/public/schedule/detail/7535
>
> Hope to see you guys there!

This is great. I especially like the "whole span" approach you're
taking. Looking forward to see the slides and the video.

If you need any help, please ask.

One major drawback of ruote "on Rails" is that, since it requires
threads for its workqueue, it can't work on passenger for instance.
Multi process mongrel packs require tweaking. It's not a big problem
for small company deployments, but the tendency in the Rails world (as
you know) is to "scale somehow", so :(

Have these drawbacks been detailed in another thread? If not, would you care to expound on them here?

[Kenneth Kalmer]

Not outside #ruote, and I lost my chat logs recently...

What I discussed with John was my concerns over the ruote engine running inside mongrel. We have three scenarios to cope with here:

1) Small app, single mongrel
2) Non-small app, multiple mongrels (single or multiple hosts, doesn't matter)
3) New-kid-on-the-block Passenger deployments

The problem will make itself clear when one grasps that only one instance of the engine can be running at any given time. The threading issues might have been resolved (or severely masked) by Rails 2.2.2. The remaining issue is that of a single rufus-scheduler thread that needs to run.

In the first scenario, no problem whatsoever. Single mongrel, single scheduler.

The second one I'm not too sure on. With some initializer magic you can overcome the multiple scheduler/engine instances problem, but how would you interact with the engine in a mongrel where the engine isn't running?

Passenger is out, just because of how it will shutdown your app if it doesn't need to serve requests.

OK, problems aside, lets investigate one solution: ruote-rest.

As John mentioned, we use ruote-rest to run our processes. Our processes involve a lot of participants (mostly custom made) and some very long running steps in between as well. The processes are mostly used to provision services on our infrastructure (DNS, hosting, mail, internet connectivity, etc). I manipulate the engine completely through httparty-based classes that provide some additional convenience to my apps. I also have my process definitions as ERb files, and they get parsed before sending them to the engine, thus making dynamic participants a breeze.

I saw the problems mentioned above immediately, and went for ruote-rest from the word go. I thought it was clear to everyone and didn't want to mention it in fear of looking like a n00b. Seems I was wrong...

What we've been discussing in #ruote is that I tackle some cleaning up of ruote-rest to make it more daemon-kit like. You'll have a pre-defined directory structure where you can save your custom configurations and participants and ruote-rest will be loaded from gems or vendor/. This simplifies customizing your own "engine", and simplifies upgrading the components to a simple "rake ruote:freeze" (think Rails) call. I'll also provide a simple ruote-rest-client (name pending) gem that can be used by Rails, or any other Ruby app, to access and manipulate the engine. As for ruote talking back to Rails, no problem, just use the ActiveResourceParticipant (http://openwferu.rubyforge.org/participants.html#ActiveResourceParticipant) by Torsten.

This gives us the following advantages:

* Easy deployment outside of your Rails code
* Process definitions live inside Rails, not ruote
* Easy customization of ruote & ruote-rest environment
* Easy upgrade of ruote & rufus libs
... and many more...

HTH

--
Kenneth Kalmer
kenneth...@gmail.com
http://opensourcery.co.za

[John Mettraux]

On Sun, Mar 1, 2009 at 2:58 AM, Kenneth Kalmer <kenneth...@gmail.com> wrote:
> On Sat, Feb 28, 2009 at 6:32 PM, J B <jbwe...@gmail.com> wrote:
>>
>>
>> On Sat, Feb 28, 2009 at 10:21 AM, John Mettraux <jmet...@openwfe.org>
>> wrote:
>>> One major drawback of ruote "on Rails" is that, since it requires
>>> threads for its workqueue, it can't work on passenger for instance.
>>> Multi process mongrel packs require tweaking. It's not a big problem
>>> for small company deployments, but the tendency in the Rails world (as
>>> you know) is to "scale somehow", so :(
>>
>> Have these drawbacks been detailed in another thread? If not, would you
>> care to expound on them here?
>
> Not outside #ruote, and I lost my chat logs recently...
>
> What I discussed with John was my concerns over the ruote engine running
> inside mongrel. We have three scenarios to cope with here:
>
> 1) Small app, single mongrel
> 2) Non-small app, multiple mongrels (single or multiple hosts, doesn't
> matter)
> 3) New-kid-on-the-block Passenger deployments

Hi John, Hi Kenneth,

great reply Kenneth, makes my sunday easier :)

Yes, Passenger seems quite radical with threading :
http://groups.google.com/group/rufus-ruby/browse_frm/thread/8ca374769edf98b5

Life is quite simple for state machines (aasm, workflow) in Rails,
they are simply "request bound", request comes, triggers transition,
resouce reach new state, done. Nothing else to do before next
transition-triggering request.

Ruote, as implemented now, needs threading 1) for its workqueue 2) for
scheduling.

State machines in Rails can be made to deal with 'scheduling' via cron
(or at), see Matt's comment at :
http://stackoverflow.com/questions/349711/ruby-on-rails-state-machines.

Back to ruote, the scheduling aspect is important, many workflow /
business processes use timeouts (this job has to be done within 3
days) or wait time (wait 1 week then proceed with the next phase)...
Ruote is using rufus-scheduler for that. (and Ruote has a plain <cron>
expression : http://openwferu.rubyforge.org/expressions.html#exp_cron).

Ruote's workqueue is essential, a launch request to ruote isn't
executed immediately, the root of the process instance is created and
an "apply" call is placed in the workqueue, and the engine immediately
replies with a pointer (flow expression id) to the root of the process
instance (so that you know the process instance id / workflow instance
id). All the job occurs asynchronously as the worker thread picks
apply/reply call on the workqueue and processes them one by one.

There's a bit of context for that at :
http://jmettraux.wordpress.com/2008/09/07/how-does-ruote-work/ (but
there have been some optimizations since)

Somehow it would be possible to have a multiple-process ruote (if
caching is disabled). But schedulers would compete for expressions, as
Kenneth said, juggling to prioritize schedulers would be required.

Ruote-rest is a nice approach, a web application for ruote alone. You
can think of it as a backend workflow server.

Ruote-web2 is more of a example of a small application (BTW, I've had
the ugly ruote-web running for months at
http://difference.openwfe.org:3000/ without issues, single Mongrel on
a cheap Linode instance, small app, small audience).

(for future versions of ruote, I was thinking about making the
workqueue HTTP driven. applies are posts and replies are merely puts
(cancel are deletes). The application would become [http] client to
itself... But that's just a crazy idea for now...)

I'm looking forward to any help about ruote-rest and ruote-web2.
Kenneth has helped a lot with ruote-rest, I'm looking forward to
continue this collaboration.

Thanks again to Kenneth for his great explanation, best regards,

[David Bock]

While this is an interesting discussion, I don't consider this a red herring.  In a well-planned architecture, mongrel and passenger are all about the request-response cycle.  It is inappropriate to try to fire up some other long-lived process in that space.  Other architectural solutions exist for this, as you mentioned, ruote-rest would be one, a ruby process and drb would be another.

I routinely build systems that have independent processes for various kinds of scheduling/sweeping/updating tasks, and have them either as things that get run occasionally by cron, or if they are long running processes, have rake tasks to start and stop them that can then be tied into a capistrano deploy.

Sounds like an interesting sidebar for the talk - too bad railsconf talks are only an hour (really, just 50 minutes).  Based on my experience with NFJS, 90 minutes really strikes a sweet spot between technical depth and mentally draining your audience. 

-db

[J B]

Do you know if deploying on JRuby/Glassfish negates this problem?

Thanks,
JB

[John Mettraux]

2009/3/4 J B <jbwe...@gmail.com>:

>
>> Passenger is out, just because of how it will shutdown your app if it doesn't need to serve requests.
>
> Do you know if deploying on JRuby/Glassfish negates this problem?

Hello,

give a look at Glassfish's manual to see how it works with respect to
threads (maybe processes).

Then give it a try (don't forget to give feedback with your findings).

Here I'm using ruote (0.9.18) inside of a classical Java web
application (Tomcat), thanks to JRuby. Works fine. Simple workflows
and time-based ones are OK.

Best regards,

[Scott Sproule]

I ran this under jruby/glassfish for about a year for a telco and it
worked well.
S.