Ruote and Heroku

[Chris Conley]

Hey all,

I'm attempting to get Ruote rolling on the Heroku Cedar stack with Rails. I'm trying to have the main app just launch workflows and have a separate process actually manage the workflows. Here's a gist of my application config, rake task, and Procfile: https://gist.github.com/1727398

Everything is working locally; workflows are launched from the app and the separate long-running rake task properly manages them. On Heroku, it seems workflows are getting launched properly but the separate long-running process isn't picking up on them.

I'm using Rails 3.1, Ruby 1.9.2 and am on the latest ruote and ruote-redis. Let me know if I can provide any other info that might be helpful.

Thanks!

Chris

[John Mettraux]

On Thu, Feb 02, 2012 at 07:38:24PM -0800, Chris Conley wrote:
>
> Everything is working locally; workflows are launched from the app and the
> separate long-running rake task properly manages them. On Heroku, it seems
> workflows are getting launched properly but the separate long-running
> process isn't picking up on them.

Hello Chris,

this looks very good. I hadn't looked at Heroku in a while, I didn't know about
the background processes, it's brilliant.

I have a series of questions and checks, I'm trying to list
them from easy to hard (to answer/investigate):

== no errors appearing in the dashboard?

---8<---
pp ::RuoteDashboard.processes
pp ::RuoteDashboard.errors
--->8---

What do they yield ?

== is it really not picking up 'msgs'?

When you launch a process, it should place a "msg" in the "msgs" slot of
redis, does calling llen like in:

---8<---
require 'rubygems'
require 'redis'

redis = ::Redis.new()
p redis.llen('msgs')
--->8---

yield 1 all the time?

It should reach 0 (no work to do for the ruote worker) at some point. If it
stays at 1, it means the worker isn't picking up the tasks.

== is the worker redis conn pointing to the same :db as the front end?

You're using default settings for the Redis connection, could those two Ruby
processes point to different numbers? You could try setting it up in hard,
like in:

---8<---
engine = Ruote::Engine.new(
Ruote::Worker.new(
Ruote::Redis::RedisStorage.new('db'=> 14, 'thread_safe' => true)))
--->8---

== could 'thread_safe' => true help?

In the code right above, 'thread_safe' is set to true. Could it help?

Looking at the redis gem code (and at ruote[-redis] code), it seems it
shouldn't matter (and you're telling me that it works locally). But it's easy
to try.

== 1 in 21 drop

You've probably seen the latest messages in

http://groups.google.com/group/openwferu-users/browse_thread/thread/3513bc6fe783c570

1 in 21 "msgs" (workflow worker operations) get lost in case of high load.

But that does not seem to fit your issue description (nothing happening).
It's probably not that.

== instrument Ruote::Redis::Storage#get_msgs

You could add some debug/log output in

https://github.com/jmettraux/ruote-redis/blob/8b5e847874fb13753541e197de806ba312e739c8/lib/ruote/redis/storage.rb#L147-155

in your ruote-redis gem/bundle to see what's going on in there.

An answer for each question would be much appreciated. If you think of
anything else that might help, please tell me.

Best regards,

--
John Mettraux - http://lambda.io/processi

[Chris Conley]

Hi John,

Thanks very much for the reply; great to know that I was on the right track and I think I've got everything working. I started with your questions and found the following:

> ::RuoteDashboard.processes

=> []

> ::RuoteDashboard.errors

=> []

> ::REDIS.llen('msgs')

=> 0

Even when I launched a definition directly, `RuoteDashboard.processes` remained an empty array even though a workflow id was returned:

> RuoteDashboard.launch(t.process_definition, {"task" => {"id" => t.id.to_s, "input" => t.input}})

=> "20120203-disokesa"

> RuoteDashboard.processes

=> []

> RuoteDashboard.errors

=> []

So I double checked my Redis configuration, which was:

Resque.redis = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password, :thread_safe=>true)

Resque.after_fork do |job|

  Resque.redis = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password, :thread_safe=>true)

end

::REDIS = Resque.redis

Resque namespaces its connection so ::REDIS was namespaced to "resque", which seems to be what was causing ruote not finding processes. Changing my config to the following seems to have fixed things up:

::REDIS = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password, :thread_safe=>true)

Resque.after_fork do |job|

  Resque.redis = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password, :thread_safe=>true)

end

Resque.redis = ::REDIS

I'll keep an eye on things and let you know if any other issues come up with running this type of configuration on Heroku.

Thanks!

Chris

[Chris Conley]

I also just saw your note about the workitems disappearing and have upgraded ruote-redis. Thanks!