Daemon Processes

[Matthew Sumner]

Ok brand new to everything here so bare with me.

Spent the last two days getting things up and running on our Snow
Leopard Sever and after battling with apache scripts and the like I
have a website up that I can see and login to *cheer*. The next
problem I'm having is that I have no email's or background processes
being run on the server. I know why they aren't running and it's
because I've only created the GEM_HOME file and not installed the
daemon process manager canvas_init. The installation instructions are
only for Debain/Ubuntu which seems to handle daemon differently. From
what I understand (which is nothing) I need to be creating a plist
file to get this script to run on startup.

Besides that if I just try "/private/var/rails/canvas/script/
canvas_init" in the terminal (Is this even kosher?) I get the error :

"readlink: illegal option -- f
usage: readlink [-n] [file ...]
usage: dirname path
/private/var/rails/canvas/script/canvas_init: line 31: /
script/delayed_job: No such file or directory"

Now I found a post here from June so I just wanted to check if this
was a hint or not:

Christopher Bennell
Jun 23 2011, 10:10 am
Just a note for anyone running the jobs daemon on OS X. The
canvas_init
script calls readlink and stat, which are slightly different
on OS X. You
will get an error message: "readlink: illegal option -- f"

I installed the coreutils brew package and replaced the
calls to readlink
and stat with greadlink and gstat, seems to have fixed the
problem.

So I went ahead and installed coreutils and modified caves_init
replacing readlink and stat with greadlink and gstat. I then ran:
"sudo /private/var/rails/canvas/script/canvas_init start"

and I get:
"Daemonizing..."

Which I hoped means it's now working? However, if I go to
https://<your-canvas-hostname>/jobs I can see all my
DelayedNotification.process showing in the job list but nothing in
running jobs.

Not sure if it's any relevance but if I run:
"/private/var/rails/canvas/script/canvas_init status"

I get:
"No delayed jobs pool running"

Can anyone shed some light on what I'm missing?

[Christopher Bennell]

I would check /private/var/rails/canvas/log/delayed_job.log for error messages. If the script is crashing after it daemonizes, the error message will go to the log rather than to the terminal. 

Also, we've had some trouble running on OS X. Passenger crashes sporadically and there are some problems with MySQL connections. We are currently migrating to Linux. FYI!

[Matthew Sumner]

Thanks for pointing me in the right direction. The log reads as
follows:

NOTE: Gem.source_index is deprecated, use Specification. It will be
removed on or after 2011-11-01.
Gem.source_index called from /Users/admin/.rvm/gems/ruby-1.8.7-p357/
gems/rails-2.3.14/lib/rails/gem_dependency.rb:21.
/private/var/rails/canvas/vendor/plugins/delayed_job/lib/delayed/
pool.rb:89:in `send': undefined method `fatal' for nil:NilClass
(NoMethodError)
from /private/var/rails/canvas/vendor/plugins/delayed_job/lib/delayed/
pool.rb:89:in `say'
from /private/var/rails/canvas/vendor/plugins/delayed_job/lib/delayed/
pool.rb:81:in `start'
from /private/var/rails/canvas/vendor/plugins/delayed_job/lib/delayed/
pool.rb:56:in `run'
from /private/var/rails/canvas/script/delayed_job:5

Which it repeats. Looking into it and I'll post here if I find a
solution. A little wary of starting again with a linux box as we have
a big OSX server here doing nothing and although there's only me
messing around in Canvas, it's been fairly stable. That might all
change when some more traffic is hitting it but we'll see.

[Brian Palmer]

It looks like it's failing when trying to log a message. For some reason your logger is nil, which isn't something I've seen before. What does your config/logging.yml look like?

-- Brian Palmer

Software Developer

Instructure, Inc.

[Matthew Sumner]

This must be the root of the problem. I only have the
logger.yml.example file. Looking back through the installation
instructions the only place I can see where it could have been
generated is the File Generation step. The other .yml files with
only .yml.example I have are facebooker, incoming_mail, memcache,
redis, saml, scribd, selenium, session_store, tinychat, twitter and
zendesk. Not sure how many if any should have been copied to .yml
files, for instance I know I don't need redis.yml as it's covered by
cache_store.yml.

Having copied logger.yml.example to logger.yml and restarted
everything I now get the following in delayed_job.log:

/Users/admin/.rvm/rubies/ruby-1.8.7-p357/lib/ruby/1.8/yaml.rb:143:in
`initialize': Permission denied - /private/var/rails/canvas/config/
cache_store.yml (Errno::EACCES)

So it's a permission error with cache_store.yml? Remade the
cache_store.yml (permissions must have been messed up while trying to
fix things) and now get:

NOTE: Gem.source_index is deprecated, use Specification. It will be
removed on or after 2011-11-01.
Gem.source_index called from /Users/admin/.rvm/gems/ruby-1.8.7-p357/
gems/rails-2.3.14/lib/rails/gem_dependency.rb:21.

[- -] [37390]P Job master died with error: #<ArgumentError: syntax
error on line 8, col 3: ` cache_store: redis_store'>
/Users/admin/.rvm/rubies/ruby-1.8.7-p357/lib/ruby/1.8/yaml.rb:133:in
`load'
/Users/admin/.rvm/rubies/ruby-1.8.7-p357/lib/ruby/1.8/yaml.rb:133:in
`load'
/Users/admin/.rvm/rubies/ruby-1.8.7-p357/lib/ruby/1.8/yaml.rb:144:in
`load_file'
/Users/admin/.rvm/rubies/ruby-1.8.7-p357/lib/ruby/1.8/yaml.rb:143:in
`open'
/Users/admin/.rvm/rubies/ruby-1.8.7-p357/lib/ruby/1.8/yaml.rb:143:in
`load_file'
/private/var/rails/canvas/app/models/setting.rb:84:in `from_config'
/private/var/rails/canvas/lib/canvas.rb:57:in `cache_store_config'
/private/var/rails/canvas/config/environments/production.rb:22:in
`load_environment'
/Users/admin/.rvm/gems/ruby-1.8.7-p357/gems/rails-2.3.14/lib/
initializer.rb:386:in `load_environment'
/Users/admin/.rvm/gems/ruby-1.8.7-p357/gems/activesupport-2.3.14/lib/
active_support/core_ext/kernel/reporting.rb:11:in `silence_warnings'
/Users/admin/.rvm/gems/ruby-1.8.7-p357/gems/rails-2.3.14/lib/
initializer.rb:379:in `load_environment'
/Users/admin/.rvm/gems/ruby-1.8.7-p357/gems/rails-2.3.14/lib/
initializer.rb:137:in `process'
/Users/admin/.rvm/gems/ruby-1.8.7-p357/gems/rails-2.3.14/lib/
initializer.rb:113:in `send'
/Users/admin/.rvm/gems/ruby-1.8.7-p357/gems/rails-2.3.14/lib/
initializer.rb:113:in `run'
/private/var/rails/canvas/config/environment.rb:9
/private/var/rails/canvas/vendor/plugins/delayed_job/lib/delayed/
pool.rb:96:in `require'
/private/var/rails/canvas/vendor/plugins/delayed_job/lib/delayed/
pool.rb:96:in `load_rails'
/private/var/rails/canvas/vendor/plugins/delayed_job/lib/delayed/
pool.rb:67:in `start'
/private/var/rails/canvas/vendor/plugins/delayed_job/lib/delayed/
pool.rb:45:in `run'
/private/var/rails/canvas/script/delayed_job:5
/Users/admin/.rvm/rubies/ruby-1.8.7-p357/lib/ruby/1.8/yaml.rb:133:in
`load': syntax error on line 8, col 3: ` cache_store:
redis_store' (ArgumentError)
from /Users/admin/.rvm/rubies/ruby-1.8.7-p357/lib/ruby/1.8/
yaml.rb:133:in `load'
from /Users/admin/.rvm/rubies/ruby-1.8.7-p357/lib/ruby/1.8/
yaml.rb:144:in `load_file'
from /Users/admin/.rvm/rubies/ruby-1.8.7-p357/lib/ruby/1.8/
yaml.rb:143:in `open'
from /Users/admin/.rvm/rubies/ruby-1.8.7-p357/lib/ruby/1.8/
yaml.rb:143:in `load_file'
from /private/var/rails/canvas/app/models/setting.rb:84:in
`from_config'
from /private/var/rails/canvas/lib/canvas.rb:57:in
`cache_store_config'
from /private/var/rails/canvas/config/environments/
production.rb:22:in `load_environment'
from /Users/admin/.rvm/gems/ruby-1.8.7-p357/gems/rails-2.3.14/
lib/initializer.rb:386:in `load_environment'
from /Users/admin/.rvm/gems/ruby-1.8.7-p357/gems/
activesupport-2.3.14/lib/active_support/core_ext/kernel/
reporting.rb:11:in `silence_warnings'
from /Users/admin/.rvm/gems/ruby-1.8.7-p357/gems/rails-2.3.14/
lib/initializer.rb:379:in `load_environment'
from /Users/admin/.rvm/gems/ruby-1.8.7-p357/gems/rails-2.3.14/
lib/initializer.rb:137:in `process'
from /Users/admin/.rvm/gems/ruby-1.8.7-p357/gems/rails-2.3.14/
lib/initializer.rb:113:in `send'
from /Users/admin/.rvm/gems/ruby-1.8.7-p357/gems/rails-2.3.14/
lib/initializer.rb:113:in `run'
from /private/var/rails/canvas/config/environment.rb:9
from /private/var/rails/canvas/vendor/plugins/delayed_job/lib/
delayed/pool.rb:96:in `require'
from /private/var/rails/canvas/vendor/plugins/delayed_job/lib/
delayed/pool.rb:96:in `load_rails'
from /private/var/rails/canvas/vendor/plugins/delayed_job/lib/
delayed/pool.rb:67:in `start'
from /private/var/rails/canvas/vendor/plugins/delayed_job/lib/
delayed/pool.rb:45:in `run'
from /private/var/rails/canvas/script/delayed_job:5


Thanks for looking at this.

[Matthew Sumner]

Turns out adding the logging.yml file also freaks out passenger.
Here's the logging file I'm trying to use:


development:
logger: rails
# log_context: false
# (log_context defaults to true)

test:
logger: rails

production:
logger: rails
# logger: syslog
# app_ident: canvas-lms
# daemon_ident: canvas-lms-daemon
# facilities:
# - user
# - local3


Do I need to have uncommented anything? Without this file canvas is up
and everything is working except the automated jobs aren't getting run
and with the file passenger kicks up an error "Ruby (Rack) application
could not be started". The error message reads "Permission denied - /
private/var/rails/canvas/log/production.log". Exception class:
Errno::EACCES.

[Matthew Sumner]

Ok, the whole things appears to be a permissions error. I think the
user originally setup to run canvas changed so it appears to be
working now. Just need to look into why emails aren't sending. Thanks
again for all your help.

[walsh]

did you ever get this one solved

[DeviousDuck]

Yes thanks, email turned out to be an issue with our mail server and not canvas. We upgraded our server to Lion which added a whole host of other issues but Canvas is up and running and we're loving it.

[walsh]

I am getting DelayedNotification.process in my jobs and the joblist is filling up. I dont think that my delayed jobs is daemonizing.
I get the following error in my delyaed_jobs log file.

/var/rails/canvas/vendor/

plugins/delayed_job/lib/delayed/pool.rb:95:in `say': undefined method `logger' for Rails:Module (NoMethodError)
        from /var/rails/canvas/vendor/plugins/delayed_job/lib/delayed/pool.rb:87:in `start'
        from /var/rails/canvas/vendor/plugins/delayed_job/lib/delayed/pool.rb:47:in `run'

Any help would be greatly appreciated.

[walsh]

how did you solve the permission error

On Monday, February 6, 2012 4:22:59 PM UTC-4, DeviousDuck wrote:

[DeviousDuck]

Looking in canvas/vendor/plugins/delayed_job/lib/delayed/pool.rb around line 95 is the say class:

  def say(msg, level = :debug)

    if defined?(Rails)

      Rails.logger.send(level, "[#{Process.pid}]P #{msg}")

    else

      puts(msg)

    end

  end

Looking in the environment files there's no where the logger is defined. Try going to your canvas/config directory and cp logging.yml.example to logging.yml, this might fix the logging error which should allow you to check the logs and see what's going on.

What OS is canvas running on? Also try opening a terminal and the command: "~/var/rails/canvas/script/canvas_init status" will tell you if your daemon is running. If not use "~/var/rails/canvas/script/canvas_init start" to start it (remember to replace /var/rails with the path to your canvas install).

Hope this helps.

[walsh]

Thanks for your help...I think i am going to start over.

[walsh]

I did a complete restart but ran into the same problem. but this time im sure its a permission issue. Please help me if you can

root@ec2-23-21-123-145:/var/rails/canvas/config# /var/rails/canvas/vendor/plugins/delayed_job/lib/delayed/pool.rb:198:in `initialize': Permission denied - /var/rails/canvas/tmp/pids/delayed_jobs_pool.pid (Errno::EACCES)
        from /var/rails/canvas/vendor/plugins/delayed_job/lib/delayed/pool.rb:198:in `open'
        from /var/rails/canvas/vendor/plugins/delayed_job/lib/delayed/pool.rb:198:in `daemonize'
        from /var/rails/canvas/vendor/plugins/delayed_job/lib/delayed/pool.rb:46:in `run'
        from /var/rails/canvas/script/delayed_job:5

[DeviousDuck]

I presume your following the Canvas Production Start Guide. There is a Canvas Ownership section which details how permissions need to be set. Depending on your OS you'll want to either create a user with only access to what Canvas needs or you can use the same user as your webserver. This step is relatively importent as you don't what Canvas having access to anything it shouldn't in the event of anything getting compromised via a security oversight or Canvas changing files it shouldn't.

What's in your /private/var/rails/canvas/log/delayed_job.log?

[walsh]

root@ec2-23-21-123-145:/var/

rails/canvas/config# /var/rails/canvas/vendor/plugins/delayed_job/lib/delayed/pool.rb:198:in `initialize': Permission denied - /var/rails/canvas/tmp/pids/delayed_jobs_pool.pid (Errno::EACCES)
        from /var/rails/canvas/vendor/plugins/delayed_job/lib/delayed/pool.rb:198:in `open'
        from /var/rails/canvas/vendor/plugins/delayed_job/lib/delayed/pool.rb:198:in `daemonize'
        from /var/rails/canvas/vendor/plugins/delayed_job/lib/delayed/pool.rb:46:in `run'
        from /var/rails/canvas/script/delayed_job:5