EC2, Starling, and Workling

16 views

Skip to first unread message

Jonathan Spooner

unread,

Aug 23, 2009, 4:46:55 PM8/23/09

to ec2-on-rai...@googlegroups.com, star...@googlegroups.com, work...@googlegroups.com

Hi,

I have implemented this Workling/Paperclip/S3 tutorial and I'm now trying to deploy it to EC2 and I'm having problems.

http://aaronvb.com/blog/2009/7/19/paperclip-amazon-s3-background-upload-using-starling-and-workling

I've been working with this for the past 2 days and figure out why UploadWorker isn't being run on the server. Starling & Workling both seem to be running and I'm able to use IRB to get and send messages. I can see in starling_top that my rails app is adding messages to the que.

Does anyone else have Workling deployed on EC2? I'm kinda baffled on this one. Here is all the info, maybe I'm running something as the wrong user or a permission is incorrect. I'd appreciate any help you can give.

Thanks!

Machine

EC2 on Rails

Version 0.9.9.1

http://rubyforge.org/projects/ec2onrails/

Base AMI built using Eric Hammond's EC2 Ubuntu script:

http://alestic.com/

This is my instance set up process.

cap ec2onrails:get_public_key_from_server
cap ec2onrails:setup
cap ec2onrails:server:install_packages
ec2 < sudo apt-get -y install build-essential
ec2 < apt-get install libmagick9-dev
ec2 < sudo gem install rmagick
ec2 < sudo gem install right_aws 
ec2 < sudo gem install fiveruns-memcache-client
ec2 < gem install eventmachine
ec2 < gem install starling
cap deploy cold
ec2 < cd /mnt/app
ec2 < ls -al 
ec2 < cd current
ec2 < RAILS_ENV=production rake db:schema:load

Workling Version 0.4.2.3, 31.01.2009

production:/mnt/app/current# ls -al log/

total 208

drwxrwxr-x 2 app app 4096 2009-08-23 18:38 .

drwxrwxr-x 5 app app 4096 2009-08-23 16:40 ..

-rw-rw-rw- 1 root root 11665 2009-08-23 18:38 mongrel.8000.log

-rw-r--r-- 1 app app 4 2009-08-23 18:38 mongrel.8000.pid

-rw-rw-rw- 1 root root 11013 2009-08-23 18:38 mongrel.8001.log

-rw-r--r-- 1 app app 4 2009-08-23 18:38 mongrel.8001.pid

-rw-rw-rw- 1 root root 12344 2009-08-23 18:38 mongrel.8002.log

-rw-r--r-- 1 app app 4 2009-08-23 18:38 mongrel.8002.pid

-rw-rw-rw- 1 root root 14187 2009-08-23 18:38 mongrel.8003.log

-rw-r--r-- 1 app app 4 2009-08-23 18:38 mongrel.8003.pid

-rw-rw-rw- 1 root root 16038 2009-08-23 18:38 mongrel.8004.log

-rw-r--r-- 1 app app 4 2009-08-23 18:38 mongrel.8004.pid

-rw-rw-rw- 1 root root 14925 2009-08-23 18:38 mongrel.8005.log

-rw-r--r-- 1 app app 4 2009-08-23 18:38 mongrel.8005.pid

-rw-rw-rw- 1 app app 63615 2009-08-23 18:38 newrelic_agent.log

-rw-r--r-- 1 root root 51 2009-08-23 16:53 production.log

-rw-rw-rw- 1 app app 2752 2009-08-23 17:35 workling.log

-rw-rw-rw- 1 root root 5 2009-08-23 18:38 workling_monitor.pid

-rw-rw-rw- 1 app app 1495 2009-08-23 18:38 workling.output

-rw-rw-rw- 1 root root 5 2009-08-23 18:38 workling.pid

Workling is running.

production:/mnt/app/current# RAILS_ENV=production script/workling_client start

production:/mnt/app/current# ps aux | grep workling

root 8268 0.0 0.3 8480 6044 ? S 18:50 0:00 workling_monitor

root 8270 34.2 1.1 23740 20728 ? R 18:50 0:01 workling

root 8273 0.0 0.0 3072 720 pts/2 R+ 18:50 0:00 grep workling

Starling is running on port 15151

ps aux | grep starling

root 8078 0.0 0.4 12488 8096 ? S 18:37 0:00 /usr/bin/ruby1.8 /usr/bin/starling -d -p 15151

root 8277 0.0 0.0 3072 720 pts/2 R+ 18:51 0:00 grep starling

? maybe run starling as app - starling and workling run as root. Rails runs as app is this a problem?

My workling yaml file works in development.

app/config/workling.yml

production:

listens_on: localhost:15151

sleep_time: 1

reset_time: 1

# memcache_options:

# namespace: shrelp_development

# listeners:

# Util:

# sleep_time: 2

development:

listens_on: localhost:22122

# sleep_time: 2

# reset_time: 30

# listeners:

# Util:

# sleep_time: 20

memcache_options:

namespace: skatestock_development

Gems

root@production:/mnt/app/current# gem list --local

*** LOCAL GEMS ***

actionmailer (2.2.2, 2.1.2, 2.0.5, 1.3.6)

actionpack (2.2.2, 2.1.2, 2.0.5, 1.13.6)

actionwebservice (1.2.6)

activerecord (2.2.2, 2.1.2, 2.0.5, 1.15.6)

activeresource (2.2.2, 2.1.2, 2.0.5)

activesupport (2.2.2, 2.1.2, 2.0.5, 1.4.4)

amazon-ec2 (0.3.2)

aws-s3 (0.5.1)

builder (2.1.2)

cgi_multipart_eof_fix (2.5.0)

daemons (1.0.10)

eventmachine (0.12.8)

fastthread (1.0.1)

fiveruns-memcache-client (1.5.0.5) <-- I tried this with the original memcache-client and this fiveruns-memcache-client --

gem_plugin (0.2.3)

hoe (1.8.3)

memcache-client (1.7.4)

mime-types (1.15)

mongrel (1.1.5)

mongrel_cluster (1.0.5)

optiflag (0.6.5)

rails (2.2.2, 2.1.2, 2.0.5, 1.2.6)

rake (0.8.7, 0.8.3)

right_aws (1.10.0)

right_http_connection (1.2.4)

rmagick (2.11.0)

rubyforge (1.0.4, 1.0.2)

starling-starling (0.10.0)

SyslogLogger (1.4.0)

xml-simple (1.0.11)

ZenTest (3.11.1)

The Starling files are being created. I was also able to open up IRB, connect to Starling via localhost:15151 and get/set values on 'bla'. This makes me think starling is working.

production:/mnt/app/current# ls -al /var/spool/starling/

total 16

drwxrwxrwx 2 root root 4096 2009-08-23 18:25 .

drwxr-xr-x 6 root root 4096 2009-08-23 16:58 ..

-rw-rw-rw- 1 root root 70 2009-08-23 18:14 bla

-rw-rw-rw- 1 root root 0 2009-08-23 18:25 image_workers__moo

-rw-rw-rw- 1 root root 0 2009-08-23 18:12 o

-rw-rw-rw- 1 root root 0 2009-08-23 18:25 paperclip_background_upload_workers__upload_to_amazon_s3

-rw-rw-rw- 1 root root 0 2009-08-23 18:12 shrelp_development

-rw-rw-rw- 1 root root 14 2009-08-23 18:10 shrelp_development:image_workers__moo

-rw-rw-rw- 1 root root 0 2009-08-23 17:54 shrelp_development:paperclip_background_upload_workers__upload_to_amazon_s3

Starling_top

This is a bit strange how I can see the que of items created from Rails. All the ques I created via IRB disappear after i get them.

production:/mnt/app/current# starling_top -p 15151

bytes: 0

bytes_read: 192117

bytes_written: 41964

cmd_get: 4421

cmd_set: 7

curr_connections: 1

curr_items: 0

get_hits: 7

get_misses: 4414

limit_maxbytes: 0

pid: 8078

queue_image_workers__moo_age: 0

queue_image_workers__moo_expired_items: 0

queue_image_workers__moo_items: 0

queue_image_workers__moo_logsize: 0

queue_image_workers__moo_total_items: 0

queue_paperclip_background_upload_workers__upload_to_amazon_s3_age: 273

queue_paperclip_background_upload_workers__upload_to_amazon_s3_expired_items: 0

queue_paperclip_background_upload_workers__upload_to_amazon_s3_items: 0

queue_paperclip_background_upload_workers__upload_to_amazon_s3_logsize: 109

queue_paperclip_background_upload_workers__upload_to_amazon_s3_total_items: 7

rusage_system: 0.010000

rusage_user: 0.070000

time: 1251054934

total_connections: 14

total_items: 7

uptime: 2303

version: 0.10.0

Now this irb sessions is strange. I'm able to get and set via a new Starling instance but I'm not able to access

anything created by Rais/Workling.

IRB Session

production:/mnt/app/current# irb

irb(main):001:0> require 'rubygems'

=> true

irb(main):002:0> require 'starling'

=> true

irb(main):003:0> s = Starling.new('localhost:15151')

=> <MemCache: 1 servers, ns: nil, ro: false>

irb(main):004:0> s.set('ruby','rocks')

=> "STORED\r\n"

irb(main):005:0> s.get('ruby')

=> "rocks"

irb(main):006:0> s.get('paperclip_background_upload_workers__upload_to_amazon_s3')

^?IRB::Abort: abort then interrupt!!

from /usr/lib/ruby/1.8/irb.rb:81:in `irb_abort'

from /usr/lib/ruby/1.8/irb.rb:243:in `signal_handle'

from /usr/lib/ruby/1.8/irb.rb:66:in `start'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:16:in `call'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:16:in `sleep'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:16:in `get'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:13:in `loop'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:13:in `get'

from (irb):6

irb(main):007:0> s.set('paperclip_background_upload_workers__upload_to_amazon_s3','666')

=> "STORED\r\n"

irb(main):008:0> s.get('paperclip_background_upload_workers__upload_to_amazon_s3')

IRB::Abort: abort then interrupt!!

from /usr/lib/ruby/1.8/irb.rb:81:in `irb_abort'

from /usr/lib/ruby/1.8/irb.rb:243:in `signal_handle'

from /usr/lib/ruby/1.8/irb.rb:66:in `start'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:16:in `call'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:16:in `sleep'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:16:in `get'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:13:in `loop'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:13:in `get'

from (irb):8

irb(main):009:0> s.set('ruby','rocks')

=> "STORED\r\n"

irb(main):010:0> s.get('paperclip_background_upload_workers')

IRB::Abort: abort then interrupt!!

from /usr/lib/ruby/1.8/irb.rb:81:in `irb_abort'

from /usr/lib/ruby/1.8/irb.rb:243:in `signal_handle'

from /usr/lib/ruby/1.8/irb.rb:66:in `start'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:16:in `call'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:16:in `sleep'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:16:in `get'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:13:in `loop'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:13:in `get'

from (irb):10

Also since I installed 'starling-starling' from github the get method nolonger returns nil and it locks up.

irb(main):013:0> s.set('ruby','rocks')

=> "STORED\r\n"

irb(main):014:0> s.get('ruby')

=> "rocks"

irb(main):015:0> s.get('ruby')

=> "rocks"

irb(main):016:0> s.get('ruby')

^[[A

IRB::Abort: abort then interrupt!!

from /usr/lib/ruby/1.8/irb.rb:81:in `irb_abort'

from /usr/lib/ruby/1.8/irb.rb:243:in `signal_handle'

from /usr/lib/ruby/1.8/irb.rb:66:in `start'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:16:in `call'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:16:in `sleep'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:16:in `get'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:13:in `loop'

from /usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:13:in `get'

from (irb):16

irb(main):017:0>

Workling Log

production:/mnt/app/current# tail -f log/workling.log

Sun Aug 23 17:08:49 +0000 2009

SIGTERM

["/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:57:in `sleep'", "/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:57:in `watch'", "/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:44:in `loop'", "/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:44:in `watch'", "/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:84:in `start_with_pidfile'", "/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:64:in `fork'", "/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:64:in `start_with_pidfile'", "/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:111:in `start'", "/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/application_group.rb:113:in `create_monitor'", "/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/application.rb:244:in `start'", "/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/controller.rb:69:in `run'", "/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons.rb:139:in `run'", "/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `call'", "/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `catch_exceptions'", "/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons.rb:138:in `run'", "/mnt/app/current/script/workling_client:17"]

Sun Aug 23 17:35:30 +0000 2009

SIGTERM

I'm not sure why the starling_status doesn't work locally or in production.

production:/mnt/app/current# RAILS_ENV=production ruby script/starling_status.rb

=> Loading Rails...

** Rails loaded.

Queue state:

"#<Workling::Clients::MemcacheQueueClient:0xb71d55e4 @queueserver_urls=[\"localhost:15151\"], @connection=<MemCache: 1 servers, 1 buckets, ns: nil, ro: false>>"

"Active?: true"

"Read Only?: false"

Servers:

[<MemCache::Server: localhost:15151 [1] (CONNECTED)>]

Queue stats:

{"localhost:15151"=>

{"queue_image_workers__moo_items"=>0,

"bytes"=>0,

"pid"=>8078,

"time"=>1251055942,

"queue_paperclip_background_upload_workers__upload_to_amazon_s3_total_items"=>

"limit_maxbytes"=>0,

"queue_image_workers__moo_total_items"=>0,

"queue_image_workers__moo_expired_items"=>0,

"cmd_get"=>6426,

"version"=>"0.10.0",

"bytes_written"=>57473,

"cmd_set"=>8,

"queue_paperclip_background_upload_workers__upload_to_amazon_s3_age"=>127,

"queue_paperclip_background_upload_workers__upload_to_amazon_s3_logsize"=>

125,

"get_misses"=>6418,

"total_connections"=>16,

"curr_connections"=>1,

"curr_items"=>0,

"uptime"=>3311,

"queue_image_workers__moo_logsize"=>0,

"queue_paperclip_background_upload_workers__upload_to_amazon_s3_expired_items"=>

"queue_paperclip_background_upload_workers__upload_to_amazon_s3_items"=>0,

"get_hits"=>8,

"total_items"=>8,

"rusage_system"=>0.02,

"rusage_user"=>0.09,

"queue_image_workers__moo_age"=>0,

"bytes_read"=>278448}}

Thread Stats:

[#<Thread:0xb721edfc sleep>, #<Thread:0xb7fb86fc run>]

** Exiting

/mnt/app/releases/20090823182205/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/vendor/memcache-client-1.5.0/memcache.rb:697:in `raise_on_error_response!': bad command line format (MemCache::MemCacheError)

from /mnt/app/releases/20090823182205/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/vendor/memcache-client-1.5.0/memcache.rb:414:in `flush_all'

from /mnt/app/releases/20090823182205/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/vendor/memcache-client-1.5.0/memcache.rb:408:in `each'

from /mnt/app/releases/20090823182205/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/vendor/memcache-client-1.5.0/memcache.rb:408:in `flush_all'

from /mnt/app/releases/20090823182205/vendor/plugins/workling/lib/workling/clients/memcache_queue_client.rb:44:in `close'