Starling on EC2
1 view
Skip to first unread message
jspooner
Aug 23, 2009, 5:44:33 PM8/23/09
to ec2-on-rails-discuss
Hi,
I have implemented this Workling/Paperclip/S3 tutorial and I'm now
trying to
deploy it to EC2 and I'm having problems.
http://aaronvb.com/blog/2009/7/19/paperclip-amazon-s3-background-uplo...
I've been working with this for the past 2 days and figure out why
UploadWorker isn't being run on the server. Starling & Workling both
seem
to be running and I'm able to use IRB to get and send messages. I can
see
in starling_top that my rails app is adding messages to the que.
Does anyone else have Workling deployed on EC2? I'm kinda baffled on
this
one. Here is all the info, maybe I'm running something as the wrong
user or
a permission is incorrect. I'd appreciate any help you can give.
Thanks!
Machine EC2 on Rails
Version 0.9.9.1
http://rubyforge.org/projects/ec2onrails/
Copyright 2007 Paul Dowman, http://pauldowman.com/
Base AMI built using Eric Hammond's EC2 Ubuntu script:
http://alestic.com/
This is my instance set up process.
cap ec2onrails:get_public_key_from_server
cap ec2onrails:setup
cap ec2onrails:server:install_packages
ec2 < sudo apt-get -y install build-essential
ec2 < apt-get install libmagick9-dev
ec2 < sudo gem install rmagick
ec2 < sudo gem install right_aws
ec2 < sudo gem install fiveruns-memcache-client
ec2 < gem install eventmachine
ec2 < gem install starling
cap deploy cold
ec2 < cd /mnt/app
ec2 < ls -al
ec2 < cd current
ec2 < RAILS_ENV=production rake db:schema:load
Workling Version 0.4.2.3, 31.01.2009
*production:/mnt/app/current# ls -al log/*
total 208
drwxrwxr-x 2 app app 4096 2009-08-23 18:38 .
drwxrwxr-x 5 app app 4096 2009-08-23 16:40 ..
-rw-rw-rw- 1 root root 11665 2009-08-23 18:38 mongrel.8000.log
-rw-r--r-- 1 app app 4 2009-08-23 18:38 mongrel.8000.pid
-rw-rw-rw- 1 root root 11013 2009-08-23 18:38 mongrel.8001.log
-rw-r--r-- 1 app app 4 2009-08-23 18:38 mongrel.8001.pid
-rw-rw-rw- 1 root root 12344 2009-08-23 18:38 mongrel.8002.log
-rw-r--r-- 1 app app 4 2009-08-23 18:38 mongrel.8002.pid
-rw-rw-rw- 1 root root 14187 2009-08-23 18:38 mongrel.8003.log
-rw-r--r-- 1 app app 4 2009-08-23 18:38 mongrel.8003.pid
-rw-rw-rw- 1 root root 16038 2009-08-23 18:38 mongrel.8004.log
-rw-r--r-- 1 app app 4 2009-08-23 18:38 mongrel.8004.pid
-rw-rw-rw- 1 root root 14925 2009-08-23 18:38 mongrel.8005.log
-rw-r--r-- 1 app app 4 2009-08-23 18:38 mongrel.8005.pid
-rw-rw-rw- 1 app app 63615 2009-08-23 18:38 newrelic_agent.log
-rw-r--r-- 1 root root 51 2009-08-23 16:53 production.log
-rw-rw-rw- 1 app app 2752 2009-08-23 17:35 workling.log
-rw-rw-rw- 1 root root 5 2009-08-23 18:38 workling_monitor.pid
-rw-rw-rw- 1 app app 1495 2009-08-23 18:38 workling.output
-rw-rw-rw- 1 root root 5 2009-08-23 18:38 workling.pid
Workling is running.
*production:/mnt/app/current# RAILS_ENV=production script/
workling_client
start*
*production:/mnt/app/current# ps aux | grep workling*
root 8268 0.0 0.3 8480 6044 ? S 18:50 0:00
workling_monitor
root 8270 34.2 1.1 23740 20728 ? R 18:50 0:01
workling
root 8273 0.0 0.0 3072 720 pts/2 R+ 18:50 0:00 grep
workling
Starling is running on port 15151
*ps aux | grep starling*
root 8078 0.0 0.4 12488 8096 ? S 18:37 0:00
/usr/bin/ruby1.8 /usr/bin/starling -d -p 15151
root 8277 0.0 0.0 3072 720 pts/2 R+ 18:51 0:00 grep
starling
? maybe run starling as app - starling and workling run as root.
Rails
runs as app is this a problem?
My workling yaml file works in development.
app/config/workling.yml
production:
listens_on: localhost:15151
sleep_time: 1
reset_time: 1
# memcache_options:
# namespace: shrelp_development
# listeners:
# Util:
# sleep_time: 2
development:
listens_on: localhost:22122
# sleep_time: 2
# reset_time: 30
# listeners:
# Util:
# sleep_time: 20
memcache_options:
namespace: skatestock_development
Gems
root@production:/mnt/app/current# gem list --local
*** LOCAL GEMS ***
actionmailer (2.2.2, 2.1.2, 2.0.5, 1.3.6)
actionpack (2.2.2, 2.1.2, 2.0.5, 1.13.6)
actionwebservice (1.2.6)
activerecord (2.2.2, 2.1.2, 2.0.5, 1.15.6)
activeresource (2.2.2, 2.1.2, 2.0.5)
activesupport (2.2.2, 2.1.2, 2.0.5, 1.4.4)
amazon-ec2 (0.3.2)
aws-s3 (0.5.1)
builder (2.1.2)
cgi_multipart_eof_fix (2.5.0)
daemons (1.0.10)
eventmachine (0.12.8)
fastthread (1.0.1)
fiveruns-memcache-client (1.5.0.5) <-- I tried this with the
original
memcache-client and this fiveruns-memcache-client --
gem_plugin (0.2.3)
hoe (1.8.3)
memcache-client (1.7.4)
mime-types (1.15)
mongrel (1.1.5)
mongrel_cluster (1.0.5)
optiflag (0.6.5)
rails (2.2.2, 2.1.2, 2.0.5, 1.2.6)
rake (0.8.7, 0.8.3)
right_aws (1.10.0)
right_http_connection (1.2.4)
rmagick (2.11.0)
rubyforge (1.0.4, 1.0.2)
starling-starling (0.10.0)
SyslogLogger (1.4.0)
xml-simple (1.0.11)
ZenTest (3.11.1)
The Starling files are being created. I was also able to open up
IRB,
connect to Starling via localhost:15151 and get/set values on 'bla'.
This
makes me think starling is working.
production:/mnt/app/current# ls -al /var/spool/starling/
total 16
drwxrwxrwx 2 root root 4096 2009-08-23 18:25 .
drwxr-xr-x 6 root root 4096 2009-08-23 16:58 ..
-rw-rw-rw- 1 root root 70 2009-08-23 18:14 bla
-rw-rw-rw- 1 root root 0 2009-08-23 18:25 image_workers__moo
-rw-rw-rw- 1 root root 0 2009-08-23 18:12 o
-rw-rw-rw- 1 root root 0 2009-08-23 18:25
paperclip_background_upload_workers__upload_to_amazon_s3
-rw-rw-rw- 1 root root 0 2009-08-23 18:12 shrelp_development
-rw-rw-rw- 1 root root 14 2009-08-23 18:10
shrelp_development:image_workers__moo
-rw-rw-rw- 1 root root 0 2009-08-23 17:54
shrelp_development:paperclip_background_upload_workers__upload_to_amazon_s3
Starling_top
This is a bit strange how I can see the que of items created from
Rails.
All the ques I created via IRB disappear after i get them.
*production:/mnt/app/current# starling_top -p 15151*
bytes: 0
bytes_read: 192117
bytes_written: 41964
cmd_get: 4421
cmd_set: 7
curr_connections: 1
curr_items: 0
get_hits: 7
get_misses: 4414
limit_maxbytes: 0
pid: 8078
queue_image_workers__moo_age: 0
queue_image_workers__moo_expired_items: 0
queue_image_workers__moo_items: 0
queue_image_workers__moo_logsize: 0
queue_image_workers__moo_total_items: 0
queue_paperclip_background_upload_workers__upload_to_amazon_s3_age:
273
queue_paperclip_background_upload_workers__upload_to_amazon_s3_expired_item
s:
0
queue_paperclip_background_upload_workers__upload_to_amazon_s3_items:
0
queue_paperclip_background_upload_workers__upload_to_amazon_s3_logsize:
109
queue_paperclip_background_upload_workers__upload_to_amazon_s3_total_items:
7
rusage_system: 0.010000
rusage_user: 0.070000
time: 1251054934
total_connections: 14
total_items: 7
uptime: 2303
version: 0.10.0
Now this irb sessions is strange. I'm able to get and set via a new
Starling instance but I'm not able to access
anything created by Rais/Workling.
IRB Session
*production:/mnt/app/current# irb*
irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'starling'
=> true
irb(main):003:0> s = Starling.new('localhost:15151')
=> <MemCache: 1 servers, ns: nil, ro: false>
irb(main):004:0> s.set('ruby','rocks')
=> "STORED\r\n"
irb(main):005:0> s.get('ruby')
=> "rocks"
irb(main):006:0>
s.get('paperclip_background_upload_workers__upload_to_amazon_s3')
^?IRB::Abort: abort then interrupt!!
from /usr/lib/ruby/1.8/irb.rb:81:in `irb_abort'
from /usr/lib/ruby/1.8/irb.rb:243:in `signal_handle'
from /usr/lib/ruby/1.8/irb.rb:66:in `start'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`call'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`sleep'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`get'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
13:in
`loop'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
13:in
`get'
from (irb):6
irb(main):007:0>
s.set
('paperclip_background_upload_workers__upload_to_amazon_s3','666')
=> "STORED\r\n"
irb(main):008:0>
s.get('paperclip_background_upload_workers__upload_to_amazon_s3')
IRB::Abort: abort then interrupt!!
from /usr/lib/ruby/1.8/irb.rb:81:in `irb_abort'
from /usr/lib/ruby/1.8/irb.rb:243:in `signal_handle'
from /usr/lib/ruby/1.8/irb.rb:66:in `start'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`call'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`sleep'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`get'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
13:in
`loop'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
13:in
`get'
from (irb):8
irb(main):009:0> s.set('ruby','rocks')
=> "STORED\r\n"
irb(main):010:0> s.get('paperclip_background_upload_workers')
IRB::Abort: abort then interrupt!!
from /usr/lib/ruby/1.8/irb.rb:81:in `irb_abort'
from /usr/lib/ruby/1.8/irb.rb:243:in `signal_handle'
from /usr/lib/ruby/1.8/irb.rb:66:in `start'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`call'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`sleep'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`get'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
13:in
`loop'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
13:in
`get'
from (irb):10
Also since I installed 'starling-starling' from github the get method
nolonger returns nil and it locks up.
irb(main):013:0> s.set('ruby','rocks')
=> "STORED\r\n"
irb(main):014:0> s.get('ruby')
=> "rocks"
irb(main):015:0> s.get('ruby')
=> "rocks"
irb(main):016:0> s.get('ruby')
^[[A
IRB::Abort: abort then interrupt!!
from /usr/lib/ruby/1.8/irb.rb:81:in `irb_abort'
from /usr/lib/ruby/1.8/irb.rb:243:in `signal_handle'
from /usr/lib/ruby/1.8/irb.rb:66:in `start'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`call'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`sleep'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`get'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
13:in
`loop'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
13:in
`get'
from (irb):16
irb(main):017:0>
Workling Log
production:/mnt/app/current# tail -f log/workling.log
Sun Aug 23 17:08:49 +0000 2009
SIGTERM
["/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
57:in
`sleep'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
57:in
`watch'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
44:in
`loop'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
44:in
`watch'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
84:in
`start_with_pidfile'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
64:in
`fork'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
64:in
`start_with_pidfile'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
111:in
`start'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/
application_group.r b:113:in
`create_monitor'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/application.rb:
244: in
`start'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/controller.rb:
69:in
`run'", "/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons.rb:
139:in
`run'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/cmdline.rb:
105:in
`call'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/cmdline.rb:
105:in
`catch_exceptions'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons.rb:138:in
`run'",
"/mnt/app/current/script/workling_client:17"]
Sun Aug 23 17:35:30 +0000 2009
SIGTERM
["/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
57:in
`sleep'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
57:in
`watch'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
44:in
`loop'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
44:in
`watch'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
84:in
`start_with_pidfile'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
64:in
`fork'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
64:in
`start_with_pidfile'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
111:in
`start'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/
application_group.r b:113:in
`create_monitor'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/application.rb:
244: in
`start'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/controller.rb:
69:in
`run'", "/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons.rb:
139:in
`run'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/cmdline.rb:
105:in
`call'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/cmdline.rb:
105:in
`catch_exceptions'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons.rb:138:in
`run'",
"/mnt/app/current/script/workling_client:17"]
I'm not sure why the starling_status doesn't work locally or in
production.
*production:/mnt/app/current# RAILS_ENV=production ruby
script/starling_status.rb*
=> Loading Rails...
** Rails loaded.
Queue state:
"#<Workling::Clients::MemcacheQueueClient:0xb71d55e4
@queueserver_urls=[\"localhost:15151\"], @connection=<MemCache: 1
servers, 1
buckets, ns: nil, ro: false>>"
"Active?: true"
"Read Only?: false"
Servers:
[<MemCache::Server: localhost:15151 [1] (CONNECTED)>]
Queue stats:
{"localhost:15151"=>
{"queue_image_workers__moo_items"=>0,
"bytes"=>0,
"pid"=>8078,
"time"=>1251055942,
"queue_paperclip_background_upload_workers__upload_to_amazon_s3_total_items
"=>
8,
"limit_maxbytes"=>0,
"queue_image_workers__moo_total_items"=>0,
"queue_image_workers__moo_expired_items"=>0,
"cmd_get"=>6426,
"version"=>"0.10.0",
"bytes_written"=>57473,
"cmd_set"=>8,
"queue_paperclip_background_upload_workers__upload_to_amazon_s3_age"=>127,
"queue_paperclip_background_upload_workers__upload_to_amazon_s3_logsize"=>
125,
"get_misses"=>6418,
"total_connections"=>16,
"curr_connections"=>1,
"curr_items"=>0,
"uptime"=>3311,
"queue_image_workers__moo_logsize"=>0,
"queue_paperclip_background_upload_workers__upload_to_amazon_s3_expired_ite
ms"=>
0,
"queue_paperclip_background_upload_workers__upload_to_amazon_s3_items"=>0,
"get_hits"=>8,
"total_items"=>8,
"rusage_system"=>0.02,
"rusage_user"=>0.09,
"queue_image_workers__moo_age"=>0,
"bytes_read"=>278448}}
Thread Stats:
[#<Thread:0xb721edfc sleep>, #<Thread:0xb7fb86fc run>]
** Exiting
/mnt/app/releases/20090823182205/vendor/rails/activerecord/lib/../../
active support/lib/active_support/vendor/memcache-client-1.5.0/
memcache.rb:697:in
`raise_on_error_response!': bad command line format
(MemCache::MemCacheError)
from
/mnt/app/releases/20090823182205/vendor/rails/activerecord/lib/../../
active support/lib/active_support/vendor/memcache-client-1.5.0/
memcache.rb:414:in
`flush_all'
from
/mnt/app/releases/20090823182205/vendor/rails/activerecord/lib/../../
active support/lib/active_support/vendor/memcache-client-1.5.0/
memcache.rb:408:in
`each'
from
/mnt/app/releases/20090823182205/vendor/rails/activerecord/lib/../../
active support/lib/active_support/vendor/memcache-client-1.5.0/
memcache.rb:408:in
`flush_all'
from
/mnt/app/releases/20090823182205/vendor/plugins/workling/lib/workling/
clien ts/memcache_queue_client.rb:44:in
`close'
from script/starling_status.rb:36
I have implemented this Workling/Paperclip/S3 tutorial and I'm now
trying to
deploy it to EC2 and I'm having problems.
http://aaronvb.com/blog/2009/7/19/paperclip-amazon-s3-background-uplo...
I've been working with this for the past 2 days and figure out why
UploadWorker isn't being run on the server. Starling & Workling both
seem
to be running and I'm able to use IRB to get and send messages. I can
see
in starling_top that my rails app is adding messages to the que.
Does anyone else have Workling deployed on EC2? I'm kinda baffled on
this
one. Here is all the info, maybe I'm running something as the wrong
user or
a permission is incorrect. I'd appreciate any help you can give.
Thanks!
Machine EC2 on Rails
Version 0.9.9.1
http://rubyforge.org/projects/ec2onrails/
Copyright 2007 Paul Dowman, http://pauldowman.com/
Base AMI built using Eric Hammond's EC2 Ubuntu script:
http://alestic.com/
This is my instance set up process.
cap ec2onrails:get_public_key_from_server
cap ec2onrails:setup
cap ec2onrails:server:install_packages
ec2 < sudo apt-get -y install build-essential
ec2 < apt-get install libmagick9-dev
ec2 < sudo gem install rmagick
ec2 < sudo gem install right_aws
ec2 < sudo gem install fiveruns-memcache-client
ec2 < gem install eventmachine
ec2 < gem install starling
cap deploy cold
ec2 < cd /mnt/app
ec2 < ls -al
ec2 < cd current
ec2 < RAILS_ENV=production rake db:schema:load
Workling Version 0.4.2.3, 31.01.2009
*production:/mnt/app/current# ls -al log/*
total 208
drwxrwxr-x 2 app app 4096 2009-08-23 18:38 .
drwxrwxr-x 5 app app 4096 2009-08-23 16:40 ..
-rw-rw-rw- 1 root root 11665 2009-08-23 18:38 mongrel.8000.log
-rw-r--r-- 1 app app 4 2009-08-23 18:38 mongrel.8000.pid
-rw-rw-rw- 1 root root 11013 2009-08-23 18:38 mongrel.8001.log
-rw-r--r-- 1 app app 4 2009-08-23 18:38 mongrel.8001.pid
-rw-rw-rw- 1 root root 12344 2009-08-23 18:38 mongrel.8002.log
-rw-r--r-- 1 app app 4 2009-08-23 18:38 mongrel.8002.pid
-rw-rw-rw- 1 root root 14187 2009-08-23 18:38 mongrel.8003.log
-rw-r--r-- 1 app app 4 2009-08-23 18:38 mongrel.8003.pid
-rw-rw-rw- 1 root root 16038 2009-08-23 18:38 mongrel.8004.log
-rw-r--r-- 1 app app 4 2009-08-23 18:38 mongrel.8004.pid
-rw-rw-rw- 1 root root 14925 2009-08-23 18:38 mongrel.8005.log
-rw-r--r-- 1 app app 4 2009-08-23 18:38 mongrel.8005.pid
-rw-rw-rw- 1 app app 63615 2009-08-23 18:38 newrelic_agent.log
-rw-r--r-- 1 root root 51 2009-08-23 16:53 production.log
-rw-rw-rw- 1 app app 2752 2009-08-23 17:35 workling.log
-rw-rw-rw- 1 root root 5 2009-08-23 18:38 workling_monitor.pid
-rw-rw-rw- 1 app app 1495 2009-08-23 18:38 workling.output
-rw-rw-rw- 1 root root 5 2009-08-23 18:38 workling.pid
Workling is running.
*production:/mnt/app/current# RAILS_ENV=production script/
workling_client
start*
*production:/mnt/app/current# ps aux | grep workling*
root 8268 0.0 0.3 8480 6044 ? S 18:50 0:00
workling_monitor
root 8270 34.2 1.1 23740 20728 ? R 18:50 0:01
workling
root 8273 0.0 0.0 3072 720 pts/2 R+ 18:50 0:00 grep
workling
Starling is running on port 15151
*ps aux | grep starling*
root 8078 0.0 0.4 12488 8096 ? S 18:37 0:00
/usr/bin/ruby1.8 /usr/bin/starling -d -p 15151
root 8277 0.0 0.0 3072 720 pts/2 R+ 18:51 0:00 grep
starling
? maybe run starling as app - starling and workling run as root.
Rails
runs as app is this a problem?
My workling yaml file works in development.
app/config/workling.yml
production:
listens_on: localhost:15151
sleep_time: 1
reset_time: 1
# memcache_options:
# namespace: shrelp_development
# listeners:
# Util:
# sleep_time: 2
development:
listens_on: localhost:22122
# sleep_time: 2
# reset_time: 30
# listeners:
# Util:
# sleep_time: 20
memcache_options:
namespace: skatestock_development
Gems
root@production:/mnt/app/current# gem list --local
*** LOCAL GEMS ***
actionmailer (2.2.2, 2.1.2, 2.0.5, 1.3.6)
actionpack (2.2.2, 2.1.2, 2.0.5, 1.13.6)
actionwebservice (1.2.6)
activerecord (2.2.2, 2.1.2, 2.0.5, 1.15.6)
activeresource (2.2.2, 2.1.2, 2.0.5)
activesupport (2.2.2, 2.1.2, 2.0.5, 1.4.4)
amazon-ec2 (0.3.2)
aws-s3 (0.5.1)
builder (2.1.2)
cgi_multipart_eof_fix (2.5.0)
daemons (1.0.10)
eventmachine (0.12.8)
fastthread (1.0.1)
fiveruns-memcache-client (1.5.0.5) <-- I tried this with the
original
memcache-client and this fiveruns-memcache-client --
gem_plugin (0.2.3)
hoe (1.8.3)
memcache-client (1.7.4)
mime-types (1.15)
mongrel (1.1.5)
mongrel_cluster (1.0.5)
optiflag (0.6.5)
rails (2.2.2, 2.1.2, 2.0.5, 1.2.6)
rake (0.8.7, 0.8.3)
right_aws (1.10.0)
right_http_connection (1.2.4)
rmagick (2.11.0)
rubyforge (1.0.4, 1.0.2)
starling-starling (0.10.0)
SyslogLogger (1.4.0)
xml-simple (1.0.11)
ZenTest (3.11.1)
The Starling files are being created. I was also able to open up
IRB,
connect to Starling via localhost:15151 and get/set values on 'bla'.
This
makes me think starling is working.
production:/mnt/app/current# ls -al /var/spool/starling/
total 16
drwxrwxrwx 2 root root 4096 2009-08-23 18:25 .
drwxr-xr-x 6 root root 4096 2009-08-23 16:58 ..
-rw-rw-rw- 1 root root 70 2009-08-23 18:14 bla
-rw-rw-rw- 1 root root 0 2009-08-23 18:25 image_workers__moo
-rw-rw-rw- 1 root root 0 2009-08-23 18:12 o
-rw-rw-rw- 1 root root 0 2009-08-23 18:25
paperclip_background_upload_workers__upload_to_amazon_s3
-rw-rw-rw- 1 root root 0 2009-08-23 18:12 shrelp_development
-rw-rw-rw- 1 root root 14 2009-08-23 18:10
shrelp_development:image_workers__moo
-rw-rw-rw- 1 root root 0 2009-08-23 17:54
shrelp_development:paperclip_background_upload_workers__upload_to_amazon_s3
Starling_top
This is a bit strange how I can see the que of items created from
Rails.
All the ques I created via IRB disappear after i get them.
*production:/mnt/app/current# starling_top -p 15151*
bytes: 0
bytes_read: 192117
bytes_written: 41964
cmd_get: 4421
cmd_set: 7
curr_connections: 1
curr_items: 0
get_hits: 7
get_misses: 4414
limit_maxbytes: 0
pid: 8078
queue_image_workers__moo_age: 0
queue_image_workers__moo_expired_items: 0
queue_image_workers__moo_items: 0
queue_image_workers__moo_logsize: 0
queue_image_workers__moo_total_items: 0
queue_paperclip_background_upload_workers__upload_to_amazon_s3_age:
273
queue_paperclip_background_upload_workers__upload_to_amazon_s3_expired_item
s:
0
queue_paperclip_background_upload_workers__upload_to_amazon_s3_items:
0
queue_paperclip_background_upload_workers__upload_to_amazon_s3_logsize:
109
queue_paperclip_background_upload_workers__upload_to_amazon_s3_total_items:
7
rusage_system: 0.010000
rusage_user: 0.070000
time: 1251054934
total_connections: 14
total_items: 7
uptime: 2303
version: 0.10.0
Now this irb sessions is strange. I'm able to get and set via a new
Starling instance but I'm not able to access
anything created by Rais/Workling.
IRB Session
*production:/mnt/app/current# irb*
irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'starling'
=> true
irb(main):003:0> s = Starling.new('localhost:15151')
=> <MemCache: 1 servers, ns: nil, ro: false>
irb(main):004:0> s.set('ruby','rocks')
=> "STORED\r\n"
irb(main):005:0> s.get('ruby')
=> "rocks"
irb(main):006:0>
s.get('paperclip_background_upload_workers__upload_to_amazon_s3')
^?IRB::Abort: abort then interrupt!!
from /usr/lib/ruby/1.8/irb.rb:81:in `irb_abort'
from /usr/lib/ruby/1.8/irb.rb:243:in `signal_handle'
from /usr/lib/ruby/1.8/irb.rb:66:in `start'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`call'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`sleep'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`get'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
13:in
`loop'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
13:in
`get'
from (irb):6
irb(main):007:0>
s.set
('paperclip_background_upload_workers__upload_to_amazon_s3','666')
=> "STORED\r\n"
irb(main):008:0>
s.get('paperclip_background_upload_workers__upload_to_amazon_s3')
IRB::Abort: abort then interrupt!!
from /usr/lib/ruby/1.8/irb.rb:81:in `irb_abort'
from /usr/lib/ruby/1.8/irb.rb:243:in `signal_handle'
from /usr/lib/ruby/1.8/irb.rb:66:in `start'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`call'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`sleep'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`get'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
13:in
`loop'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
13:in
`get'
from (irb):8
irb(main):009:0> s.set('ruby','rocks')
=> "STORED\r\n"
irb(main):010:0> s.get('paperclip_background_upload_workers')
IRB::Abort: abort then interrupt!!
from /usr/lib/ruby/1.8/irb.rb:81:in `irb_abort'
from /usr/lib/ruby/1.8/irb.rb:243:in `signal_handle'
from /usr/lib/ruby/1.8/irb.rb:66:in `start'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`call'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`sleep'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`get'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
13:in
`loop'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
13:in
`get'
from (irb):10
Also since I installed 'starling-starling' from github the get method
nolonger returns nil and it locks up.
irb(main):013:0> s.set('ruby','rocks')
=> "STORED\r\n"
irb(main):014:0> s.get('ruby')
=> "rocks"
irb(main):015:0> s.get('ruby')
=> "rocks"
irb(main):016:0> s.get('ruby')
^[[A
IRB::Abort: abort then interrupt!!
from /usr/lib/ruby/1.8/irb.rb:81:in `irb_abort'
from /usr/lib/ruby/1.8/irb.rb:243:in `signal_handle'
from /usr/lib/ruby/1.8/irb.rb:66:in `start'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`call'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`sleep'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
16:in
`get'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
13:in
`loop'
from
/usr/lib/ruby/gems/1.8/gems/starling-starling-0.10.0/lib/starling.rb:
13:in
`get'
from (irb):16
irb(main):017:0>
Workling Log
production:/mnt/app/current# tail -f log/workling.log
Sun Aug 23 17:08:49 +0000 2009
SIGTERM
["/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
57:in
`sleep'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
57:in
`watch'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
44:in
`loop'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
44:in
`watch'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
84:in
`start_with_pidfile'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
64:in
`fork'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
64:in
`start_with_pidfile'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
111:in
`start'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/
application_group.r b:113:in
`create_monitor'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/application.rb:
244: in
`start'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/controller.rb:
69:in
`run'", "/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons.rb:
139:in
`run'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/cmdline.rb:
105:in
`call'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/cmdline.rb:
105:in
`catch_exceptions'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons.rb:138:in
`run'",
"/mnt/app/current/script/workling_client:17"]
Sun Aug 23 17:35:30 +0000 2009
SIGTERM
["/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
57:in
`sleep'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
57:in
`watch'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
44:in
`loop'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
44:in
`watch'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
84:in
`start_with_pidfile'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
64:in
`fork'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
64:in
`start_with_pidfile'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/monitor.rb:
111:in
`start'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/
application_group.r b:113:in
`create_monitor'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/application.rb:
244: in
`start'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/controller.rb:
69:in
`run'", "/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons.rb:
139:in
`run'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/cmdline.rb:
105:in
`call'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/cmdline.rb:
105:in
`catch_exceptions'",
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons.rb:138:in
`run'",
"/mnt/app/current/script/workling_client:17"]
I'm not sure why the starling_status doesn't work locally or in
production.
*production:/mnt/app/current# RAILS_ENV=production ruby
script/starling_status.rb*
=> Loading Rails...
** Rails loaded.
Queue state:
"#<Workling::Clients::MemcacheQueueClient:0xb71d55e4
@queueserver_urls=[\"localhost:15151\"], @connection=<MemCache: 1
servers, 1
buckets, ns: nil, ro: false>>"
"Active?: true"
"Read Only?: false"
Servers:
[<MemCache::Server: localhost:15151 [1] (CONNECTED)>]
Queue stats:
{"localhost:15151"=>
{"queue_image_workers__moo_items"=>0,
"bytes"=>0,
"pid"=>8078,
"time"=>1251055942,
"queue_paperclip_background_upload_workers__upload_to_amazon_s3_total_items
"=>
8,
"limit_maxbytes"=>0,
"queue_image_workers__moo_total_items"=>0,
"queue_image_workers__moo_expired_items"=>0,
"cmd_get"=>6426,
"version"=>"0.10.0",
"bytes_written"=>57473,
"cmd_set"=>8,
"queue_paperclip_background_upload_workers__upload_to_amazon_s3_age"=>127,
"queue_paperclip_background_upload_workers__upload_to_amazon_s3_logsize"=>
125,
"get_misses"=>6418,
"total_connections"=>16,
"curr_connections"=>1,
"curr_items"=>0,
"uptime"=>3311,
"queue_image_workers__moo_logsize"=>0,
"queue_paperclip_background_upload_workers__upload_to_amazon_s3_expired_ite
ms"=>
0,
"queue_paperclip_background_upload_workers__upload_to_amazon_s3_items"=>0,
"get_hits"=>8,
"total_items"=>8,
"rusage_system"=>0.02,
"rusage_user"=>0.09,
"queue_image_workers__moo_age"=>0,
"bytes_read"=>278448}}
Thread Stats:
[#<Thread:0xb721edfc sleep>, #<Thread:0xb7fb86fc run>]
** Exiting
/mnt/app/releases/20090823182205/vendor/rails/activerecord/lib/../../
active support/lib/active_support/vendor/memcache-client-1.5.0/
memcache.rb:697:in
`raise_on_error_response!': bad command line format
(MemCache::MemCacheError)
from
/mnt/app/releases/20090823182205/vendor/rails/activerecord/lib/../../
active support/lib/active_support/vendor/memcache-client-1.5.0/
memcache.rb:414:in
`flush_all'
from
/mnt/app/releases/20090823182205/vendor/rails/activerecord/lib/../../
active support/lib/active_support/vendor/memcache-client-1.5.0/
memcache.rb:408:in
`each'
from
/mnt/app/releases/20090823182205/vendor/rails/activerecord/lib/../../
active support/lib/active_support/vendor/memcache-client-1.5.0/
memcache.rb:408:in
`flush_all'
from
/mnt/app/releases/20090823182205/vendor/plugins/workling/lib/workling/
clien ts/memcache_queue_client.rb:44:in
`close'
from script/starling_status.rb:36
Reply all
Reply to author
Forward
0 new messages