Trying to debug Sidekiq hanging and frozen processes

76 views

Skip to first unread message

Jeffrey Krause

unread,

Oct 9, 2015, 7:05:11 PM10/9/15

to Sidekiq

Sidekiq has been awesome for us. I'm just trying to figure out how to debug processes that hang our server, this happens several times a day, and normally I just stop and restart sidekiq.

I currently have 12 threads running, chopping video, making thumbnails, etc.

I'm not sure if it's a mysql issue, a redis issue, a gem issue, or just bad code on my part. Any insights are appreciated.

Here is the output of kill -TTIN <pid> (if it helps)

2015-10-09 22:41:59 +0000: < {"_index":"interactions","_type":"interaction","_id":"348102","_version":1,"created":true}

2015-10-09 22:41:59 +0000: PUT http://10.240.239.230:9200/interactions/interaction/348103 [status:201, request:0.003s, query:n/a]

2015-10-09 22:41:59 +0000: > {"id":348103,"video_id":183018,"interaction_type":"treechopper","transcript":null,"start_time":1200.0,"end_time":1350.0,"clip_url":"183018_480_480-150-8.mp4","thumb_url":null,"team_id":null,"primary_category":null,"uuid":"93eb6ca3-2f38-492b-8566-203a

0473dc1f","clip_length":150.0,"video_title":"Community Episode 1: Ladders | Community Season Six - Yahoo Screen","video_name":"183018_480_480.mp4","people_tags":[],"source_url":"https://screen.yahoo.com/community/ladders-070001899.html"}

2015-10-09 22:41:59 +0000: < {"_index":"interactions","_type":"interaction","_id":"348103","_version":1,"created":true}

2015-10-09 22:42:00 +0000: PUT http://10.240.239.230:9200/interactions/interaction/348104 [status:201, request:0.003s, query:n/a]

2015-10-09 22:42:00 +0000: > {"id":348104,"video_id":183018,"interaction_type":"treechopper","transcript":null,"start_time":1350.0,"end_time":1500.0,"clip_url":"183018_480_480-150-9.mp4","thumb_url":null,"team_id":null,"primary_category":null,"uuid":"cb478db4-50e0-4b5b-a072-ea0a

0f8cd589","clip_length":150.0,"video_title":"Community Episode 1: Ladders | Community Season Six - Yahoo Screen","video_name":"183018_480_480.mp4","people_tags":[],"source_url":"https://screen.yahoo.com/community/ladders-070001899.html"}

2015-10-09 22:42:00 +0000: < {"_index":"interactions","_type":"interaction","_id":"348104","_version":1,"created":true}

2015-10-09 22:42:00 +0000: PUT http://10.240.239.230:9200/interactions/interaction/348105 [status:201, request:0.003s, query:n/a]

2015-10-09 22:42:00 +0000: > {"id":348105,"video_id":183018,"interaction_type":"treechopper","transcript":null,"start_time":1500.0,"end_time":1650.0,"clip_url":"183018_480_480-150-10.mp4","thumb_url":null,"team_id":null,"primary_category":null,"uuid":"f4af0306-59e5-4ddb-a48c-83a

ba92ad8e2","clip_length":150.0,"video_title":"Community Episode 1: Ladders | Community Season Six - Yahoo Screen","video_name":"183018_480_480.mp4","people_tags":[],"source_url":"https://screen.yahoo.com/community/ladders-070001899.html"}

2015-10-09 22:42:00 +0000: < {"_index":"interactions","_type":"interaction","_id":"348105","_version":1,"created":true}

2015-10-09T22:49:56.576Z 15546 TID-39eb8 WARN: Thread TID-39eb8

2015-10-09T22:49:56.582Z 15546 TID-39eb8 WARN: /home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/sidekiq-3.3.4/lib/sidekiq/cli.rb:138:in `backtrace'

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/sidekiq-3.3.4/lib/sidekiq/cli.rb:138:in `block in handle_signal'

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/sidekiq-3.3.4/lib/sidekiq/cli.rb:135:in `each'

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/sidekiq-3.3.4/lib/sidekiq/cli.rb:135:in `handle_signal'

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/sidekiq-3.3.4/lib/sidekiq/cli.rb:88:in `run'

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/sidekiq-3.3.4/bin/sidekiq:8:in `<top (required)>'

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/bin/sidekiq:23:in `load'

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/bin/sidekiq:23:in `<main>'

2015-10-09T22:49:56.582Z 15546 TID-39eb8 WARN: Thread TID-5tfwo

2015-10-09T22:49:56.585Z 15546 TID-39eb8 WARN: /home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/mailbox.rb:62:in `sleep'

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/mailbox.rb:62:in `wait'

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/mailbox.rb:62:in `block in check'

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/timers-4.0.1/lib/timers/wait.rb:14:in `for'

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/mailbox.rb:57:in `check'

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/actor.rb:153:in `block in run'

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/timers-4.0.1/lib/timers/group.rb:52:in `wait'

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/actor.rb:150:in `run'

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/actor.rb:130:in `block in start'

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/thread_handle.rb:13:in `block in initialize'

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/actor_system.rb:32:in `block in get_thread'

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/internal_pool.rb:130:in `call'

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/internal_pool.rb:130:in `block in create'

2015-10-09T22:49:56.585Z 15546 TID-39eb8 WARN: Thread TID-5uhgc

/home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/activejob-4.2.1/lib/active_job/logging.rb:23:in `block (4 levels) in <module:Logging>'

2015-10-09T22:49:56.586Z 15546 TID-39eb8 WARN: /home/jeffrey/sites/video/vendor/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/mailbox.rb:62:in `sleep'