Hang of process after network failure during read data from mongodb socket

65 views
Skip to first unread message

Eugene Batogov

unread,
Nov 11, 2013, 10:56:07 AM11/11/13
to mon...@googlegroups.com
Hi
I'm using mondoid-3.1.5 and Resque.
When my resque worker go to mongodb and happen network failure during:
moped/connection.rb:177:in `read_data'
then worker hang forever, while I don't invoke kill -9.
Can I tune mongoid and moped for normal exit from this situation ?
Full log: https://gist.github.com/JohnBat26/7412849/raw/8e9d9910077073fb4cc958b76f97d3d07f8d6f3e/gistfile1.txt
Issue: https://github.com/mongoid/mongoid/issues/3391

Jon

unread,
Mar 1, 2014, 9:59:53 AM3/1/14
to mon...@googlegroups.com
I just started getting this, too. Dozens of my Sidekiq workers got "stuck" and when I printed out the backtraces of the threads, it's all Moped. Processes are stuck for hours not doing anything until I kill -9 them:

Has anyone else seen this? What kind of workarounds have you done? It's happening at various Mongoid calls in my code.

  •  worker-prd-iad-ded-06 production.log: [ 2:21:58.998520585] (22221) [INFO] /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/sockets/connectable.rb:46:in `read'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/sockets/connectable.rb:46:in `block in read'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/sockets/connectable.rb:118:in `handle_socket_errors'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/sockets/connectable.rb:46:in `read'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/connection.rb:177:in `read_data'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/connection.rb:99:in `block in read'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/connection.rb:202:in `with_connection'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/connection.rb:97:in `read'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/protocol/query.rb:163:in `receive_replies'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/connection.rb:135:in `block in receive_replies'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/connection.rb:134:in `map'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/connection.rb:134:in `receive_replies'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/node.rb:617:in `block (2 levels) in flush'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/node.rb:153:in `ensure_connected'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/node.rb:615:in `block in flush'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/node.rb:630:in `logging'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/gems/newrelic_moped-0.0.8/lib/newrelic_moped/instrumentation.rb:40:in `block in logging_with_newrelic_trace'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/gems/newrelic_moped-0.0.8/lib/newrelic_moped/instrumentation.rb:60:in `call'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/gems/newrelic_moped-0.0.8/lib/newrelic_moped/instrumentation.rb:60:in `logging_with_newrelic_trace'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/node.rb:614:in `flush'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/node.rb:603:in `process'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/node.rb:92:in `command'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/node.rb:436:in `refresh'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/cluster.rb:171:in `block in refresh'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/cluster.rb:185:in `each'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/cluster.rb:185:in `refresh'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/cluster.rb:135:in `nodes'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/cluster.rb:247:in `with_secondary'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/session/context.rb:104:in `with_node'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/cursor.rb:137:in `load_docs'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/cursor.rb:25:in `each'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/query.rb:76:in `each'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/bundler/gems/moped-6af3788a1f35/lib/moped/query.rb:76:in `each'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/gems/mongoid-3.1.6/lib/mongoid/contextual/mongo.rb:122:in `block in each'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/gems/mongoid-3.1.6/lib/mongoid/contextual/mongo.rb:619:in `selecting'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/gems/mongoid-3.1.6/lib/mongoid/contextual/mongo.rb:121:in `each'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/gems/mongoid-3.1.6/lib/mongoid/contextual.rb:19:in `each'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/gems/mongoid-3.1.6/lib/mongoid/criterion/findable.rb:125:in `entries'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/gems/mongoid-3.1.6/lib/mongoid/criterion/findable.rb:125:in `from_database'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/gems/mongoid-3.1.6/lib/mongoid/criterion/findable.rb:93:in `multiple_from_map_or_db'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/gems/mongoid-3.1.6/lib/mongoid/criterion/findable.rb:19:in `execute_or_raise'
  •  worker-prd-iad-ded-06 production.log: /var/www/dashboard/shared/bundle/ruby/2.0.0/gems/mongoid-3.1.6/lib/mongoid/criterion/findable.rb:40:in `find'

Jon

unread,
Mar 2, 2014, 11:17:46 PM3/2/14
to mon...@googlegroups.com
It seems this might have been upstream firewall changes timing out outbound connections on idle workers. Not sure why Moped never timed out in the process, but in my specific case I think we figured it out.
Reply all
Reply to author
Forward
0 new messages