Segfaulting in Bunny on Application Start after other errors

25 views
Skip to first unread message

Mike Harris

unread,
Jan 28, 2020, 8:58:56 AM1/28/20
to Ruby RabbitMQ libraries
To preface, our applications are failing to start because of something else that is going wrong, but when Passenger goes to shut down the worker process we're seeing a segmentation fault inside of Bunny. Once we work out our deployment issues (trying to put the apps on a new set of VMs) this should stop happening since this has all been working for a while on our old VMs but I thought it would be worth reporting the segfault since I would expect to see a regular exception or something instead.

Bunny: 2.14.3
Ruby: CRuby 2.5.1
Rails: 5.1.6.1
RabbitMQ: 3.7.10

We have this in our config.ru:
if defined?(PhusionPassenger)
PhusionPassenger.on_event(:starting_worker_process) do |forked|
SubscriberSetup.restart_subscribers if forked
end

PhusionPassenger.on_event(:stopping_worker_process) do
SubscriberSetup.stop_subscribers
end
end


This is the SubscriberSetup class:
class SubscriberSetup
cattr_accessor :active_subscribers, instance_accessor: false do
[]
end
def self.start_subscribers
Dir[Rails.root.join('app/messengers/*_subscriber.rb')].each do |file|
klass = file.split('/').last.chomp('.rb').classify.safe_constantize
next if klass.nil?

subs = klass.new
subs.subscribe
active_subscribers << subs
end
rescue StandardError => exception
Rails.logger.info exception.message
end

def self.stop_subscribers
s = active_subscribers.first
s.exchange.channel.connection.close(false) unless s.nil?
SubscriberSetup.active_subscribers = []
rescue Timeout::Error
Rails.logger.warn 'Timed out closing old subscriber connections'
end

def self.restart_subscribers
stop_subscribers
start_subscribers
end
end

I'm suspicious of the way we are closing the connection here.

App 14757 output: /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/consumer_work_pool.rb:33:
App 14757 output: [BUG] Segmentation fault at 0x00007f4a5cececd5
App 14757 output: ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux]
App 14757 output:
App 14757 output: -- Control frame information -----------------------------------------------
App 14757 output: c:0031 p:---- s:0149 e:000148 CFUNC
App 14757 output:  :push
App 14757 output: c:0030 p:0018 s:0144 e:000143 METHOD /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/consumer_work_pool.rb:33
App 14757 output: c:0029 p:0006 s:0138 E:0024f0 BLOCK  /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/consumer_work_pool.rb:65 [FINISH]
App 14757 output: c:0028 p:---- s:0135 e:000134 CFUNC  :times
App 14757 output: c:0027 p:0022 s:0131 E:0006d0 METHOD /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/consumer_work_pool.rb:64
App 14757 output: c:0026 p:0024 s:0125 e:000124 METHOD /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/channel.rb:252
App 14757 output: c:0025 p:0006 s:0121 e:000120 BLOCK  /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/session.rb:571
App 14757 output: c:0024 p:0030 s:0118 E:001b28
App 14757 output: BLOCK  /usr/local/rbenv/versions/2.5.1/lib/ruby/2.5.0/timeout.rb:93 [FINISH]
App 14757 output: c:0023 p:0094 s:0112 E:002450 METHOD /usr/local/rbenv/versions/2.5.1/lib/ruby/2.5.0/timeout.rb:103
App 14757 output: c:0022 p:0027 s:0100 E:0005b0 BLOCK  /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/session.rb:571 [FINISH]
App 14757 output: c:0021 p:---- s:0095 e:000094 CFUNC  :each
App 14757 output: c:0020 p:0012 s:0091 E:0024d0 BLOCK  /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/session.rb:570
App 14757 output:
App 14757 output: c:0019 p:0007 s:0088 e:000087 METHOD /usr/local/rbenv/versions/2.5.1/lib/ruby/2.5.0/monitor.rb:226
App 14757 output: c:0018 p:0007 s:0084 E:002490 METHOD /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/session.rb:569
App 14757 output: c:0017 p:0022 s:0080 e:000079 BLOCK  /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/session.rb:379 [FINISH]
App 14757 output: c:0016 p:0006 s:0077 e:000076 METHOD /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/session.rb:1439
App 14757 output: c:0015 p:0017 s:0071 E:000548 METHOD
App 14757 output:  /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/session.rb:376
App 14757 output: c:0014 p:0032 s:0066 e:000065 METHOD /srv/workflowdb/releases/20200122134140/app/services/subscriber_setup.rb:22
App 14757 output: c:0013 p:0004 s:0061 e:000060 METHOD /srv/workflowdb/releases/20200122134140/app/services/subscriber_setup.rb:29
App 14757 output: c:0012 p:0015 s:0057 e:000056 BLOCK  config.ru:7 [FINISH]
App 14757 output: c:0011 p:0010 s:0053 e:000052 BLOCK  /usr/share/ruby/vendor_ruby/phusion_passenger/public_api.rb:41 [FINISH]
App 14757 output: c:0010 p:---- s:0049 e:000048 CFUNC  :each
App 14757 output: c:0009 p:0010 s:0045 e:000044 METHOD /usr/share/ruby/vendor_ruby/phusion_passenger/public_api.rb:40
App 14757 output: c:0008 p:0199 s:0039 e:000038 METHOD /usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:326
App 14757 output: c:0007
App 14757 output: p:0017 s:0032 e:000031 BLOCK
App 14757 output:  /usr/share/passenger/helper-scripts/rack-preloader.rb:159
App 14757 output: c:0006 p:0015 s:0029 e:000028 METHOD /usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:380
App 14757 output: c:0005
App 14757 output: p:0063
App 14757 output: s:0022
App 14757 output: e:000021
App 14757 output: METHOD
App 14757 output:  /usr/share/passenger/helper-scripts/rack-preloader.rb:158
App 14757 output:
App 14757 output: c:0004
App 14757 output: p:0229

... every file loaded at the time which I can provide if needed.

Michael Klishin

unread,
Jan 28, 2020, 9:46:25 AM1/28/20
to Ruby RabbitMQ libraries
Hi Mike,

Bunny has no native code so it cannot be held responsible for a segmentation fault. I'm afraid I don't have the spare cycles to troubleshoot
a segfault in Passenger or CRuby in general.

Have you tried a more recent Ruby version?

Mike Harris

unread,
Jan 28, 2020, 11:22:15 AM1/28/20
to Ruby RabbitMQ libraries
It's strange that the stack trace was pointing to a line in the bunny library but if there's no native code then maybe it's actually Passenger seg faulting.

This isn't actually a blocker for us, making our app start properly on the new VMs got rid of the segfault, I was just thinking I would bring this up in case there was something that could be done to throw a regular ruby exception instead of a segfault. If this can't have actually come from bunny then never mind.
Reply all
Reply to author
Forward
0 new messages