Segfaulting in Bunny on Application Start after other errors

[Mike Harris]

To preface, our applications are failing to start because of something else that is going wrong, but when Passenger goes to shut down the worker process we're seeing a segmentation fault inside of Bunny. Once we work out our deployment issues (trying to put the apps on a new set of VMs) this should stop happening since this has all been working for a while on our old VMs but I thought it would be worth reporting the segfault since I would expect to see a regular exception or something instead.

Bunny: 2.14.3

Ruby: CRuby 2.5.1

Rails: 5.1.6.1

RabbitMQ: 3.7.10

We have this in our config.ru:

if defined?(PhusionPassenger)

PhusionPassenger.on_event(:starting_worker_process) do |forked|

SubscriberSetup.restart_subscribers if forked

end

PhusionPassenger.on_event(:stopping_worker_process) do

SubscriberSetup.stop_subscribers

end

end

The line cited as the segfault is https://github.com/ruby-amqp/bunny/blob/master/lib/bunny/consumer_work_pool.rb#L33

This is the SubscriberSetup class:

class SubscriberSetup

cattr_accessor :active_subscribers, instance_accessor: false do

[]

end

def self.start_subscribers

Dir[Rails.root.join('app/messengers/*_subscriber.rb')].each do |file|

klass = file.split('/').last.chomp('.rb').classify.safe_constantize

next if klass.nil?

subs = klass.new

subs.subscribe

active_subscribers << subs

end

rescue StandardError => exception

Rails.logger.info exception.message

end

def self.stop_subscribers

s = active_subscribers.first

s.exchange.channel.connection.close(false) unless s.nil?

SubscriberSetup.active_subscribers = []

rescue Timeout::Error

Rails.logger.warn 'Timed out closing old subscriber connections'

end

def self.restart_subscribers

stop_subscribers

start_subscribers

end

end

I'm suspicious of the way we are closing the connection here.

App 14757 output: /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/consumer_work_pool.rb:33:

App 14757 output: [BUG] Segmentation fault at 0x00007f4a5cececd5

App 14757 output: ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux]

App 14757 output:

App 14757 output: -- Control frame information -----------------------------------------------

App 14757 output: c:0031 p:---- s:0149 e:000148 CFUNC

App 14757 output:  :push

App 14757 output: c:0030 p:0018 s:0144 e:000143 METHOD /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/consumer_work_pool.rb:33

App 14757 output: c:0029 p:0006 s:0138 E:0024f0 BLOCK  /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/consumer_work_pool.rb:65 [FINISH]

App 14757 output: c:0028 p:---- s:0135 e:000134 CFUNC  :times

App 14757 output: c:0027 p:0022 s:0131 E:0006d0 METHOD /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/consumer_work_pool.rb:64

App 14757 output: c:0026 p:0024 s:0125 e:000124 METHOD /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/channel.rb:252

App 14757 output: c:0025 p:0006 s:0121 e:000120 BLOCK  /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/session.rb:571

App 14757 output: c:0024 p:0030 s:0118 E:001b28

App 14757 output: BLOCK  /usr/local/rbenv/versions/2.5.1/lib/ruby/2.5.0/timeout.rb:93 [FINISH]

App 14757 output: c:0023 p:0094 s:0112 E:002450 METHOD /usr/local/rbenv/versions/2.5.1/lib/ruby/2.5.0/timeout.rb:103

App 14757 output: c:0022 p:0027 s:0100 E:0005b0 BLOCK  /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/session.rb:571 [FINISH]

App 14757 output: c:0021 p:---- s:0095 e:000094 CFUNC  :each

App 14757 output: c:0020 p:0012 s:0091 E:0024d0 BLOCK  /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/session.rb:570

App 14757 output:

App 14757 output: c:0019 p:0007 s:0088 e:000087 METHOD /usr/local/rbenv/versions/2.5.1/lib/ruby/2.5.0/monitor.rb:226

App 14757 output: c:0018 p:0007 s:0084 E:002490 METHOD /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/session.rb:569

App 14757 output: c:0017 p:0022 s:0080 e:000079 BLOCK  /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/session.rb:379 [FINISH]

App 14757 output: c:0016 p:0006 s:0077 e:000076 METHOD /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/session.rb:1439

App 14757 output: c:0015 p:0017 s:0071 E:000548 METHOD

App 14757 output:  /srv/workflowdb/shared/bundle/ruby/2.5.0/gems/bunny-2.14.3/lib/bunny/session.rb:376

App 14757 output: c:0014 p:0032 s:0066 e:000065 METHOD /srv/workflowdb/releases/20200122134140/app/services/subscriber_setup.rb:22

App 14757 output: c:0013 p:0004 s:0061 e:000060 METHOD /srv/workflowdb/releases/20200122134140/app/services/subscriber_setup.rb:29

App 14757 output: c:0012 p:0015 s:0057 e:000056 BLOCK  config.ru:7 [FINISH]

App 14757 output: c:0011 p:0010 s:0053 e:000052 BLOCK  /usr/share/ruby/vendor_ruby/phusion_passenger/public_api.rb:41 [FINISH]

App 14757 output: c:0010 p:---- s:0049 e:000048 CFUNC  :each

App 14757 output: c:0009 p:0010 s:0045 e:000044 METHOD /usr/share/ruby/vendor_ruby/phusion_passenger/public_api.rb:40

App 14757 output: c:0008 p:0199 s:0039 e:000038 METHOD /usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:326

App 14757 output: c:0007

App 14757 output: p:0017 s:0032 e:000031 BLOCK

App 14757 output:  /usr/share/passenger/helper-scripts/rack-preloader.rb:159

App 14757 output: c:0006 p:0015 s:0029 e:000028 METHOD /usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:380

App 14757 output: c:0005

App 14757 output: p:0063

App 14757 output: s:0022

App 14757 output: e:000021

App 14757 output: METHOD

App 14757 output:  /usr/share/passenger/helper-scripts/rack-preloader.rb:158

App 14757 output:

App 14757 output: c:0004

App 14757 output: p:0229

... every file loaded at the time which I can provide if needed.

[Michael Klishin]

Hi Mike,

Bunny has no native code so it cannot be held responsible for a segmentation fault. I'm afraid I don't have the spare cycles to troubleshoot

a segfault in Passenger or CRuby in general.

Have you tried a more recent Ruby version?

[Mike Harris]

It's strange that the stack trace was pointing to a line in the bunny library but if there's no native code then maybe it's actually Passenger seg faulting.

This isn't actually a blocker for us, making our app start properly on the new VMs got rid of the segfault, I was just thinking I would bring this up in case there was something that could be done to throw a regular ruby exception instead of a segfault. If this can't have actually come from bunny then never mind.