Segmentation Fault on "require 'pg'"

1,252 views

Skip to first unread message

BeeRich33

unread,

Nov 10, 2016, 3:10:12 PM11/10/16

to ruby-pg

Segmentation faults all over the place. Narrowed it down to my app (Sinatra) requiring 'pg'.

Ruby 2.3.1

Sinatra on Passenger

pg 0.19.0

All gems updated

The same code works in IRB, but not in my Sinatra route as a post. It worked before, and not now. Seg Faults were coming up all over the place and this is how simple I've narrowed it down.

BTW this group doesn't look very active. Hopefully someone is browsing. Any advice appreciated. Cheers

[ 2016-11-10 15:01:52.0885 29145/0x700000290000 age/Cor/CoreMain.cpp:819 ]: Checking whether to disconnect long-running connections for process 65287, application /Library/WebServer/Documents/usersys/public (development)
App 65296 stdout: 
App 65374 stdout: 
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/pg-0.19.0/lib/pg_ext.bundle: [BUG] 
App 65296 stderr: Segmentation fault
App 65296 stderr: ruby 2.0.0p648 (2015-12-16 revision 53162) [universal.x86_64-darwin15]
App 65296 stderr: 
App 65296 stderr: -- Crash Report log information --------------------------------------------
App 65296 stderr:    See Crash Report log file under the one of following:
App 65296 stderr:      * ~/Library/Logs/CrashReporter
App 65296 stderr:      * /Library/Logs/CrashReporter
App 65296 stderr:      * ~/Library/Logs/DiagnosticReports
App 65296 stderr:      * /Library/Logs/DiagnosticReports
App 65296 stderr:    the more detail of.
App 65296 stderr: 
App 65296 stderr: -- Control frame information -----------------------------------------------
App 65296 stderr: c:0057 p:-17543180344806 s:0278 e:000277 TOP    [FINISH]
App 65296 stderr: c:0056 p:---- s:0276 e:000275 CFUNC  :require
App 65296 stderr: c:0055 p:0009 s:0272 e:000271 TOP    /Users/rich/.rvm/gems/ruby-2.3.1/gems/pg-0.19.0/lib/pg.rb:4 [FINISH]
App 65296 stderr: c:0054 p:---- s:0268 e:000267 CFUNC  :require
App 65296 stderr: c:0053 p:0016 s:0264 e:000263 LAMBDA /Library/WebServer/Documents/usersys/app.rb:48 [FINISH]
App 65296 stderr: c:0052 p:---- s:0262 e:000261 CFUNC  :call
App 65296 stderr: c:0051 p:0012 s:0259 e:000258 BLOCK  /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611 [FINISH]
App 65296 stderr: c:0050 p:---- s:0255 e:000254 CFUNC  :[]
App 65296 stderr: c:0049 p:0011 s:0250 e:000249 BLOCK  /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:975
App 65296 stderr: c:0048 p:0009 s:0248 e:000245 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:994
App 65296 stderr: c:0047 p:0023 s:0243 e:000242 BLOCK  /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:975
App 65296 stderr: c:0046 p:0029 s:0240 e:000239 BLOCK  /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1015 [FINISH]
App 65296 stderr: c:0045 p:---- s:0238 e:000237 CFUNC  :catch
App 65296 stderr: c:0044 p:0132 s:0234 e:000233 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013
App 65296 stderr: c:0043 p:0013 s:0223 e:000222 BLOCK  /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:973 [FINISH]
App 65296 stderr: c:0042 p:---- s:0216 e:000215 CFUNC  :each
App 65296 stderr: c:0041 p:0034 s:0213 e:000212 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:972
App 65296 stderr: c:0040 p:0042 s:0207 e:000206 BLOCK  /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1085
App 65296 stderr: c:0039 p:0006 s:0205 e:000204 BLOCK  /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067 [FINISH]
App 65296 stderr: c:0038 p:---- s:0203 e:000202 CFUNC  :catch
App 65296 stderr: c:0037 p:0009 s:0199 
App 65296 stderr: e:000198 
App 65296 stderr: METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067
App 65296 stderr: c:0036 p:0007 s:0195 e:000194 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1082
App 65296 stderr: c:0035 p:0007 s:0191 
App 65296 stderr: e:000190 
App 65296 stderr: BLOCK  /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:907
App 65296 stderr: c:0034 p:0006 s:0189 e:000188 BLOCK  /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067 [FINISH]
App 65296 stderr: c:0033 
App 65296 stderr: p:---- s:0187 e:000186 CFUNC  :catch
App 65296 stderr: c:0032 p:0009 s:0183 e:000182 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067
App 65296 stderr: c:0031 p:0092 s:0179 e:000178 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:907
App 65296 stderr: c:0030 p:0011 s:0175 e:000174 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:895
App 65296 stderr: c:0029 p:0023 s:0171 e:000170 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:225
App 65296 stderr: c:0028 p:0009 s:0163 e:000162 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:220
App 65296 stderr: c:0027 p:0011 s:0159 e:000158 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18
App 65296 stderr: c:0026 p:0043 s:0152 e:000151 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49
App 65296 stderr: c:0025 p:0043 s:0147 e:000146 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49
App 65296 stderr: c:0024 p:0047 s:0142 e:000141 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16
App 65296 stderr: c:0023 p:0026 s:0137 e:000136 
App 65296 stderr: METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18
App 65296 stderr: c:0022 p:0043 s:0129 e:000128 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49
App 65296 stderr: c:0021 p:0043 s:0124 e:000123 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49
App 65296 stderr: c:0020 p:0011 s:0119 e:000118 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31
App 65296 stderr: c:0019 p:0023 s:0112 e:000111 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:225
App 65296 stderr: c:0018 p:0009 s:0104 e:000103 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:220
App 65296 stderr: c:0017 p:0054 s:0100 e:000099 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/logger.rb:15
App 65296 stderr: c:0016 p:0024 s:0095 e:000094 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/commonlogger.rb:33
App 65296 stderr: c:0015 p:0020 s:0087 e:000086 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:219
App 65296 stderr: c:0014 p:0029 s:0083 e:000082 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:212
App 65296 stderr: c:0013 p:0011 
App 65296 stderr: s:0079 e:000078 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/head.rb:13
App 65296 stderr: c:0012 p:0011 s:0072 e:000071 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/show_exceptions.rb:25
App 65296 stderr: c:0011 p:0011 s:0063 e:000062 
App 65296 stderr: METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:182
App 65296 stderr: c:0010 p:0011 s:0057 e:000056 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:2013
App 65296 stderr: c:0009 p:0011 s:0053 e:000052 
App 65296 stderr: BLOCK  /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487
App 65296 stderr: c:0008 p:0023 s:0051 e:000050 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1787
App 65296 stderr: c:0007 p:0007 s:0047 E:000240 METHOD /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487
App 65296 stderr: c:0006 p:0340 s:0043 
App 65296 stderr: E:0010b0 METHOD /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/rack/thread_handler_extension.rb:97
App 65296 stderr: c:0005 p:0283 
App 65296 stderr: s:0027 e:000026 METHOD
App 65296 stderr:  /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:160
App 65296 stderr: c:0004 p:0092 s:0017 e:000016 METHOD /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:113
App 65296 stderr: c:0003 p:0059 s:0009 e:000008 BLOCK  /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/request_handler.rb:416
App 65296 stderr: c:0002 p:0026 s:0005 e:000004 BLOCK  /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/utils.rb:113 [FINISH]
App 65296 stderr: c:0001 p:---- s:0002 e:000001 TOP   
App 65296 stderr:  [FINISH]
App 65296 stderr: 
App 65296 stderr: /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'
App 65296 stderr: /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'
App 65296 stderr: /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:113:in `main_loop'
App 65296 stderr: /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:160:in `accept_and_process_next_request'
App 65296 stderr: /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1787:in `synchronize'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `block in call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:2013:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:182:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/show_exceptions.rb:25:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:212:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:219:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/commonlogger.rb:33:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/logger.rb:15:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:220:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:225:in `context'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:220:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:225:in `context'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:895:in `call'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `call!'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `block in call!'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1082:in `dispatch!'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1085:in `block in dispatch!'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `route!'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `each'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:973:in `block in route!'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `process_route'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `catch'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1015:in `block in process_route'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (2 levels) in route!'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:994:in `route_eval'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (3 levels) in route!'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `[]'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `block in compile!'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `call'
App 65296 stderr: /Library/WebServer/Documents/usersys/app.rb:48:in `block in <class:MyApp>'
App 65296 stderr: /Library/WebServer/Documents/usersys/app.rb:48:in `require'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/pg-0.19.0/lib/pg.rb:4:in `<top (required)>'
App 65296 stderr: /Users/rich/.rvm/gems/ruby-2.3.1/gems/pg-0.19.0/lib/pg.rb:4:in `require'
App 65296 stderr: 
App 65296 stderr: -- C level backtrace information -------------------------------------------
App 65296 stderr: 
App 65296 stderr: -- Other runtime information -----------------------------------------------
App 65296 stderr: 
App 65296 stderr: * Loaded script: Passenger RubyApp: /Library/WebServer/Documents/usersys/public (development)
App 65296 stderr: 
App 65296 stderr: * Loaded features:
App 65296 stderr: 
App 65296 stderr:     0 enumerator.so
App 65296 stderr:     1 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/universal-darwin15/enc/encdb.bundle
App 65296 stderr:     2 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/universal-darwin15/enc/trans/transdb.bundle
App 65296 stderr:     3 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/universal-darwin15/rbconfig.rb
App 65296 stderr:     4 /Library/Ruby/Site/2.0.0/rubygems/compatibility.rb
App 65296 stderr:     5 /Library/Ruby/Site/2.0.0/rubygems/defaults.rb
App 65296 stderr:     6 /Library/Ruby/Site/2.0.0/rubygems/deprecate.rb
App 65296 stderr:     7 /Library/Ruby/Site/2.0.0/rubygems/errors.rb
App 65296 stderr:     8 /Library/Ruby/Site/2.0.0/rubygems/version.rb
App 65296 stderr:     9 /Library/Ruby/Site/2.0.0/rubygems/requirement.rb
App 65296 stderr:    10 /Library/Ruby/Site/2.0.0/rubygems/platform.rb
App 65296 stderr:    11 /Library/Ruby/Site/2.0.0/rubygems/basic_specification.rb
App 65296 stderr:    12 /Library/Ruby/Site/2.0.0/rubygems/stub_specification.rb
App 65296 stderr:    13 /Library/Ruby/Site/2.0.0/rubygems/util/stringio.rb
App 65296 stderr:    14 /Library/Ruby/Site/2.0.0/rubygems/specification.rb
App 65296 stderr:    15 /Library/Ruby/Site/2.0.0/rubygems/exceptions.rb
App 65296 stderr:    16 /Library/Ruby/Site/2.0.0/rubygems/core_ext/kernel_gem.rb
App 65296 stderr:    17 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/thread.rb
App 65296 stderr:    18 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/monitor.rb
App 65296 stderr:    19 /Library/Ruby/Site/2.0.0/rubygems/core_ext/kernel_require.rb
App 65296 stderr:    20 /Library/Ruby/Site/2.0.0/rubygems.rb
App 65296 stderr:    21 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger.rb
App 65296 stderr:    22 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/constants.rb
App 65296 stderr:    23 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/universal-darwin15/etc.bundle
App 65296 stderr:    24 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/fileutils.rb
App 65296 stderr:    25 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/tmpdir.rb
App 65296 stderr:    26 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/utils/tmpio.rb
App 65296 stderr:    27 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/platform_info.rb
App 65296 stderr:    28 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/platform_info/operating_system.rb
App 65296 stderr:    29 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/platform_info/ruby.rb
App 65296 stderr:    30 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/platform_info/binary_compatibility.rb
App 65296 stderr:    31 /usr/local/Cellar/passenger/5.0.30_1/libexec/buildout/ruby/ruby-2.0.0-universal-macosx/passenger_native_support.bundle
App 65296 stderr:    32 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/native_support.rb
App 65296 stderr:    33 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/universal-darwin15/socket.bundle
App 65296 stderr:    34 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/socket.rb
App 65296 stderr:    35 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/universal-darwin15/pathname.bundle
App 65296 stderr:    36 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/pathname.rb
App 65296 stderr:    37 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/ruby_core_enhancements.rb
App 65296 stderr:    38 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/ruby_core_io_enhancements.rb
App 65296 stderr:    39 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/base64.rb
App 65296 stderr:    40 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/utils.rb
App 65296 stderr:    41 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/preloader_shared_helpers.rb
App 65296 stderr:    42 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/public_api.rb
App 65296 stderr:    43 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/debug_logging.rb
App 65296 stderr:    44 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/shellwords.rb
App 65296 stderr:    45 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/utils/shellwords.rb
App 65296 stderr:    46 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/loader_shared_helpers.rb
App 65296 stderr:    47 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/universal-darwin15/fcntl.bundle
App 65296 stderr:    48 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/message_channel.rb
App 65296 stderr:    49 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/message_client.rb
App 65296 stderr:    50 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/utils/native_support_utils.rb
App 65296 stderr:    51 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/utils/unseekable_socket.rb
App 65296 stderr:    52 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb
App 65296 stderr:    53 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/request_handler.rb
App 65296 stderr:    54 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/universal-darwin15/stringio.bundle
App 65296 stderr:    55 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/utils/tee_input.rb
App 65296 stderr:    56 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/rack/thread_handler_extension.rb
App 65296 stderr:    57 /Library/Ruby/Site/2.0.0/rubygems/path_support.rb
App 65296 stderr:    58 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/vendor/union_station_hooks_core/lib/union_station_hooks_core/version.rb
App 65296 stderr:    59 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/vendor/union_station_hooks_core/lib/union_station_hooks_core.rb
App 65296 stderr:    60 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/vendor/union_station_hooks_rails/lib/union_station_hooks_rails/version.rb
App 65296 stderr:    61 /usr/local/Cellar/passenger/5.0.30_1/libexec/src/ruby_supportlib/phusion_passenger/vendor/union_station_hooks_rails/lib/union_station_hooks_rails.rb
App 65296 stderr:    62 /Library/Ruby/Site/2.0.0/rubygems/dependency.rb
App 65296 stderr:    63 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/postit_trampoline.rb
App 65296 stderr:    64 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/constants.rb
App 65296 stderr:    65 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/universal-darwin15/io/console.bundle
App 65296 stderr:    66 /Library/Ruby/Site/2.0.0/rubygems/user_interaction.rb
App 65296 stderr:    67 /Library/Ruby/Site/2.0.0/rubygems/config_file.rb
App 65296 stderr:    68 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/rubygems_integration.rb
App 65296 stderr:    69 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/current_ruby.rb
App 65296 stderr:    70 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/shared_helpers.rb
App 65296 stderr:    71 /Library/Ruby/Site/2.0.0/rubygems/ext/build_error.rb
App 65296 stderr:    72 /Library/Ruby/Site/2.0.0/rubygems/ext/builder.rb
App 65296 stderr:    73 /Library/Ruby/Site/2.0.0/rubygems/ext/configure_builder.rb
App 65296 stderr:    74 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/delegate.rb
App 65296 stderr:    75 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/tempfile.rb
App 65296 stderr:    76 /Library/Ruby/Site/2.0.0/rubygems/ext/ext_conf_builder.rb
App 65296 stderr:    77 /Library/Ruby/Site/2.0.0/rubygems/ext/rake_builder.rb
App 65296 stderr:    78 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/optparse.rb
App 65296 stderr:    79 /Library/Ruby/Site/2.0.0/rubygems/command.rb
App 65296 stderr:    80 /Library/Ruby/Site/2.0.0/rubygems/ext/cmake_builder.rb
App 65296 stderr:    81 /Library/Ruby/Site/2.0.0/rubygems/ext.rb
App 65296 stderr:    82 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/errors.rb
App 65296 stderr:    83 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/environment_preserver.rb
App 65296 stderr:    84 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/timeout.rb
App 65296 stderr:    85 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/protocol.rb
App 65296 stderr:    86 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/uri/common.rb
App 65296 stderr:    87 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/uri/generic.rb
App 65296 stderr:    88 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/uri/ftp.rb
App 65296 stderr:    89 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/uri/http.rb
App 65296 stderr:    90 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/uri/https.rb
App 65296 stderr:    91 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/uri/ldap.rb
App 65296 stderr:    92 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/uri/ldaps.rb
App 65296 stderr:    93 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/uri/mailto.rb
App 65296 stderr:    94 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/uri.rb
App 65296 stderr:    95 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/universal-darwin15/zlib.bundle
App 65296 stderr:    96 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/http/exceptions.rb
App 65296 stderr:    97 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/http/header.rb
App 65296 stderr:    98 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/http/generic_request.rb
App 65296 stderr:    99 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/http/request.rb
App 65296 stderr:   100 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/http/requests.rb
App 65296 stderr:   101 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/http/response.rb
App 65296 stderr:   102 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/http/responses.rb
App 65296 stderr:   103 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/http/proxy_delta.rb
App 65296 stderr:   104 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/http/backward.rb
App 65296 stderr:   105 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/http.rb
App 65296 stderr:   106 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/universal-darwin15/date_core.bundle
App 65296 stderr:   107 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/date/format.rb
App 65296 stderr:   108 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/date.rb
App 65296 stderr:   109 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/time.rb
App 65296 stderr:   110 /Library/Ruby/Site/2.0.0/rubygems/request.rb
App 65296 stderr:   111 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/cgi/core.rb
App 65296 stderr:   112 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/cgi/util.rb
App 65296 stderr:   113 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/cgi/cookie.rb
App 65296 stderr:   114 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/cgi.rb
App 65296 stderr:   115 /Library/Ruby/Site/2.0.0/rubygems/uri_formatter.rb
App 65296 stderr:   116 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/universal-darwin15/digest.bundle
App 65296 stderr:   117 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/digest.rb
App 65296 stderr:   118 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/universal-darwin15/openssl.bundle
App 65296 stderr:   119 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/openssl/bn.rb
App 65296 stderr:   120 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/openssl/cipher.rb
App 65296 stderr:   121 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/openssl/config.rb
App 65296 stderr:   122 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/openssl/digest.rb
App 65296 stderr:   123 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/openssl/x509.rb
App 65296 stderr:   124 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/openssl/buffering.rb
App 65296 stderr:   125 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/openssl/ssl.rb
App 65296 stderr:   126 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/openssl.rb
App 65296 stderr:   127 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/securerandom.rb
App 65296 stderr:   128 /S
App 65296 stderr: ystem/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/resolv.rb
App 65296 stderr:   129 /Library/Ruby/Site/2.0.0/rubygems/remote_fetcher.rb
App 65296 stderr:   130 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/gem_remote_fetcher.rb
App 65296 stderr:   131 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/universal-darwin15/digest/sha1.bundle
App 65296 stderr:   132 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/plugin/api/source.rb
App 65296 stderr:   133 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/plugin/api.rb
App 65296 stderr:   134 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/plugin.rb
App 65296 stderr:   135 /Library/Ruby/Site/2.0.0/rubygems/util.rb
App 65296 stderr:   136 /Library/Ruby/Site/2.0.0/rubygems/source/git.rb
App 65296 stderr:   137 /Library/Ruby/Site/2.0.0/rubygems/source/installed.rb
App 65296 stderr:   138 /Library/Ruby/Site/2.0.0/rubygems/source/specific_file.rb
App 65296 stderr:   139 /Library/Ruby/Site/2.0.0/rubygems/source/local.rb
App 65296 stderr:   140 /Library/Ruby/Site/2.0.0/rubygems/source/lock.rb
App 65296 stderr:   141 /Library/Ruby/Site/2.0.0/rubygems/source/vendor.rb
App 65296 stderr:   142 /Library/Ruby/Site/2.0.0/rubygems/source.rb
App 65296 stderr:   143 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/gem_helpers.rb
App 65296 stderr:   144 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/match_platform.rb
App 65296 stderr:   145 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/rubygems_ext.rb
App 65296 stderr:   146 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/version.rb
App 65296 stderr:   147 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler.rb
App 65296 stderr:   148 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/settings.rb
App 65296 stderr:   149 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/universal-darwin15/strscan.bundle
App 65296 stderr:   150 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/source.rb
App 65296 stderr:   151 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/source/path.rb
App 65296 stderr:   152 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/source/git.rb
App 65296 stderr:   153 /Library/Ruby/Site/2.0.0/rubygems/text.rb
App 65296 stderr:   154 /Library/Ruby/Site/2.0.0/rubygems/name_tuple.rb
App 65296 stderr:   155 /Library/Ruby/Site/2.0.0/rubygems/spec_fetcher.rb
App 65296 stderr:   156 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/source/rubygems.rb
App 65296 stderr:   157 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/lockfile_parser.rb
App 65296 stderr:   158 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/set.rb
App 65296 stderr:   159 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/definition.rb
App 65296 stderr:   160 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/dependency.rb
App 65296 stderr:   161 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/ruby_dsl.rb
App 65296 stderr:   162 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/dsl.rb
App 65296 stderr:   163 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/source_list.rb
App 65296 stderr:   164 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/lazy_specification.rb
App 65296 stderr:   165 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/tsort.rb
App 65296 stderr:   166 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/forwardable.rb
App 65296 stderr:   167 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/spec_set.rb
App 65296 stderr:   168 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/gem_version_promoter.rb
App 65296 stderr:   169 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/source/gemspec.rb
App 65296 stderr:   170 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/runtime.rb
App 65296 stderr:   171 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/dep_proxy.rb
App 65296 stderr:   172 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/ui.rb
App 65296 stderr:   173 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/ui/silent.rb
App 65296 stderr:   174 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/ui/rg_proxy.rb
App 65296 stderr:   175 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/index.rb
App 65296 stderr:   176 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/remote_specification.rb
App 65296 stderr:   177 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/stub_specification.rb
App 65296 stderr:   178 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/endpoint_specification.rb
App 65296 stderr:   179 /Users/rich/.rvm/gems/ruby-2.3.1/gems/bundler-1.13.6/lib/bundler/setup.rb
App 65296 stderr:   180 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/multipart.rb
App 65296 stderr:   181 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/utils.rb
App 65296 stderr:   182 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack.rb
App 65296 stderr:   183 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/builder.rb
App 65296 stderr:   184 /Users/rich/.rvm/gems/ruby-2.3.1/gems/tilt-2.0.5/lib/tilt/dummy.rb
App 65296 stderr:   185 /Users/rich/.rvm/gems/ruby-2.3.1/gems/tilt-2.0.5/lib/tilt/mapping.rb
App 65296 stderr:   186 /Users/rich/.rvm/gems/ruby-2.3.1/gems/tilt-2.0.5/lib/tilt/template.rb
App 65296 stderr:   187 /Users/rich/.rvm/gems/ruby-2.3.1/gems/tilt-2.0.5/lib/tilt.rb
App 65296 stderr:   188 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/version.rb
App 65296 stderr:   189 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection.rb
App 65296 stderr:   190 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/ostruct.rb
App 65296 stderr:   191 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/erb.rb
App 65296 stderr:   192 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/request.rb
App 65296 stderr:   193 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/showexceptions.rb
App 65296 stderr:   194 /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/show_exceptions.rb
App 65296 stderr:   195 /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/ext.rb
App 65296 stderr:   196 /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/version.rb
App 65296 stderr:   197 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/body_proxy.rb
App 65296 stderr:   198 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/response.rb
App 65296 stderr:   199 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/commonlogger.rb
App 65296 stderr:   200 /Users/rich/.rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb
App 65296 stderr:   201 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/session/abstract/id.rb
App 65296 stderr:   202 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/session/pool.rb
App 65296 stderr:   203 /Library/WebServer/Documents/usersys/app.rb
App 65296 stderr:   204 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/head.rb
App 65296 stderr:   205 /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/logger.rb
App 65296 stderr:   206 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/logger.rb
App 65296 stderr:   207 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/session/cookie.rb
App 65296 stderr:   208 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/base.rb
App 65296 stderr:   209 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb
App 65296 stderr:   210 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/http_origin.rb
App 65296 stderr:   211 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/ip_spoofing.rb
App 65296 stderr:   212 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb
App 65296 stderr:   213 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb
App 65296 stderr:   214 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/authenticity_token.rb
App 65296 stderr:   215 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/remote_token.rb
App 65296 stderr:   216 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/session_hijacking.rb
App 65296 stderr:   217 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb
App 65296 stderr:   218 /Users/rich/.rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/multipart/parser.rb
App 65296 stderr:   219 /Users/rich/.rvm/gems/ruby-2.3.1/gems/tilt-2.0.5/lib/tilt/erb.rb
App 65296 stderr:   220 /Library/WebServer/Documents/usersys/inc/methods.rb
App 65296 stderr:   221 /Library/WebServer/Documents/usersys/inc/session.rb
App 65296 stderr: 
App 65296 stderr: [NOTE]
App 65296 stderr: You may have encountered a bug in the Ruby interpreter or extension libraries.
App 65296 stderr: Bug reports are welcome.
App 65296 stderr: Don't forget to include the above Crash Report log file.
App 65296 stderr: For details: http://www.ruby-lang.org/bugreport.html
App 65296 stderr: 
[ 2016-11-10 15:01:55.0100 29145/0x700000041000 age/Cor/App/Poo/AnalyticsCollection.cpp:102 ]: Process (pid=65374, group=/Library/WebServer/Documents/usersys/public (development)) no longer exists! Detaching it from the pool.
[ 2016-11-10 15:01:55.0102 29145/0x700000041000 age/Cor/CoreMain.cpp:819 ]: Checking whether to disconnect long-running connections for process 65374, application /Library/WebServer/Documents/usersys/public (development)
App 65382 stdout:

Jeremy Evans

unread,

Nov 10, 2016, 3:34:59 PM11/10/16

to ruby-pg

On Thursday, November 10, 2016 at 12:10:12 PM UTC-8, BeeRich33 wrote:

Segmentation faults all over the place. Narrowed it down to my app (Sinatra) requiring 'pg'.

Ruby 2.3.1
Sinatra on Passenger
pg 0.19.0
All gems updated

The same code works in IRB, but not in my Sinatra route as a post. It worked before, and not now. Seg Faults were coming up all over the place and this is how simple I've narrowed it down.

BTW this group doesn't look very active. Hopefully someone is browsing. Any advice appreciated. Cheers

Does the problem go away if you disable passenger "smart" spawning? The most likely cause of this is you set up the pg connection in the master process, all child processes inherit it and attempt to use the connection at the same time, which causes the segfault. If disabling smart spawning makes the problem go away, the solution is either to keep smart spawning disabled, or disconnect the pg connection before forking and reconnect after.

Thanks,

Jeremy

BeeRich33

unread,

Nov 10, 2016, 4:34:01 PM11/10/16

to ruby-pg

HI Jeremy. Thanks for the insight.

Well I tried using the direct method through nginx.conf and that didn't work. config test worked, and restart worked.

Then I tried logging out and trying a simple site reload. There is only a Bundler.require which asks for gems 'pg' and 'nokogiri'. Upon commenting those out again, the site loads. There is no connection to the database on the top page of that site. Not sure what's going on there.

I have restarted the box and the same happens.

On the second site, the site loads. 'pg' is installed properly. I fill out a simple form, and up on the post route (again, Sinatra), I get this in the browser:

Incomplete response received from application

Error.log shows the same segmentation fault. Now, at this point, no connection has been opened. The fault is talking about the pg gem and "FINISH" being an issue, but no connections are now open. Not too sure how to fix this. The require is before any connection is being asked for.

Michael Granger

unread,

Nov 10, 2016, 8:15:30 PM11/10/16

to ruby-pg

From the backtrace, it looks like it's happening when the extension is loaded, which is usually one of two things:

1. Your Ruby's OpenSSL library (or some other dynamic library that's already been loaded) is compiled against a different version of OpenSSL than libpq (the PostgreSQL client library).

2. Your extension was installed under a different version of Ruby that you're loading it with.

The CrashReporter log mentioned at the top of the error report might provide additional clues (possibly under a section called "Dyld Error Message"), but I'd check to make sure that the OpenSSL that's loaded in the app is the same as the one linked against libpq.

BeeRich33

unread,

Nov 10, 2016, 8:19:32 PM11/10/16

to ruby-pg

Wouldn't I have to install the PG gem in any version of RVM Ruby that I wish to use? It would raise an error that the gem isn't installed, correct? In any case I'm going to have a look at other versions to see if it shakes it out.

Michael Granger

unread,

Nov 10, 2016, 8:43:27 PM11/10/16

to ruby-pg

On Thursday, November 10, 2016 at 5:19:32 PM UTC-8, BeeRich33 wrote:

Wouldn't I have to install the PG gem in any version of RVM Ruby that I wish to use? It would raise an error that the gem isn't installed, correct? In any case I'm going to have a look at other versions to see if it shakes it out.

Yes, but that might be why it works under IRB but not under Passenger. If Nginx, for example, is compiled against OpenSSL 0.9.8:

$ otool -L `which nginx` | grep ssl
	/usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 47.1.0)

but libpq is compiled against OpenSSL 1.0.2j:

$ otool -L /usr/local/opt/postgresql/lib/libpq.5.dylib | grep ssl
	/usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)

then loading the `pg` gem from a Ruby embedded in the nginx process will segfault because the OpenSSL symbols don't match. I'm not sure exactly how Passenger works, but in my experience if `pg` segfaults on require, it's one of the two causes I mentioned.

BeeRich33

unread,

Nov 10, 2016, 8:50:59 PM11/10/16

to ruby-pg

Yes, I'm clear that it involves Passenger somewhat. But Passenger has been installed all this time. I will change versions of Ruby and see how that plays out. Other than that, nothing else has changed. I'm not sure what to do after that.

BeeRich33

unread,

Nov 10, 2016, 9:12:31 PM11/10/16

to ruby-pg

OK just removed the gem 'pg' and reinstalled it. All under Ruby 2.3.0 through RVM.

Same problem as stated before. Using your 'otool':

rich$ otool -L `which nginx` | grep ssl
       	/usr/local/opt/openssl@1.1/lib/libssl.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
       	/usr/local/opt/openssl@1.1/lib/libcrypto.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
rich$ otool -L /usr/local/opt/postgresql/lib/libpq.5.dylib | grep ssl

       	/usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
       	/usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)

Not sure how to proceed from this point.

p...@citylink.dinoex.sub.org

unread,

Jun 28, 2017, 10:54:47 AM6/28/17

to ruby-pg

Am Freitag, 11. November 2016 03:12:31 UTC+1 schrieb BeeRich33:

Not sure how to proceed from this point.

Same problem here. :(

Happens with pg 0.18.4 and 0.21.0
Happens with ruby 2.3.1p112 and 2.3.4p301

Has nothing to do with passenger, but:

[admin@rail ~]$ GEM_PATH=/ext/gems/2.3 ruby23
require 'pg'
[admin@rail ~]$ GEM_PATH=/ext/gems/2.3 ruby23
require 'rails'
[admin@rail ~]$ GEM_PATH=/ext/gems/2.3 ruby23
require 'pg'
require 'rails'
[admin@rail ~]$ GEM_PATH=/ext/gems/2.3 ruby23
require 'rails'
require 'pg'
KABUMM!

All stuff compiled onsite, so lib mismatch is unlikely.
Same versions work on another machine (but thats different platform hardware)

Investigating further...

p...@citylink.dinoex.sub.org

unread,

Jun 28, 2017, 12:53:00 PM6/28/17

to ruby-pg

Problem seems solved.

Am Freitag, 11. November 2016 02:15:30 UTC+1 schrieb Michael Granger:

1. Your Ruby's OpenSSL library (or some other dynamic library that's already been loaded) is compiled against a different version of OpenSSL than libpq (the PostgreSQL client library).

Yep. Had to rebuild the ruby under the older OS-version used to build postgres-client (libpq).

The relation to passenger/rails comes because rails depends on active_support, and active_support pulls in a lot of ruby binary extensions which were compiled together with ruby.

Reply all

Reply to author

Forward

0 new messages