Emergency call for help - production server down after updating Ruby

[dana tassler]

Updated Ruby on a FreeBSD server Friday night. Web application seemed to work fine. No problems were reported all weekend.

This morning, people have encountered a passenger error:

Since I'm connected remotely using teamviewer, I can't get the screen shot of the back trace. It says, "Passenger encountered the following error:

The application spawner server exited unexpectedly: Unexpected end-of-file detected.

Exception class:

PhusionPassenger::Rack::ApplicationSpawner::Error"

We are quite desperate to resolve this problem. 

I've attempted deploying to the server using Capistrano and receive SSL errors about the first gemspec in Gemfile.lock.

Is there anyone in the area we could contract to fix this issue? I need help.

Thanks,

Dana

[dana tassler]

Was able to take a screen shot, it's attached. 

[Mark McEahern]

I don't have the bandwidth to commit to helping right now, but a google search (which I'm sure you've done, so ignore if you've already tried this), led to:

http://stackoverflow.com/questions/1429708/passenger-error-the-application-spawner-server-exited-unexpectedly-unexpected

which makes me wonder if you've been able to view the Passenger logs or other relevant logs.

// m

--
You received this message because you are subscribed to the MadRailers Google Group group.
To visit the MadRailers home page, go to http://madrailers.org
To post to this group, send email to mad-r...@googlegroups.com
---
You received this message because you are subscribed to the Google Groups "MadRailers: Madison's Ruby/Rails User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mad-railers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mark McEahern]

The error seems to indicate that some dependencies are not available.  As if you need to:

  bundle install

// m

--

[Jim Remsik]

With regard to the `bundle install` failing perhaps this will help? http://stackoverflow.com/questions/10246023/bundle-install-fails-with-ssl-certificate-verification-error

[David Lee]

How big of a Ruby update?  Passenger will probably have to be re-compiled to use the new version of Ruby properly. After re-compiling the passenger gem you'd also have to re-build the web server module as well.

-David

[David Southard]

Pasenger works a little differently as it runs as a mod on apache or nginx. Its essentially C code which requires there to be a version of ruby for some items. Chances are your mod links in apache or nginx are pointing to the wrong passenger... i.e. they're pointing to the old version of ruby. You can check this by checking the mods settings in /etc/apache/mods-available/passenger.load and passenger.conf 

The fix is easier than messing with these files. 

Basically install the latest passenger gem in your new ruby and then run passenger-install-apache2-module or passenger-install-nginx-module

http://www.modrails.com/documentation/Users%20guide%20Nginx.html#install_on_debian_ubuntu

At the end of this it'll tell you to add some lines to passenger.load and passenger.conf. 

When you restart or reload apache or nginx you should be back in business. 

Dave

[Mark McEahern]

In terms of process, you may want to look into having a non-production environment where you can vet these sorts of changes before you roll them out to production.  If you do have one, how come this error didn't get noticed there?

// m

--

[dana tassler]

Gentlemen,

I did issue "gem update --system" and also "bundle install --without development test"

Error which bundle install command yields is:

"Could not verify the SSL certificate for https://rubygems.org/quick/Marshal.4.8/Ascii85-1.0.2.gemspec.rz.

There is a chance you are experiencing a man-in-the-middle attack, but most likely your system doesn't have the CA certificates needed for

verification. For information about OpenSSL certificates, see bit.ly/ruby-ssl. To connect without using SSL, edit your Gemfile sources and

change 'https' to 'http'."

There are no calls to https in Gemfile so I feel I can disregard the suggestions online to change "https" in Gemfile to "http."

David - I'm not sure which patch level the server's Ruby version was on prior to my update but now it's at 1.9.3p484.

You mentioned recompiling Passenger, how do I do that?

--

"One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies."

- C.A.R. Hoare, in talking about designing software.

[David Southard]

I'm going to assume you're using apache. 

I just noticed you're running FreeBSD, so the directory for apache is different in FreeBSD. But in the apache directory there should be a mods-available directory and the files for passenger are looking for the old version of ruby. You need to install the passenger gem and then passenger-install-apache2-module. Then the passenger install will give you the directions at the end. 

Also note if you just rollback to the old version of ruby (the version specified in the passenger.conf and passenger.load files), you can probably get your site back up until you can upgrade in a more organized manner. 

[dana tassler]

Did compile passenger using Apache and added the entry to the end of Apache's config file. It's at /usr/local/etc/apache22/httpd.conf

Restarted Apache and received: 

Syntax error on line 480 of /usr/local/etc/apache22/httpd.conf:

Invalid command 'PassengerDefaultRuby', perhaps misspelled or defined by a module not included in the server configuration

The entry I was instructed to make is:

Please edit your Apache configuration file, and add these lines:

   LoadModule passenger_module /usr/local/lib/ruby/gems/1.9/gems/passenger-4.0.40/buildout/apache2/mod_passenger.so

   <IfModule mod_passenger.c>

     PassengerRoot /usr/local/lib/ruby/gems/1.9/gems/passenger-4.0.40

     PassengerDefaultRuby /usr/local/bin/ruby19

   </IfModule>

Did I place the code in the wrong location? 

[David Southard]

is there a passenger.load and passenger.conf file in the mods-available directory? 

because you should put the LoadModule line in the .load file and the rest in the .conf file you should see similar lines to these already there just with the old version of ruby

[dana tassler]

I know this sounds like a rookie statement, but I can't find the mods-enabled directory.

Sort of working in the blind here with an application I inherited from someone who's no longer here.

[dana tassler]

After updated the locate database, executing "locate passenger.conf" yields:

/usr/local/lib/ruby/gems/1.8/gems/passenger-3.0.2/debian/passenger.conf

/usr/local/lib/ruby/gems/1.9/gems/passenger-3.0.2/debian/passenger.conf

/usr/local/lib/ruby/gems/1.9/gems/passenger-4.0.40/debian.template/passenger.conf

/usr/local/lib/ruby/gems/1.9/gems/passenger-4.0.40/rpm/apache-passenger.conf.in

/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.2/debian/passenger.conf

/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.7/debian/passenger.conf

/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/debian/passenger.conf

4.0.4 is what I just installed.

[David Southard]

it should be a directory in the apache root directory... where http.conf is located... I say should because I am used to ubuntu linux so its probably called something different but it will be a file with similar settings to the ones you were told to put in http.conf

[David Southard]

you could also just grep for LoadModule passenger_module and that'll probably get you to where the existing files are... 

[dana tassler]

After updating the locate database, issuing "locate passenger.load" yields:

usr/local/lib/ruby/gems/1.8/gems/passenger-3.0.2/debian/passenger.load

/usr/local/lib/ruby/gems/1.9/gems/passenger-3.0.2/debian/passenger.load

/usr/local/lib/ruby/gems/1.9/gems/passenger-4.0.40/debian.template/passenger.load

/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.2/debian/passenger.load

/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.7/debian/passenger.load

/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/debian/passenger.load

Since 4.0.40 was the version of Passenger I installed this morning, I added the suggested lines to it. Restarted apache using "sudo service apache22 restart"

This yielded:

Performing sanity check on apache22 configuration:

Syntax OK

Stopping apache22.

Waiting for PIDS: 2582.

Performing sanity check on apache22 configuration:

Syntax OK

Starting apache22.

(48)Address already in use: make_sock: could not bind to address [::]:80

Tried using web application and it's still displaying the same Passenger error.

Am I on the right track here?

Also, CANNOT thank you enough for your help, David!!!

[David Southard]

I'm afraid my FreeBSD ignorance is showing. Those look like the paths to the ruby passenger gems not the path to the apache where apache loads passenger. I would try grep-ing for the file that contains the text 

LoadModule passenger_module

and then if the config portion isn't in there grep for 

PassengerRoot 

Also you'll need to take the lines out of http.conf if its loaded separately. 

[dana tassler]

Grepping shows that entry in /home/deploy/httpd.conf.

Relevant line from httpd.conf:

LoadModule passenger_module /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.11/ext/apache2/mod_passenger.so

[dana tassler]

I think I finally understand what has to happen.

The instructions after recompiling passenger say to add:

   LoadModule passenger_module /usr/local/lib/ruby/gems/1.9/gems/passenger-4.0.40/buildout/apache2/mod_passenger.so

   <IfModule mod_passenger.c>

     PassengerRoot /usr/local/lib/ruby/gems/1.9/gems/passenger-4.0.40

     PassengerDefaultRuby /usr/local/bin/ruby19

   </IfModule>

...to the Apache configuration file.

This is /home/deploy/httpd.conf and those entries are actually located in different places.

I replaced

"LoadModule passenger_module /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.11/ext/apache2/mod_passenger.so"

with 

"LoadModule passenger_module /usr/local/lib/ruby/gems/1.9/gems/passenger-4.0.40/buildout/apache2/mod_passenger.so"

and also commented out, further in the config file,

"#<IfModule passenger_module>

#PassengerRoot /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.11

#PassengerRuby /usr/local/bin/ruby18

#</IfModule>"

and adding directly afterward:

"<IfModule mod_passenger.c>

PassengerRoot /usr/local/lib/ruby/gems/1.9/gems/passenger-4.0.40

PassengerDefaultRuby /usr/local/bin/ruby19

</IfModule>"

Also restarted Apache once more but that didn't resolve the issue.

[David Southard]

yup. that's where you'd change it... then you need to find the other PassegerRoot lines and replace them too. IIRC older versions might have just had the PassengerRoot line 

note. you've got a lot of moving parts here!

It looks like you were running passenger 2.2. IIRC you may not be able to jump all the way to passenger 4 directly. 

You may have to install the passenger 2 version and then recompile and repeat what you've been doing with the load and conf settings. Annoying I know but you've got a lot of variables in play here... ruby upgrade, and passenger upgrade. Also you're coming from ruby 1.8, so is your code 1.9 compatible? you may be biting off a lot more than you want right now. 

[Abe Voelker]

I don't know anything about Passenger but the message "Address already in use" means something is already bound to that port.  If Apache isn't restarting due to that, you might have to kill whatever process is bound to port 80.  You can find what process is using that port with lsof: sudo lsof -i :80

[dana tassler]

I believe we were running Ruby 1.9.2 prior to this upgrade, so hopefully that's not the big issue.

Did also just look in /var/log/httpd-error.log and found multiple duplicate entries:

[ASYNC BUG] thread_timer: select

EBADF

ruby 1.9.3p484 (2013-11-22 revision 43786) [amd64-freebsd8]

[NOTE]

You may have encountered a bug in the Ruby interpreter or extension libraries.

Bug reports are welcome.

For details: http://www.ruby-lang.org/bugreport.html

I'm attempting to do all this remotely and I have to admit I'm not well-suited for it. Is there any way you can visit our location and attempt to repair it?

We've been down for nearly four hours now and it's crippling our business. 

[dana tassler]

Mr. Abe,

sudo lsof -i :80 yields:

httpd   1951 root    3u  IPv6 0xffffff004e9a2000      0t0  TCP *:http (LISTEN)

httpd   2030  www    3u  IPv6 0xffffff004e9a2000      0t0  TCP *:http (LISTEN)

httpd   2031  www    3u  IPv6 0xffffff004e9a2000      0t0  TCP *:http (LISTEN)

httpd   2033  www    3u  IPv6 0xffffff004e9a2000      0t0  TCP *:http (LISTEN)

httpd   2036  www    3u  IPv6 0xffffff004e9a2000      0t0  TCP *:http (LISTEN)

httpd   2038  www    3u  IPv6 0xffffff004e9a2000      0t0  TCP *:http (LISTEN)

httpd   2040  www    3u  IPv6 0xffffff004e9a2000      0t0  TCP *:http (LISTEN)

httpd   2042  www    3u  IPv6 0xffffff004e9a2000      0t0  TCP *:http (LISTEN)

httpd   2044  www    3u  IPv6 0xffffff004e9a2000      0t0  TCP *:http (LISTEN)

httpd   2994 root    4u  IPv4 0xffffff015d163b10      0t0  TCP *:http (LISTEN)

httpd   3008  www    4u  IPv4 0xffffff015d163b10      0t0  TCP *:http (LISTEN)

httpd   3009  www    4u  IPv4 0xffffff015d163b10      0t0  TCP *:http (LISTEN)

httpd   3010  www    4u  IPv4 0xffffff015d163b10      0t0  TCP *:http (LISTEN)

httpd   3011  www    4u  IPv4 0xffffff015d163b10      0t0  TCP *:http (LISTEN)

httpd   3012  www    4u  IPv4 0xffffff015d163b10      0t0  TCP *:http (LISTEN)

httpd   3013  www    4u  IPv4 0xffffff015d163b10      0t0  TCP *:http (LISTEN)

httpd   3014  www    4u  IPv4 0xffffff015d163b10      0t0  TCP *:http (LISTEN)

httpd   3015  www    4u  IPv4 0xffffff015d163b10      0t0  TCP *:http (LISTEN)

httpd   3073  www    4u  IPv4 0xffffff015d163b10      0t0  TCP *:http (LISTEN)

What exactly does that mean?

[David Southard]

indeed you should check the apache logs for more info. 

It looks like they put the passenger loading in the http.conf file and the Passenger settings in the vhost file which is fine. Swapping these out with the new settings should work. 

If apache is starting and passenger is not... check your rails logs to see what the error is. 

[dana tassler]

Production log shows a get request at 8:13 this morning followed by several:

MONGODB [WARNING] Please note that logging negatively impacts client-side performance. You should set your logging level no lower than :info in production.

[Abe Voelker]

That's a list of processes (2nd column is the PID) that are bound to port 80. The "TCP *" means that they are bound to all interfaces, which usually means publicly-accessible.

It's been a while since I used Apache, but some http daemons keep track of the running instance using a .pid file stored somewhere, and that can sometimes get out-of-sync with what is actually running (like if it issues a kill and wipes the .pid file, but the targeted process doesn't actually die).

Long story short if you continually get the "Address in use" message and the PIDs you just listed using lsof don't change after issuing restarts to Apache, you might want to do a

sudo kill 1951

sudo kill 2994

and then restart Apache to make sure it is restarting and reading in your config changes.

[Abe Voelker]

Also I'd agree with Dave on forgetting about the update and backing out your changes and getting the app running again with the old code and config if you can.

[dana tassler]

How can I back out the changes and revert to a previous version of Ruby using ports on FreeBSD?

What a lovely Monday it's turning out to be.

[David Southard]

The MongoDB warning is a separate issue related to how your log level is set. 

Ok. Here's how you need to triage. 

- Is apache loading? 

- If not, why? 

  - the logs for apache will be your friend here

Once Apache (with the new passenger settings) is up and running then its an issue of the Rails application

- Is Rails loading? 

  - if not there should be a error logged either in the rails

[Abe Voelker]

Sorry, don't know much about FreeBSD or its package management.  If you have a backup of this box it might be worth it to restore from that, and redeploy the most recent code.

[dana tassler]

Looks like the Passenger/Apache issue has been resolved!

New error, though. It can't find rake. Is this due to running "gem update --system?"

Any ideas how to fix this?

[David Southard]

now you're onto 

bundle install --without development test

Then it should work

[dana tassler]

Executed that same code moments ago. Received:

Fetching source index from http://gemcutter.org/

Fetching source index from http://rubygems.org/

Could not verify the SSL certificate for https://rubygems.org/quick/Marshal.4.8/Ascii85-1.0.2.gemspec.rz.

There is a chance you are experiencing a man-in-the-middle attack, but most likely your system doesn't have the CA certificates needed for

verification. For information about OpenSSL certificates, see bit.ly/ruby-ssl. To connect without using SSL, edit your Gemfile sources and

change 'https' to 'http'.

Same results I'd seen earlier.

[dana tassler]

Manually installing Ascii85 resolves that issue, the error then shifts to ActionMailer.

It seems as though there's some sort of permissions issue installing these gems using bundler.

[David Southard]

That's another separate issue. Its not a bundler issue really. It related to the sources you have added to your rubygems although you can also put sources in your Gemfile too that would be checked first, this isn't that. 

http://guides.rubygems.org/command-reference/#gem_sources

Now you're onto gem resolution, so at this point there should be plenty of resources on the web. These are dependency issues related to your upgrade. 

Good Luck. 

[dana tassler]

Well, keep your fingers crossed.

Thanks much for all the help!

[dana tassler]

For those of you who have stayed tuned and are interested, our production application is once again up and running!

I upgraded Ruby on a FreeBSD server using ports ("sudo portmaster /usr/ports/lang/ruby19"), I did not update Passenger/Apache.

After installing and recompiling Passenger/Apache and updating the appropriate httpd.conf file, (which was found by using: 'grep -irl "LoadModule passenger_module" .') and restarting Apache, I received errors related to gems which were missing from the server. I was not able to resolve this by issuing "bundle install --without development test." I suspect this is because at one point while troubleshooting I issued "gem update --system" which installed an updated version of rubygems. 

I installed each of the missing gems manually. Once they were finally installed, the application was once more functional!

I'm indebted to you all for your help. If I can help any of you in any way, please do not hesitate to ask!

Now I'm going to draft up a plan for a staging/backup server so this never happens again.

-- Dana

[Christopher Kintner]

On the server does the rails console for the application start? That's always a good way to determine if you have a passenger issue or a rails issue.

http://guides.rubyonrails.org/command_line.html#rails-console

Chris

On Mon, Apr 7, 2014 at 9:54 AM, Jim Remsik <jrem...@gmail.com> wrote:

With regard to the `bundle install` failing perhaps this will help? http://stackoverflow.com/questions/10246023/bundle-install-fails-with-ssl-certificate-verification-error

On Mon, Apr 7, 2014 at 9:52 AM, Mark McEahern <mark.m...@gmail.com> wrote:

The error seems to indicate that some dependencies are not available.  As if you need to:

  bundle install

// m

On Mon, Apr 7, 2014 at 9:50 AM, dana tassler <dana.t...@gmail.com> wrote:

Was able to take a screen shot, it's attached. 

On Monday, April 7, 2014 10:46:34 AM UTC-4, dana tassler wrote:

Updated Ruby on a FreeBSD server Friday night. Web application seemed to work fine. No problems were reported all weekend.

This morning, people have encountered a passenger error:

Since I'm connected remotely using teamviewer, I can't get the screen shot of the back trace. It says, "Passenger encountered the following error:

The application spawner server exited unexpectedly: Unexpected end-of-file detected.

Exception class:

PhusionPassenger::Rack::ApplicationSpawner::Error"

We are quite desperate to resolve this problem. 

I've attempted deploying to the server using Capistrano and receive SSL errors about the first gemspec in Gemfile.lock.

Is there anyone in the area we could contract to fix this issue? I need help.

Thanks,

Dana

--
You received this message because you are subscribed to the MadRailers Google Group group.
To visit the MadRailers home page, go to http://madrailers.org
To post to this group, send email to mad-r...@googlegroups.com
---
You received this message because you are subscribed to the Google Groups "MadRailers: Madison's Ruby/Rails User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mad-railers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the MadRailers Google Group group.
To visit the MadRailers home page, go to http://madrailers.org
To post to this group, send email to mad-r...@googlegroups.com
---
You received this message because you are subscribed to the Google Groups "MadRailers: Madison's Ruby/Rails User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mad-railers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bradley Grzesiak]

Apologies to Chris Kintner, who sent that email yesterday, but I couldn't get around to moderating it for the group until just now.

Welcome to the mad-railers group, Chris!

:brad

Bradley Grzesiak
co-founder, bendyworks inc
http://bendyworks.com/