Passenger crashes/ hangs Apache startup when using gem and module built by passenger-install-apache2-module

700 views
Skip to first unread message

Tom Harrison

unread,
May 7, 2012, 3:30:39 PM5/7/12
to phusion-...@googlegroups.com
When I start Apache (sudo apache2ctl start) with the passenger module enabled, the server never starts (apache2ctl hangs).  After Ctrl-C, I see the following in my Apache log:

[ pid=9511 thr=139649603958624 file=ext/apache2/Hooks.cpp:1389 time=2012-05-07 19:08:17.598 ]: Initializing Phusion Passenger...
[ pid=9511 thr=139649603958624 file=ext/apache2/Hooks.cpp:1664 time=2012-05-07 19:08:17.633 ]: Shutting down Phusion Passenger...
[ pid=9520 thr=139806630336352 file=ext/common/LoggingAgent/Main.cpp:283 time=2012-05-07 19:08:17.634 ]: Logging agent online, listening at unix:/tmp/passenger.1.0.9511/generation-0/logging.socket

ps shows these lines at this point.  (I need to kill the PassengerWatchdog process).

4 S root      9512     1  0  80   0 -  5938 hrtime 19:08 ?        00:00:00 PassengerWatchdog
0 Z root      9515  9512  0  80   0 -     0 exit   19:08 ?        00:00:00 [PassengerHelper] <defunct>
4 Z nobody    9520  9512  0  80   0 -     0 exit   19:08 ?        00:00:00 [PassengerLoggin] <defunct>

I installed Passenger 3.0.12 using "gem install passenger", then built the apache module using "passenger-install-apache2-module", all on Ubuntu 11.10 64-bit, ruby 1.9.3-p194 .

I am using RVM in "system" mode, so all gems and binaries are under /usr/local/rvm... -- the passenger.load and passenger.conf apache files point to the location of the module and associated ruby.

The original Ubuntu install had an older version of ruby and passenger installed.  In diagnosing the problem, I tried removing them (sudo apt-get remove xxx) -- no change.  I re-compiled the apache module afterwards.

I have eliminated my app from the picture -- it's not being loaded as an apache virtual host.

I have googled and read numerous similar issues all of which seem to be related to recompiling the apache module.  Not so much for me :-(

Any ideas?

Thanks in advance!

Tom Harrison

Tom Harrison

unread,
May 7, 2012, 4:57:17 PM5/7/12
to phusion-...@googlegroups.com
I also noticed the following in syslog

May  7 19:08:17 test kernel: [17535.269421] alignment check: 0000 [#11] SMP 
May  7 19:08:17 test kernel: [17535.269822] CPU 0 
May  7 19:08:17 test kernel: [17535.269934] Modules linked in: acpiphp
May  7 19:08:17 test kernel: [17535.270275] 
May  7 19:08:17 test kernel: [17535.270287] Pid: 9525, comm: PassengerWatchd Tainted: G      D     3.0.0-12-virtual #20-Ubuntu  
May  7 19:08:17 test kernel: [17535.270298] RIP: e030:[<ffffffff8100a2f5>]  [<ffffffff8100a2f5>] __setup_rt_frame+0x105/0x330
May  7 19:08:17 test kernel: [17535.270315] RSP: e02b:ffff88002384fda8  EFLAGS: 00050246
May  7 19:08:17 test kernel: [17535.270320] RAX: ffff88002384ffd8 RBX: ffff88002384fee8 RCX: ffff880023e7ae00
May  7 19:08:17 test kernel: [17535.270326] RDX: ffff88002384fe68 RSI: ffff88002384ffd8 RDI: 0000000000000000
May  7 19:08:17 test kernel: [17535.270332] RBP: ffff88002384fdf8 R08: 0000000000000000 R09: 0000000000000000
May  7 19:08:17 test kernel: [17535.270338] R10: 0000000000000400 R11: 0000000000000202 R12: ffff880023e7b348
May  7 19:08:17 test kernel: [17535.270344] R13: 00007f50458a0b38 R14: ffff88002384ff58 R15: 00007f50458a0d00
May  7 19:08:17 test kernel: [17535.270353] FS:  00007f50458a1700(0000) GS:ffff88002657b000(0000) knlGS:0000000000000000
May  7 19:08:17 test kernel: [17535.270360] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
May  7 19:08:17 test kernel: [17535.270365] CR2: 00007f9c8f895760 CR3: 0000000002704000 CR4: 0000000000002620
May  7 19:08:17 test kernel: [17535.270372] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May  7 19:08:17 test kernel: [17535.270378] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
May  7 19:08:17 test kernel: [17535.270384] Process PassengerWatchd (pid: 9525, threadinfo ffff88002384e000, task ffff880023e7ae00)
May  7 19:08:17 test kernel: [17535.270391] Stack:
May  7 19:08:17 test kernel: [17535.270395]  ffff88002384fe68 00007f50458a0b38 ffff880023e7ae00 0000000a81073aea
May  7 19:08:17 test kernel: [17535.270406]  ffff880023e7b248 000000000000000a ffff88002384fee8 000000000000000a
May  7 19:08:17 test kernel: [17535.270417]  0000000000000000 0000000000000003 ffff88002384fe18 ffffffff8100a588
May  7 19:08:17 test kernel: [17535.270428] Call Trace:
May  7 19:08:17 test kernel: [17535.270435]  [<ffffffff8100a588>] setup_rt_frame+0x68/0x90
May  7 19:08:17 test kernel: [17535.270441]  [<ffffffff8100a622>] handle_signal+0x72/0x1a0
May  7 19:08:17 test kernel: [17535.270448]  [<ffffffff8100a7c3>] do_signal+0x73/0x180
May  7 19:08:17 test kernel: [17535.270456]  [<ffffffff81166a6d>] ? vfs_read+0x10d/0x180
May  7 19:08:17 test kernel: [17535.270462]  [<ffffffff81166b47>] ? sys_read+0x67/0x90
May  7 19:08:17 test kernel: [17535.270469]  [<ffffffff8100aa95>] do_notify_resume+0x65/0x80
May  7 19:08:17 test kernel: [17535.270477]  [<ffffffff81057767>] ? schedule_tail+0x27/0xb0
May  7 19:08:17 test kernel: [17535.270486]  [<ffffffff8160bc90>] int_signal+0x12/0x17
May  7 19:08:17 test kernel: [17535.270490] Code: ff 0f 85 3a 02 00 00 f6 43 08 04 0f 85 05 02 00 00 65 48 8b 04 25 48 cd 00 00 44 8b 80 88 e0 ff ff c7 80 88 e0 ff ff 00 00 00 00 
May  7 19:08:17 test kernel: [17535.270578] RIP  [<ffffffff8100a2f5>] __setup_rt_frame+0x105/0x330
May  7 19:08:17 test kernel: [17535.270586]  RSP <ffff88002384fda8>
May  7 19:08:17 test kernel: [17535.270592] ---[ end trace 623a667566218b53 ]---

Tom Harrison

unread,
May 7, 2012, 5:42:48 PM5/7/12
to phusion-...@googlegroups.com
Additional information:
  • with the application enabled (virtual host set up) apache comes up clean on a reboot
  • I can access static files (i.e. robots.txt) from the document root
  • but any access to the Rails part causes a crash. kern.log dump below
  • note the last line: Fixing recursive fault but reboot is needed!
May  7 21:35:30 test kernel: [  255.546147] alignment check: 0000 [#1] SMP 
May  7 21:35:30 test kernel: [  255.546165] CPU 0 
May  7 21:35:30 test kernel: [  255.546168] Modules linked in: acpiphp
May  7 21:35:30 test kernel: [  255.546176] 
May  7 21:35:30 test kernel: [  255.546181] Pid: 1158, comm: ruby Not tainted 3.0.0-12-virtual #20-Ubuntu  
May  7 21:35:30 test kernel: [  255.546189] RIP: e030:[<ffffffff8112cac4>]  [<ffffffff8112cac4>] zap_pte_range+0x44/0x3d0
May  7 21:35:30 test kernel: [  255.546205] RSP: e02b:ffff8800246dba78  EFLAGS: 00050296
May  7 21:35:30 test kernel: [  255.546210] RAX: ffff880023fb5c00 RBX: ffff8800244d9018 RCX: 0000000000400000
May  7 21:35:30 test kernel: [  255.546216] RDX: ffff8800244d9010 RSI: ffff8800240c0a50 RDI: ffff8800246dbc18
May  7 21:35:30 test kernel: [  255.546222] RBP: ffff8800246dbb18 R08: 0000000000401000 R09: 0000000000000000
May  7 21:35:30 test kernel: [  255.546227] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000400000
May  7 21:35:30 test kernel: [  255.546233] R13: 0000000000401000 R14: ffff880002733c00 R15: 0000000000401000
May  7 21:35:30 test kernel: [  255.546243] FS:  00007f784475d700(0000) GS:ffff88002657b000(0000) knlGS:0000000000000000
May  7 21:35:30 test kernel: [  255.546249] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
May  7 21:35:30 test kernel: [  255.546254] CR2: 0000000001e286f0 CR3: 0000000001c03000 CR4: 0000000000002620
May  7 21:35:30 test kernel: [  255.546261] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May  7 21:35:30 test kernel: [  255.546267] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
May  7 21:35:30 test kernel: [  255.546273] Process ruby (pid: 1158, threadinfo ffff8800246da000, task ffff880023fb5c00)
May  7 21:35:30 test kernel: [  255.546279] Stack:
May  7 21:35:30 test kernel: [  255.546282]  00000000238b8000 ffff8800246dbb28 ffffffff8100471d ffff8800246dbac0
May  7 21:35:30 test kernel: [  255.546293]  ffffffff81005c49 ffff880023fb5c00 ffff8800244d9010 ffff8800244d9010
May  7 21:35:30 test kernel: [  255.546304]  ffff8800240c0a50 ffff8800246dbc18 0000000000000000 ffff8800246dbba8
May  7 21:35:30 test kernel: [  255.546315] Call Trace:
May  7 21:35:30 test kernel: [  255.546323]  [<ffffffff8100471d>] ? __raw_callee_save_xen_make_pte+0x11/0x1e
May  7 21:35:30 test kernel: [  255.546330]  [<ffffffff81005c49>] ? pte_mfn_to_pfn+0x89/0xf0
May  7 21:35:30 test kernel: [  255.546337]  [<ffffffff81004759>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
May  7 21:35:30 test kernel: [  255.546343]  [<ffffffff8112cffa>] unmap_page_range+0x1aa/0x2f0
May  7 21:35:30 test kernel: [  255.546350]  [<ffffffff8112d6da>] unmap_vmas+0xca/0x160
May  7 21:35:30 test kernel: [  255.546357]  [<ffffffff81134ba7>] exit_mmap+0x97/0x140
May  7 21:35:30 test kernel: [  255.546363]  [<ffffffff81007b02>] ? check_events+0x12/0x20
May  7 21:35:30 test kernel: [  255.546370]  [<ffffffff810073ed>] ? xen_force_evtchn_callback+0xd/0x10
May  7 21:35:30 test kernel: [  255.546378]  [<ffffffff8105c1b2>] mmput.part.21+0x42/0x130
May  7 21:35:30 test kernel: [  255.546384]  [<ffffffff8105c2c5>] mmput+0x25/0x30
May  7 21:35:30 test kernel: [  255.546391]  [<ffffffff810629da>] exit_mm+0x12a/0x160
May  7 21:35:30 test kernel: [  255.546398]  [<ffffffff810d71e3>] ? taskstats_exit+0x1a3/0x240
May  7 21:35:30 test kernel: [  255.546407]  [<ffffffff816038e5>] ? _raw_spin_lock_irq+0x15/0x20
May  7 21:35:30 test kernel: [  255.546413]  [<ffffffff81062b8e>] do_exit+0x17e/0x440
May  7 21:35:30 test kernel: [  255.546420]  [<ffffffff8107123a>] ? __dequeue_signal+0x6a/0xb0
May  7 21:35:30 test kernel: [  255.546426]  [<ffffffff81062ff4>] do_group_exit+0x44/0xa0
May  7 21:35:30 test kernel: [  255.546433]  [<ffffffff81073a8d>] get_signal_to_deliver+0x27d/0x3f0
May  7 21:35:30 test kernel: [  255.546440]  [<ffffffff8100a7a6>] do_signal+0x56/0x180
May  7 21:35:30 test kernel: [  255.546445]  [<ffffffff8100401b>] ? xen_write_cr0+0x3b/0x90
May  7 21:35:30 test kernel: [  255.546451]  [<ffffffff81003b93>] ? xen_write_msr_safe+0xa3/0xc0
May  7 21:35:30 test kernel: [  255.546458]  [<ffffffff81009702>] ? __switch_to+0x132/0x310
May  7 21:35:30 test kernel: [  255.546465]  [<ffffffff8104db02>] ? finish_task_switch+0x52/0xf0
May  7 21:35:30 test kernel: [  255.546472]  [<ffffffff8100aa95>] do_notify_resume+0x65/0x80
May  7 21:35:30 test kernel: [  255.546478]  [<ffffffff81057767>] ? schedule_tail+0x27/0xb0
May  7 21:35:30 test kernel: [  255.546485]  [<ffffffff8160bc90>] int_signal+0x12/0x17
May  7 21:35:30 test kernel: [  255.546490] Code: 65 48 8b 04 25 80 cd 00 00 48 89 7d a8 48 89 75 a0 49 89 cc 48 89 55 98 4d 89 c7 4c 89 4d b0 48 89 45 88 0f 1f 40 00 48 8b 55 98 
May  7 21:35:30 test kernel: [  255.546576] RIP  [<ffffffff8112cac4>] zap_pte_range+0x44/0x3d0
May  7 21:35:30 test kernel: [  255.546583]  RSP <ffff8800246dba78>
May  7 21:35:30 test kernel: [  255.546589] ---[ end trace fdbba357d21adceb ]---
May  7 21:35:30 test kernel: [  255.546594] Fixing recursive fault but reboot is needed!

Hongli Lai

unread,
May 7, 2012, 5:53:59 PM5/7/12
to phusion-...@googlegroups.com
Sounds to me like a kernel bug or a hardware problem. Even if Phusion
Passenger crashes it should never cause any problems at the kernel
level.
> --
> You received this message because you are subscribed to the Google Groups
> "Phusion Passenger Discussions" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/phusion-passenger/-/a2lB3HsNt6kJ.
> To post to this group, send email to phusion-...@googlegroups.com.
> To unsubscribe from this group, send email to
> phusion-passen...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/phusion-passenger?hl=en.



--
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

Tom Harrison

unread,
May 7, 2012, 6:00:45 PM5/7/12
to phusion-...@googlegroups.com
Thank you for your quick reply -- I am dead in the water.  

I neglected to mention that I am running on EC2 using a micro-64 instance.  Ug.  

Maybe I'll try another version of Linux.

Thanks again --

Tom


On Monday, May 7, 2012 5:53:59 PM UTC-4, Hongli Lai wrote:
Sounds to me like a kernel bug or a hardware problem. Even if Phusion
Passenger crashes it should never cause any problems at the kernel
level.

> To post to this group, send email to phusion-passenger@googlegroups.com.
> To unsubscribe from this group, send email to

Tom Harrison

unread,
Jul 19, 2012, 9:45:48 AM7/19/12
to phusion-...@googlegroups.com
Greg --

I and several others have had to work around the issue: details in this thread: https://groups.google.com/forum/#!searchin/phusion-passenger/tom/phusion-passenger/gHEIYqBnxAY/6EPd-iOtNBUJ

Tom

On Wednesday, July 18, 2012 10:57:20 PM UTC-4, Greg Woods wrote:
Hi there. Did you ever find a resolution to this? I'm having a similar issue, and I also happen to be running a micro EC2 instance. Only in my case I'm running passenger on nginx, not apache, I still get very similar behavior. 

- Nginx boots up
- Static html files are accessible
- Attempting to hit any rails code just hangs and causes a crash to show up in kern.log.

Pid: 1348, comm: ruby Tainted: G      D      3.2.0-26-virtual #41-Ubuntu 

Tom Harrison

unread,
Jul 19, 2012, 4:51:50 PM7/19/12
to phusion-...@googlegroups.com
Wow, I sure wouldn't count on everything working because of availability zone -- maybe there's some system call to make to determine which version of Xen is installed (apparently that's where the bug is) ... but if it fails on restart, you're ... um ... hosed :-)
 

On Thu, Jul 19, 2012 at 1:27 PM, Greg Woods <gregw...@gmail.com> wrote:
Thanks for the responses. Based on what I read here and in the other thread Tom linked to, I ended up scrapping my instance and starting up a new one in a different availability zone*. All other factors remained the same - identical OS, identical packages installed, etc. Somehow magically everything worked from then on. 

I can only imagine that switching availability zones got me onto better hardware. Crazy! 

*For those curious, I originally had us-east-1a and switched to us-east-1d. 
--
You received this message because you are subscribed to the Google Groups "Phusion Passenger Discussions" group.
To view this discussion on the web visit https://groups.google.com/d/msg/phusion-passenger/-/tiyxE_G8lY8J.

To post to this group, send email to phusion-...@googlegroups.com.
To unsubscribe from this group, send email to phusion-passen...@googlegroups.com.

Zachary Salzbank

unread,
Apr 26, 2013, 9:43:14 AM4/26/13
to phusion-...@googlegroups.com
I'm having the same issue.  Anybody know what is going on?

[   57.048688] alignment check: 0000 [#1] SMP 
[   57.048945] CPU 0 
[   57.048950] Modules linked in: mod_usbdpfp(O) isofs acpiphp
[   57.048961] 
[   57.048965] Pid: 1045, comm: ruby Tainted: G           O 3.2.0-40-virtual #64-Ubuntu  
[   57.048973] RIP: e030:[<ffffffff8113acf4>]  [<ffffffff8113acf4>] zap_pte_range+0x44/0x3d0
[   57.048986] RSP: e02b:ffff880023ff3a58  EFLAGS: 00050292
[   57.048990] RAX: ffff880023e32dc0 RBX: ffff880024068018 RCX: 0000000000400000
[   57.048996] RDX: ffff880024068010 RSI: ffff880024071dc0 RDI: ffff880023ff3c00
[   57.049001] RBP: ffff880023ff3af8 R08: 0000000000401000 R09: 0000000000000000
[   57.049006] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000400000
[   57.049011] R13: 0000000000401000 R14: ffff8800237a9c00 R15: 0000000000401000
[   57.049020] FS:  00007f8d6b487700(0000) GS:ffff880026400000(0000) knlGS:0000000000000000
[   57.049026] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[   57.049030] CR2: 00007f8d6b2483a8 CR3: 0000000001c05000 CR4: 0000000000002620
[   57.049036] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   57.049041] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
[   57.049046] Process ruby (pid: 1045, threadinfo ffff880023ff2000, task ffff880023e32dc0)
[   57.049053] Stack:
[   57.049055]  ffff8800237a9c00 ffff8800237a9c68 ffff88000295b000 ffff880023ff3aa0
[   57.049065]  ffffffff810062ed ffff880023e32dc0 ffff880024068010 ffff880024068010
[   57.049073]  ffff880024071dc0 ffff880023ff3c00 0000000000000000 ffff880023ff3b88
[   57.049081] Call Trace:
[   57.049089]  [<ffffffff810062ed>] ? pte_mfn_to_pfn+0x8d/0x110
[   57.049095]  [<ffffffff81005309>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
[   57.049101]  [<ffffffff8113b22a>] unmap_page_range+0x1aa/0x300
[   57.049107]  [<ffffffff8113b90a>] unmap_vmas+0xca/0x1a0
[   57.049114]  [<ffffffff81142e4f>] exit_mmap+0x9f/0x140
[   57.049120]  [<ffffffff810646e2>] mmput.part.16+0x42/0x130
[   57.049125]  [<ffffffff810647f5>] mmput+0x25/0x30
[   57.049131]  [<ffffffff8106b043>] exit_mm+0x113/0x130
[   57.049138]  [<ffffffff810e2e53>] ? taskstats_exit+0x1a3/0x240
[   57.049146]  [<ffffffff816575b5>] ? _raw_spin_lock_irq+0x15/0x20
[   57.049151]  [<ffffffff8106b1ce>] do_exit+0x16e/0x450
[   57.049160]  [<ffffffff810798ca>] ? __dequeue_signal+0x6a/0xb0
[   57.049166]  [<ffffffff8106b654>] do_group_exit+0x44/0xa0
[   57.049172]  [<ffffffff8107c49c>] get_signal_to_deliver+0x21c/0x420
[   57.049180]  [<ffffffff81014825>] do_signal+0x45/0x130
[   57.049185]  [<ffffffff816574de>] ? _raw_spin_unlock_irqrestore+0x1e/0x30
[   57.049191]  [<ffffffff8165ddb0>] ? kprobe_flush_task+0xc0/0x150
[   57.049198]  [<ffffffff8105560f>] ? finish_task_switch+0xcf/0xf0
[   57.049204]  [<ffffffff81014ad5>] do_notify_resume+0x65/0x80
[   57.049210]  [<ffffffff8105faa7>] ? schedule_tail+0x27/0xb0
[   57.049216]  [<ffffffff8165fb90>] int_signal+0x12/0x17
[   57.049220] Code: 65 48 8b 04 25 00 c5 00 00 48 89 7d a8 48 89 75 a0 49 89 cc 48 89 55 98 4d 89 c7 4c 89 4d b0 48 89 45 88 0f 1f 40 00 48 8b 55 98 <48> c7 45 c4 00 00 00 00 c7 45 cc 00 00 00 00 48 8b 3a ff 14 25 
[   57.049277] RIP  [<ffffffff8113acf4>] zap_pte_range+0x44/0x3d0
[   57.049283]  RSP <ffff880023ff3a58>
[   57.049288] ---[ end trace d71a8bd6f446587a ]---
[   57.049292] Fixing recursive fault but reboot is needed!

On Wednesday, July 18, 2012 10:57:20 PM UTC-4, Greg Woods wrote:
Hi there. Did you ever find a resolution to this? I'm having a similar issue, and I also happen to be running a micro EC2 instance. Only in my case I'm running passenger on nginx, not apache, I still get very similar behavior. 

- Nginx boots up
- Static html files are accessible
- Attempting to hit any rails code just hangs and causes a crash to show up in kern.log.

Pid: 1348, comm: ruby Tainted: G      D      3.2.0-26-virtual #41-Ubuntu 

> To post to this group, send email to phusion-...@googlegroups.com.
> To unsubscribe from this group, send email to

Hongli Lai

unread,
Apr 26, 2013, 10:05:28 AM4/26/13
to phusion-passenger
Looks like a kernel bug, try a different kernel.


To unsubscribe from this group and stop receiving emails from it, send an email to phusion-passen...@googlegroups.com.

To post to this group, send email to phusion-...@googlegroups.com.
Visit this group at http://groups.google.com/group/phusion-passenger?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply all
Reply to author
Forward
0 new messages