Re: [modwsgi] configuration corruption in memory (directives applied twice)

[Graham Dumpleton]

The dual call to ap_process_config_tree() is normal and has never cause an issue with Apache 1.3, 2,0 and 2.2.

What I would suggest first is that rather than use mod_wsgi 3.4, that you grab down from the source code repository either 3.5 or 4.0. There are some changes in those to address core dumps on Apache 2.4, although the trigger for those is not configuration but related to lingering close of connections.

http://code.google.com/p/modwsgi/wiki/ChangesInVersion0305

http://code.google.com/p/modwsgi/wiki/ChangesInVersion0400

So lets eliminate that as potential problem first.

If still get it, then will need to get some stack traces by using methods described in:

http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Debugging_Crashes_With_GDB

Graham

On 19 December 2012 11:53, Harry Kao <ahai...@gmail.com> wrote:

I've got a weird problem.  I believe what's happening is that the apache configuration directives are all applied twice.  In the Apache 2.4.2 code, main() has two calls to ap_process_config_tree().  Between the two calls the original APR pool seems to get released so anything allocated from it (namely wsgi_daemon_list) gets munged which, for me, somehow ends in a fork bomb.

Any idea why this is happening and what I can do about it?  (The two calls to the config processor with the pool release in between, I mean.)  I must be doing something strange since nobody else is hitting it.

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To view this discussion on the web visit https://groups.google.com/d/msg/modwsgi/-/afKn-FDvKBgJ.
To post to this group, send email to mod...@googlegroups.com.
To unsubscribe from this group, send email to modwsgi+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.

[Harry Kao]

Thanks for the quick reply.  I tried sticking mod_wsgi 4.0 in and it didn't build.  I'm working on an embedded system so building and debugging are more involved than normal.  I'm willing to dig into it more but I likely won't have time for a few days.

I think the problem lies elsewhere though.  It doesn't result in a crash so there's no backtrace in the end.  However, here's the backtrace for the first call into wsgi_add_daemon_process():

#0  wsgi_add_daemon_process (cmd=0x7fffffffdb10, mconfig=0x8d7ee8,

    args=0x863a20 "rvbd_web python-path=/opt/tms/web3/dj:/usr/lib/python2.6/site

#1  0x000000000042b97e in invoke_cmd (cmd=0x54ceb0, parms=0x7fffffffdb10,

    mconfig=0x8d7ee8,

    args=0x863a20 "rvbd_web python-path=/opt/tms/web3/dj:/usr/lib/python2.6/site

#2  0x000000000042d473 in ap_walk_config_sub (current=0x8639d8,

    parms=0x7fffffffdb10, section_vector=0x8d7a70) at config.c:1296

#3  ap_walk_config (current=0x8639d8, parms=0x7fffffffdb10,

    section_vector=0x8d7a70) at config.c:1329

#4  0x000000000041ef6e in virtualhost_section (cmd=0x7fffffffdb10, 

    dummy=<value optimized out>, arg=<value optimized out>) at core.c:2598

#5  0x000000000042b97e in invoke_cmd (cmd=0x51cf50, parms=0x7fffffffdb10,

    mconfig=0x861d60, args=0x863928 "_default_:80>") at config.c:874

#6  0x000000000042d473 in ap_walk_config_sub (current=0x8638e0,

    parms=0x7fffffffdb10, section_vector=0x85ed28) at config.c:1296

#7  ap_walk_config (current=0x8638e0, parms=0x7fffffffdb10, 

    section_vector=0x85ed28) at config.c:1329

#8  0x000000000042d8f9 in ap_process_config_tree (s=<value optimized out>,

    conftree=<value optimized out>, p=0x7a0138, ptemp=<value optimized out>)

    at config.c:2053

#9  0x000000000040e566 in main (argc=5, argv=0x7fffffffdd88) at main.c:639

That initializes wsgi_daemon_list and pushes a WSGIProcessGroup onto it.  So far, so good.  I put a watchpoint on the "name" member, which is one of the parts that gets corrupted.  The watchpoint triggers when that memory gets overwritten:

#0  apr_pool_create_ex (newpool=0x7fffffffdb68, parent=0x7a0138, 

    abort_fn=<value optimized out>, allocator=<value optimized out>)

    at memory/unix/apr_pools.c:935

#1  0x00000000004f4c83 in sort_hook () at hooks/apr_hooks.c:195

#2  apr_hook_sort_all () at hooks/apr_hooks.c:252

#3  0x000000000040e765 in main (argc=5, argv=0x7fffffffdd88) at main.c:731

So somewhere between main.c:639 and main.c:731 that memory is probably released.  After this, wsgi_add_daemon_process() is called a second time:

#0  wsgi_add_daemon_process (cmd=0x7fffffffdb10, mconfig=0x83adb8, 

    args=0x864610 "rvbd_web python-path=/opt/tms/web3/dj:/usr/lib/python2.6/site

#1  0x000000000042b97e in invoke_cmd (cmd=0x54ceb0, parms=0x7fffffffdb10,

    mconfig=0x83adb8,

    args=0x864610 "rvbd_web python-path=/opt/tms/web3/dj:/usr/lib/python2.6/site

#2  0x000000000042d473 in ap_walk_config_sub (current=0x8645c8,

    parms=0x7fffffffdb10, section_vector=0x83a940) at config.c:1296

#3  ap_walk_config (current=0x8645c8, parms=0x7fffffffdb10, 

    section_vector=0x83a940) at config.c:1329

#4  0x000000000041ef6e in virtualhost_section (cmd=0x7fffffffdb10, 

    dummy=<value optimized out>, arg=<value optimized out>) at core.c:2598

#5  0x000000000042b97e in invoke_cmd (cmd=0x51cf50, parms=0x7fffffffdb10,

    mconfig=0x7e31e8, args=0x864518 "_default_:80>") at config.c:874

#6  0x000000000042d473 in ap_walk_config_sub (current=0x8644d0,

    parms=0x7fffffffdb10, section_vector=0x7e43f8) at config.c:1296

#7  ap_walk_config (current=0x8644d0, parms=0x7fffffffdb10,

    section_vector=0x7e43f8) at config.c:1329

#8  0x000000000042d8f9 in ap_process_config_tree (s=<value optimized out>,

    conftree=<value optimized out>, p=0x7a0138, ptemp=<value optimized out>)

    at config.c:2053

#9  0x000000000040e7dc in main (argc=5, argv=0x7fffffffdd88) at main.c:739

I took a closer look at what's happening between lines 639 and 731.  On 702 there's a apr_pool_destroy(ptemp).  ptemp is the pool that was passed to ap_process_config_tree() the first time.  That seems bad, right?

Harry

[Graham Dumpleton]

Can you try mod_wsgi 3.5 instead then. There are a lot more substantial changes in 4.0 which could cause other issues.

I'll look at the memory pool management and get back to you.

Graham

To view this discussion on the web visit https://groups.google.com/d/msg/modwsgi/-/Low2p5SgQD4J.

[Harry Kao]

I just tried 3.5 and I'm seeing the same behavior.

Harry