UMDH on Windows 2003 x64 [stack tracing issues]

[Complet...@gmail.com]

Hi,

I'm trying to use UMDH to diagnose possible leaks in my 32 bit
application running on Win2k3 x64, and am finding it unable to detect
that stack tracing has already been enabled.

1. I have already used gflags /i myapp.exe +ust to enable stack
tracing for the application, and checked the registry Image settings
for the application in question (they read right, 000001000).

2. -After- setting the above, I have launched the application afresh.

3. I have set the symbol store to _NT_SYMBOL_PATH=SRV*c:\windows
\symbols*http://msdl.microsoft.com/download/symbols

[C:\windows\symbols has the retail symbols for Win2k3 x64 preinstalled
anyway]

3. When I run umdh -p:PID -d -v - I get the following
response
**Error: Stack trace collection is not enabled for this process.
Please use the gflags tool with the +ust option to enable it.**

4. (After reading Ivan [MSFT]'s suggestion on usenet) When I run umdh
in mode 3 with umdh -snap PID -file f1.dat - I get the following
response

saved information for 5 loaded modules
saved information for 0 stack traces
saved 0000000b Allocations in 2 heaps

This makes very little sense, and is almost useless - the app has
hundreds/thousands of allocations, being an IOCP server that uses
pools of sockets, and other per-socket and per-io structs.

5. I tried LeakDiag, with no useful results - setup the web based
symbol store, turned on leak checking for the C Runtime Heap, and set
the call stack depth count to 32, but all it gave me was a 5 depth
call stack, ending in an msvcrt call and never touching my program
explicitly (despite the symbols - PDB - being available)

UMDH worked fine on the Win32 machine I used to test on -
unfortunately, to really test for leaks I have to use this x64
machine.

Any ideas on what I can do to fix this? Any help/suggestions
appreciated. Tools like boundchecker etc. are practically useless when
you want to instrument a high performance server - they slow it down
to the point of unusability.

Thank You,
CompletionPort

[Skywing [MVP]]

Are you using the x86 or x64 UMDH?

--
Ken Johnson (Skywing)
Windows SDK MVP
http://www.nynaeve.net
<Complet...@gmail.com> wrote in message
news:1185448954.7...@j4g2000prf.googlegroups.com...

[Complet...@gmail.com]

Thanks for the response - its the x64 Debugging Tools for Windows
package, so I presume that is UMDH x64.
(C:\Program Files\Debugging Tools for Windows 64-bit>)

Windbg runs fine on the same process, once I use the !effmach x86 or !
wow64exts.sw functions.

One small additional bit of info. The process being debugged is a
usermode service. I can launch it as a usermode application too - this
fails with the same issues. Mentioned this since services run in the
Console session so you can't attach to them with LeakDiag from a
session other than the console session.

On Jul 26, 7:54 pm, "Skywing [MVP]"

<skywing_NO_SP...@valhallalegends.com> wrote:
> Are you using the x86 or x64 UMDH?
>
> --
> Ken Johnson (Skywing)

> Windows SDK MVPhttp://www.nynaeve.net<Completion.P...@gmail.com> wrote in message

[Mario Hewardt [MSFT]]

Have you tried running the x86 version of the tools (gflags, umdh)?

<Complet...@gmail.com> wrote in message
news:1185466737....@d30g2000prg.googlegroups.com...

[naeem ahmad]

Hi Guys,
I am getting similar problem using umdh 64 bit on x64 machine.

I am trying to use umdh(64 bit) to find memory leak in x64 application,
gflags -i leakapp.exe +ust
umdh -p:? -f:out1.log
It does not show symbols information for stack.
If i build 32 bit version of my application, use 32 bit umdh it works fine
on x64. shows symbols information for stack trace.

Thanks
Naeem

[Complet...@gmail.com]

Mario - Thanks for that tip.

I ran the x86 version of gflags + umdh after disabling the 64 bit
gflags for the app, and allowed it to run for several hours - only to
bump up against a fresh issue. Note that I checked that umdh was
picking up the right symbols for the application (symopt 0x80000000
and checking stack traces to verify that app-specific function names
were showing up).

The leak I found - 143 MB in this time, apparently (believable).
+ 143724226 ( 153787752 - 10063526) 78405 allocs BackTrace00377

1) Now here is the weird part - if I look at BackTrace0377, below is
the call stack I see, for every -single- allocation. Some allocations
allocate around 0x1048 bytes, some 0xCx bytes, and some 0x78 bytes as
below.

The call stack shows nothing from my application itself.

00000078 bytes in 0x1 allocations (@ 0x00000018 + 0x00000018) by:
BackTrace00377
ntdll!RtlAllocateHeapSlowly+00000041
ntdll!RtlAllocateHeap+00000E9F
MSVCR80!malloc+0000007A

Any idea what is happening? Mine is obviously a native Win32
application that uses nothing but Winsock, IOCP, WinInet (very
little).

2) As I recall, UMDH used to group the allocations for a single
callstack together - just dump one callstack trace for a backtrace,
list the size involved and list all the memory locations corresponding
to that backtrace. This isn't happening, and this seems to indicate
that UMDH is unable to resolve the memory allocations at some point
into its components (each kind of allocation) allocated at different
points in my code. Perhaps this is a hint as to what is going wrong
above?

Thanks!

On Jul 26, 10:11 pm, "Mario Hewardt [MSFT]"

<mar...@online.microsoft.com> wrote:
> Have you tried running the x86 version of the tools (gflags, umdh)?
>

[Complet...@gmail.com]

Hi Naeem,

Thought this might help -

Try setting -symopt:0x80000000 in the umdh commandline, and scrolling
through the umdh output, check if it has found the symbol (PDB) for
your application.

Had to use this myself to get umdh to locate the symbols for my
leaking app.

On Jul 27, 2:20 pm, naeem ahmad <naeemah...@discussions.microsoft.com>
wrote:

> Hi Guys,
> I am getting similar problem using umdh 64 bit on x64 machine.
>
> I am trying to use umdh(64 bit) to find memory leak in x64 application,
> gflags -i leakapp.exe +ust
> umdh -p:? -f:out1.log
> It does not show symbols information for stack.
> If i build 32 bit version of my application, use 32 bit umdh it works fine
> on x64. shows symbols information for stack trace.
>
> Thanks
> Naeem
>
>
>
>
>
> "Mario Hewardt [MSFT]" wrote:
> > Have you tried running the x86 version of the tools (gflags, umdh)?
>

[Skywing [MVP]]

The stack trace logging code does not operate with symbols. Instead for x86
it simply assumes that EBP is essentially a linked list of call frames
(return address / frame pointer) that it follows until the internal limit on
frames is reached or it runs into a next frame link (e.g. saved EBP) value
that doesn't appear to make sense.

UMDH asks the OS (NTDLL in this case) for the saved stack traces and then
uses symbols to translate them to functions after the fact. So the stack
traces you get with UMDH are not as "high-quality" as stack traces that you
would get for a debugger that is using symbols while recording the stack
traces (OTOH, the OS's built in stack tracing mechanism is much faster and
much less invasive than that).

What this means for you is that functions that do not use EBP will often
confuse or break entirely the stack tracing mechanism on x86. There are
some other cases that can confuse it, but functions that don't use EBP are
typically what ruins most stack traces for UMDH in my experience - to an
extent this can be controlled by setting the appropriate compiler option
(/Oy- if I am remembering correctly) to not remove frame pointers during
optimization, but some functions that do not use the stack at all might
still not use EBP and can still break the stack trace code if they are in
the allocation call stack.

--
Ken Johnson (Skywing)
Windows SDK MVP
http://www.nynaeve.net

<Complet...@gmail.com> wrote in message
news:1185529277.4...@m37g2000prh.googlegroups.com...

[Complet...@gmail.com]

Very interesting insight. Thank you.

You mentioned that functions that don't use the stack won't use EBP
(and hence will break the chain of EBP values). Is there any other
behaviour a function might exhibit that might result in its not using
EBP ? Would be nice to know this - and it might help narrow down on
the functions that might be leaking.

A slightly ugly workaround might be to add a simple local variable
(dummy) to each function, and reference it somewhere in the code of
the function with a debug trace.

Disabling FPO - I will give that a shot. Is there any other workaround
you (or anybody else) think might work? No obstacle to my running the
debug build ...could try that.

Also - the build in OS stack tracing - just curious as to why this
isn't used by UMDH (for instance).

On Jul 27, 8:26 pm, "Skywing [MVP]"

<skywing_NO_SP...@valhallalegends.com> wrote:
> The stack trace logging code does not operate with symbols. Instead for x86
> it simply assumes that EBP is essentially a linked list of call frames
> (return address / frame pointer) that it follows until the internal limit on
> frames is reached or it runs into a next frame link (e.g. saved EBP) value
> that doesn't appear to make sense.
>
> UMDH asks the OS (NTDLL in this case) for the saved stack traces and then
> uses symbols to translate them to functions after the fact. So the stack
> traces you get with UMDH are not as "high-quality" as stack traces that you
> would get for a debugger that is using symbols while recording the stack
> traces (OTOH, the OS's built in stack tracing mechanism is much faster and
> much less invasive than that).
>
> What this means for you is that functions that do not use EBP will often
> confuse or break entirely the stack tracing mechanism on x86. There are
> some other cases that can confuse it, but functions that don't use EBP are
> typically what ruins most stack traces for UMDH in my experience - to an
> extent this can be controlled by setting the appropriate compiler option
> (/Oy- if I am remembering correctly) to not remove frame pointers during
> optimization, but some functions that do not use the stack at all might
> still not use EBP and can still break the stack trace code if they are in
> the allocation call stack.
>
> --
> Ken Johnson (Skywing)

[Skywing [MVP]]

UMDH does use the built-in stack tracing, which is what is limited to not
work with symbols. This is because it operates in fairly restricted
environments - i.e. it may have no Internet access, it might be in kernel
mode (i.e. for handle operation tracing), there might not even be symbols
available for relevant modules (i.e. third party DLLs crossing the stack
trace). It just doesn't work out very well to try and use PDBs for stack
tracing with respect to how the stack trace logging code works and where it
is used.

N.B. The compiler will typically optimize away obviously untouched locals,
beware.

--
Ken Johnson (Skywing)

Windows SDK MVP
http://www.nynaeve.net
<Complet...@gmail.com> wrote in message

news:1185552407.1...@d30g2000prg.googlegroups.com...

[Jochen Kalmbach [MVP]]

Hi Completion!

>> What this means for you is that functions that do not use EBP will often
>> confuse or break entirely the stack tracing mechanism on x86. There are
>> some other cases that can confuse it, but functions that don't use EBP are
>> typically what ruins most stack traces for UMDH in my experience - to an
>> extent this can be controlled by setting the appropriate compiler option
>> (/Oy- if I am remembering correctly) to not remove frame pointers during
>> optimization, but some functions that do not use the stack at all might
>> still not use EBP and can still break the stack trace code if they are in
>> the allocation call stack.

If you are using CRT, you can use my LeakFinder, whcih also works for
x86/x64/IA64:
http://blog.kalmbachnet.de/files/LeakFinder-RC5.zip

It is derived from my StackWalker:
http://www.codeproject.com/threads/StackWalker.asp

and my previous Leakfinder:
http://www.codeproject.com/tools/leakfinder.asp

it saves the (complete) callstack during allocation and at termination
of the program (or if you trigger it), it will show all not-yet-freed
allocation with the complete callstack using dbghelp-api (and of course,
symbols).

This will mainly serve your needs...

--
Greetings
Jochen

My blog about Win32 and .NET
http://blog.kalmbachnet.de/

[Complet...@gmail.com]

Thanks Jochen :) Will try it out and let you know how it goes.

On Jul 27, 10:03 pm, "Jochen Kalmbach [MVP]" <nospam-

[Complet...@gmail.com]

I see.

Will try with the debug build as well and see how it goes...

On Jul 27, 9:46 pm, "Skywing [MVP]"

<skywing_NO_SP...@valhallalegends.com> wrote:
> UMDH does use the built-in stack tracing, which is what is limited to not
> work with symbols. This is because it operates in fairly restricted
> environments - i.e. it may have no Internet access, it might be in kernel
> mode (i.e. for handle operation tracing), there might not even be symbols
> available for relevant modules (i.e. third party DLLs crossing the stack
> trace). It just doesn't work out very well to try and use PDBs for stack
> tracing with respect to how the stack trace logging code works and where it
> is used.
>
> N.B. The compiler will typically optimize away obviously untouched locals,
> beware.
>
> --
> Ken Johnson (Skywing)