Debugging on windows

[Byron Hawkins]

I'm having some trouble getting cdb to start a program under DynamoRIO. My command line in msys looks like this (line breaks added to make it readable):

/wkits/8.0/Debuggers/x64/cdb.exe  

-cf /dev/crowd-safe-dynamo-rio/build/break-on-entry.cdb

/dev/crowd-safe-dynamo-rio/build/bin64/drinject.exe

/dev/crowd-safe-dynamo-rio/build/lib64/debug/dynamorio.dll

"/bin/ls.exe"

The script break-on-entry.cdb contains:

.sympath /dev/crowd-safe-dynamo-rio/build/lib64/debug;srv*/dev/lib/windows-symbols*http://msdl.microsoft.com/download/symbols

.reload dynamorio.dll=15000000

$></dev/crowd-safe-dynamo-rio/tools/windbg-scripts/load_syms64

.childdbg 1

g

l+t

l+s

bp dynamo_auto_start

g

There are two problems:

1. Somehow my dynamorio.dll is not acceptable to cdb. The error message is strange because it mentions "Win32", even though it is the x64 build of cdb. The response to the first "g" command is:

0:000> g

ERROR: Failed to create process for "C:\dev\crowd-safe-dynamo-rio\build\lib64\debug\dynamorio.dll": %1 is not a valid Win32 application.

ntdll!ZwTerminateProcess+0xa:

2. The $DYNAMORIO_HOME/tools/load_syms64 script is causing an error:

0:000> $></dev/crowd-safe-dynamo-rio/tools/windbg-scripts/load_syms64

 

Memory access error at ') & 0xfffff000);;$$ Check magic values to avoid executing random command w/o DynamoRIO;.if (dwo(@$t1) = b1d2ae58) {;    .if (dwo(@$t1 +

4) = ca50c356) {;        .if (dwo(@$t1 + 8) = 63000089) {;            .if (dwo(@$t1 + c) = 3fa898f0) {;                aS /c ${/v:loadpriv} .printf "%ma", @$t1

+ 40;                .block { ${loadpriv} };                ad ${/v:loadpriv};            } .else {;                .echo "DynamoRIO not detected";            };        } .else {;            .echo "DynamoRIO not detected";        };    } .else {;        .echo "DynamoRIO not detected";    };} .else {;    .echo "DynamoRIO not detected";};'

The environment here is Windows 7 64-bit on a 64-bit Intel processor, with all commands executed from msys on the x64 cdb. It looks like cdb is handling the msys paths ok, since it will complain about file not found if I give an incorrect path in the same format. 

So far I've tried several variations of these commands, but I keep getting the exact same errors. Please let me know if anyone is familiar with windows debugging and can see what might be wrong here. Thanks.

Byron

[Derek Bruening]

You're trying to execute dynamorio.dll as though it were an executable, hence the error message.  If you want to just examine a dll in these debuggers you want to open it as a crash dump.  Probably you just forgot the right args to drinject since you seem to want to run a new process, not examine a dll in isolation.

- Derek

Byron

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To post to this group, send email to dynamor...@googlegroups.com.
Visit this group at http://groups.google.com/group/dynamorio-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Byron Hawkins]

Well, I did actually try it without having dynamorio.dll on the command line, and still got the same result. My launch command line was taken from the DynamoRIO wiki, which suggests:

% cd $DYNAMORIO_HOME/exports/lib32
% windbg ./drinject.exe c:\\path\\to\\exports\\lib32\\dynamorio.dll c:\\path\\to\\targetapp.exe

(URL is http://code.google.com/p/dynamorio/wiki/Debugging#Launching_Within_WinDbg). Have I misunderstood how this is meant to work? It does look like the dynamorio.dll is given as a kind of target, though like I say, it also doesn't work when I load it only from within the cdb script. 

Byron

[Derek Bruening]

See http://dynamorio.org/docs/page_deploy.html#win_deploy for how to run an application under DR.  That wiki command line you reference with drinject is very old and out of date.  Look at the drinject usage message to see what args it takes.  I just updated the wiki with drrun syntax.

No native-windows tool understands "/bin/ls.exe" as being "c:/cygwin/bin/ls.exe" or "c:/mingw/bin/ls.exe".  None.  You can use forward slashes in most cases (though not for the app to run in cmd, nor in most FIle Open dialogs, etc.) but you can't skip the cygwin or whatnot prefix (unless you went and put NTFS junctions/symlinks in the root dir yourself).

- Derek

[Byron Hawkins]

So far I am able to run any program under DR, but I can't get a breakpoint. I really need to get a breakpoint so I can step through and see how things start up in the windows environment. My modifications to DR are not being invoked, and I can't even guess what's happening without getting a breakpoint and stepping through line by line to see what's happening. 

This command works to launch ls.exe under DR under cdb:

cdb /dev/crowd-safe-dynamo-rio/build/bin64/drrun.exe -debug -- /bin/ls.exe

All I do is enter g from the cdb prompt, and it runs ls.exe with the expected output (msys translates the path to c:\... on the way in). But I can't get a breakpoint. Every attempt to set a breakpoint gets the response:

Bp expression 'dynamo_auto_start' could not be resolved, adding deferred bp

*** Bp expression 'dynamo_auto_start' contains symbols not qualified with module name.

I've tried this with all different breakpoint addresses, but cdb does not understand what I am saying. It's as if dynamorio.dll is not loaded--though if the .sympath does not reach dynamorio.dll, then drrun complains and exits--so it must be loading dynamorio.dll, right? I've been working on this for several days now, and I just can't get a single thing to respond in cdb. Please let me know if there is a simple "abc" recipe for getting cdb to run an app under DR and stop at a breakpoint, so that I can step along and see what it's doing. Sorry if it's a trivial problem--I really appreciate your help.

Byron

[Qin Zhao]

I only tried debug using windbg.

Some debug tips can be found at
http://code.google.com/p/dynamorio/wiki/Debugging

Interested in Yoga? Be careful of The Yoga Cult or The Scary Yoga Obsession.

More information from  Lorie Anderson and Rick Ross.

[Derek Bruening]

You're running drrun.exe initially.  It does not load dynamorio.dll.  Hence the debugger must use a deferred bp if you add it while inside drrun.  Deferred bp's should work: and you never said they didn't, you just complained about the message from cdb.  You can also use a module load event to see dynamorio.dll loaded and add bp's there.  Both should work for initial drrun injection, which is actually quite late (for historic reasons it's not using the earlier from-parent injection).

load_syms* will not work until DR finishes initializing.

Note that planned future improvements in injection will make both deferred bp's and module load events fail as DR will not be loaded by the system loader and you'll have to watch for the map.

- Derek

[Byron Hawkins]

Yes, I've tried every debugging tip on the wiki, but I can't get even one command to respond. The debuggers cdb and windbg are identical under the hood, cdb just runs in the command prompt instead of opening its own window. How do you get breakpoints in windbg? Can you give me a list of steps?

[Byron Hawkins]

The deferred breakpoints don't work. Here's what happens after setting the breakpoint and pressing "g":

*** Unable to resolve unqualified symbol in Bp expression 'dynamorio_app_init' from module 'C:\windows\system32\apphelp.dll'.

*** Unable to resolve unqualified symbol in Bp expression 'dynamo.c:200' from module 'C:\windows\system32\apphelp.dll'.

ModLoad: 000007fe`fd480000 000007fe`fd4d7000   C:\windows\system32\apphelp.dll

*** Unable to resolve unqualified symbol in Bp expression 'dynamorio_app_init' from module 'image00000000`00b90000'.

*** Unable to resolve unqualified symbol in Bp expression 'dynamo.c:200' from module 'image00000000`00b90000'.

ModLoad: 00000000`00b90000 00000000`00bac000   image00000000`00b90000

I've added some printf statements to dynamorio_app_init(), but nothing is printed to the command line. Changes to drdeploy.c are visible, but changes to sources that compile into dynamorio.dll are somehow not executed. There are no other instances of dynamorio.dll on my machine, and the timestamp of the file shows that it is current. Grep shows that my printf statements are compiled into the dynamorio.dll, yet somehow nothing is printed when I run under DR. 

Setting breakpoints for other programs is working like normal in cdb. For example, I can run ls.exe in cdb, and it stops at program entry and allows me to step through. But when I execute the identical command prefixed with "drrun.exe -debug", then cdb will execute the target program without stopping at any breakpoints. 

It really seems like something is wrong with the way I have things set up here, but I can't seem to find a starting place to explore from. Everything simply fails to connect. Code changes to dynamorio.dll go into the binary, but do not execute. Breakpoints are ignored. I tried all the same steps in windbg in case it is somehow different, but the results are the same. Let me know if you have any suggestions.

[Derek Bruening]

I think the problem you're hitting is that you're trying to set the deferred breakpoint in the parent and have it still be there in the child.  While regular breakpoints will be preserved, I believed deferred are reset once in the child.  That's why the wiki command sequence waits for the initial default bp in the child before setting the deferred bp.  Follow the wiki sequence exactly and it should work (it does for me with cdb 6.3).

You can make the unqualified warnings go away by qualifying the breakpoint (i.e., prefix "dynamorio!").

I don't know what you mean by "printf" (hopefully not literal "printf"), but note that raw DR-style output will not show up in cmd windows.

- Derek

[Byron Hawkins]

I'm using dr_printf, and that's good to know it doesn't show up on the command line (because it isn't). But if I put infinite loops in dynamorio.dll, the loops are not executed. For example, I put an infinite loop in dispatch(), and still the program runs like normal. It really seems to be running the target program without invoking any part of dynamorio.dll. When I give arguments like "-stats", it prints the stats, and they look normal for the target program. But nothing I put into dynamorio.dll seems to be in effect. The drrun.exe binary is definitely the one built from my code, because it does show changes to the command line output (for example, changes to the "usage" text appear). Is it possible that my drrun.exe is somehow just skipping dynamorio altogether and running the target program natively?

Meanwhile I'll try the breakpoints one more time.

[Byron Hawkins]

I finally found the problem: all my target apps were 32-bit, even though they were installed from a 64-bit package. So the OS was jumping to wow64 and never loading dynamorio.dll. It didn't even load drconfiglib.dll, just went to wow64 and executed the app natively. I found some 64-bit targets and now my breakpoints are working :-) Thanks for pointing out that breakpoints are reset after the child process starts, I was missing that point also. 

Byron

[Derek Bruening]

drrun today prints "ERROR: Target process <path> is for the wrong architecture" in such a situation.  Are you running an old version that doesn't have that message?

http://code.google.com/p/dynamorio/issues/detail?id=803 will make 64-to-32 just work.

[Byron Hawkins]

My build is based on r1822 from 2013-01-14, and it looks like this change was not there yet. I don't merge often because I have a lot of changes, but maybe I should keep up :-) Anyway, I learned a lot of good stuff about how windows launches an app. Thanks for your help.

Byron