A long strange trip through WinDbg all because of a powershell script, followed by some linkdumps

[Justin Dearing]

I want to write up some blog posts, but that might not happen. I figured I'd put in the long unedited form here. Hopefully I can edit it later, or at least someone can find this via google if it wold benefit them.

It all started when I wrote a powershell script to update the diff and merge tools in Visual Studio. During that time I discovered that while VS2010 has exes for trhe builtin diff and merge, in 2012 it was a DLL in the GAC:

Microsoft.VisualStudio.TeamFoundation.VersionControl, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a

So I spent some time looking through the decompiled sources of that dll with ILSpy. I found the diff and merge classes, but I didn't know where the hooks were if I wanted to write my own diff and merge dll. 

I then got the crazy idea that maybe I could debug devenv.exe, set a Conditional breakpoint  to RegOpenKeyEx pointing to the registry key that pointed to the diff program, and watch the diff class get initiated.

Well I figured out how to do that with Visual Studio developer, WinDbg, and Rohits API Mointor. Here are some links relating to that:

• http://stackoverflow.com/questions/16287930/how-do-i-set-a-breakpoint-in-managed-code-based-on-a-registry-key-being-read
  
• http://reverseengineering.stackexchange.com/questions/1979/how-do-i-see-the-parameters-passed-to-regopenkeyex-and-set-a-conditional-breakp
  
• http://www.zachburlingame.com/2011/09/setting-a-debug-breakpoint-in-a-win32-api-in-visual-studio/
  

However, in order to do this, I needed the pdb symbols for advapi32.dll. So I had to learn how to use the NT Symbol Server. So I set the environment variable _NT_SYMBOL_PATH. During all this I figured out how to use dumpbin to list the decorated function names with the symbols.

Then I discovered on Paul Randal's blog, callstacks generated in SQL Server don't seem to respect the symbol server. Jonathan Kehayias showed an easy way to generate call stacks in the ringbuffer. I did some inconclusive research, but from what I've discovered, it should work.

So anyway, I have not actually fully solved either original problem (reverse engineering the DLL entry point for diff merge in VS2012, and using the symbol server for ringbuffer stack traces in Sql 2012), but I've gotten more comfortable with windbg, 

Finally a linkdumo

• http://windbg.info - excellent windbg docs
  
• http://windbg.org - windbg links
  
• http://reverseengineering.stackexchange.com - finally windows internals knowledge will become googable. Got some heavy hitters actively contributing to the site.
  
• SOSEX.dll - a windbg plugin for .NET development. An enhancement of the microsoft provided sos.dll
  
• http://www.digitaloperatives.com/paint/index.html Wireshark mods that let you associate processes with traffic. Windwos only.
  
• https://code.google.com/p/windbgshark/ wndbg interface for wireshark