How to get past SYSENTER?
Agoston Bejo
I'm not that much of a Windows hacker, the last time I used some debugger it
was td.exe on DOS.
Anyway, I have tried to disassembly some old DOS exe with WinDbg on my
Windows XP.
At the beginning there are a lot of calls into ntdll, then a SYSENTER
instruction. I'm trying to "trace into" it, but what happens is that the exe
itself runs to the end immediately. I found a few things about this on the
net, and it seems to me that this instruction takes the processing to the
program itself after the OS's initialization stuff. How could I acquire the
address the processor goes to so that I can finally disassemble the code of
this old exe?
Thx
Brian Catlin
news:eqnJ2FGk...@TK2MSFTNGP09.phx.gbl...
The executable is calling a system service, which is a routine in the operating
system, generally exported by the Executive, and is not part of your executable.
Disassembling the operating system isn't really going to get you anything, but
since we're on the topic, here's how it all works:
NTDLL is used to call into the operating system, which is (generally) in the
address range (0x80000000-0xFFFFFFFF). The operating system addresses are not
accessible in user-mode; therefore a special protected mechanism (using a CPU
instruction) is used to control the transition from user-mode to kernel-mode.
NTDLL loads the system service number into the EAX register, then copies the
address the processor-specific kernel-mode transition code on the Kernel-User
shared page (0x7FFE0000 + 0x300) into the EDX register, then calls through the
EDX register.
MOV EAX, Service Number
MOV EDX, MM_SHARED_USER_DATA_VA + UsSystemCall
CALL EDX
RET n
The processor-specific kernel-mode transition code depends upon whether the CPU
is Intel, AMD or Pentium2 and earlier (Win2K and earlier). INT 2E vectors
through the IDT (entry number 0x2E), while SYSCALL and SYSENTER vector through
model-specific registers that are initialized at system boot time.
Win2K and earlier:
LEA EDX, [ESP+4]
INT 2E ; Ends up calling KiSystemService
RET
WinXP and later (Intel):
MOV EDX, ESP
SYSENTER ; Ends up calling KiFastCallEntry, which then calls
KiSystemService
RET
AMD K6 and later
MOV EDX, ESP
SYSCALL ; Ends up calling KiSystemCall, which then calls
KiSystemService
RET
KiSystemService uses the system service number(in EAX) as an index into the
system service dispatch table (actually, there are up to 4), which contains the
address of the routine in the operating system to call. This prevents an
application from calling any random address in the system; an application can
only call those routines that are listed in the system service dispatch table.
This is probably way more than you wanted to know, but once I get on a roll,
it's hard to stop.
-Brian
Brian Catlin, Sannas Consulting 310-944-9492
Windows Network, Video, WDM Device Driver Training & Consulting
See WWW.AZIUS.COM.bad for courses and scheduling
REMOVE .BAD FROM EMAIL AND WEB ADDRESS
Jochen Kalmbach
> At the beginning there are a lot of calls into ntdll, then a SYSENTER
> instruction. I'm trying to "trace into" it
For this you need a kernel debugger attached to your system.
If you have WinDbg, connect it to the other PC do some settings in boot.ini
and start debugging.
--
Greetings
Jochen
Agoston Bejo
"Brian Catlin" <bri...@sannas.org.bad> wrote in message
news:%23TXX5lK...@TK2MSFTNGP10.phx.gbl...
operating
> system, generally exported by the Executive, and is not part of your
executable.
> Disassembling the operating system isn't really going to get you anything,
but
> since we're on the topic, here's how it all works:
Do you mean by 'The executable' my DOS-exe file? I also debugged it with an
ancient turbo debugger, and there weren't any SYSENTER calls in it - which
is not surprising in the case of a real-mode application.
So I supposed it wasn't the executable that called the system service but
NTVDM or whatever is doing some startup before running the exe. So I thought
I should somehow get over this startup-messing to get to the code itself.
(Actually at the end of the log file I found something that looked like the
first instruction of a real-mode program since the address of the
instruction was given in the form XXXX:XXXX which reminds me strongly to
CS:IP-like addresses.)
So, anyway, to put the question simply:
how do I debug a real-mode DOS exe with WinDbg?
Thx
Yes, your answer was deep indeed, but more is always better than none. :)
James Antognini [MSFT]
kernel space or, I believe, in any situation where the stack frame changes.
That means that even if you put a breakpoint (F9) at the right place in the
exception (or is it "interrupt"?) handler on the kernel side, WinDbg would
probably choke if you tried to instruction-step after hitting the
breakpoint. You'd also probably have the problem of catching lots of threads
hitting that breakpoint.
What I have done is to set a breakpoint in code after the frame switch, code
that is called by the switcher. What I add is a statement to make the
breakpoint conditional on a match of whatever thread (or process, if there's
only one thread of interest in the process) I'm debugging. That way
(skipping across the sensitive transition code and matching the thread),
I've successfully used WinDbg across user-kernel-user transitions.
--
James Antognini
Windows DDK Support
This posting is provided "AS IS" with no warranties, and confers no rights.
"Agoston Bejo" <gu...@freemail.hu> wrote in message
news:eqnJ2FGk...@TK2MSFTNGP09.phx.gbl...
Pavel Lebedinsky
> So, anyway, to put the question simply:
> how do I debug a real-mode DOS exe with WinDbg?
As far as I know, there is no way to do that. WinDbg
can only debug win32 (or win64) code.
Pavel Lebedinsky
where stepping over a sysenter instruction resulted in an
access violation or something like that.
I have SP2 here and I can trace through a sysenter with no
problems (of course, this simply steps over the system call,
you can't "step into" it from a user mode debugger).