How to get past SYSENTER? - microsoft.public.windbg

Message from discussion How to get past SYSENTER?

Brian Catlin

View profile

More options Sep 2 2004, 12:58 am

Newsgroups: microsoft.public.windbg

From: "Brian Catlin" <bri...@sannas.org.bad>

Date: Wed, 1 Sep 2004 21:58:21 -0700

Local: Thurs, Sep 2 2004 12:58 am

Subject: Re: How to get past SYSENTER?

  Reply to author
  |
  Forward
  | Print
  | View thread
  | Show original
  | Report this message
  | Find messages by this author
  

"Agoston Bejo" <gu...@freemail.hu> wrote in message

news:eqnJ2FGkEHA.3372@TK2MSFTNGP09.phx.gbl...

> Hi,
> I'm not that much of a Windows hacker, the last time I used some debugger it
> was td.exe on DOS.
> Anyway, I have tried to disassembly some old DOS exe with WinDbg on my
> Windows XP.
> At the beginning there are a lot of calls into ntdll, then a SYSENTER
> instruction. I'm trying to "trace into" it, but what happens is that the exe
> itself runs to the end immediately. I found a few things about this on the
> net, and it seems to me that this instruction takes the processing to the
> program itself after the OS's initialization stuff. How could I acquire the
> address the processor goes to so that I can finally disassemble the code of
> this old exe?

The executable is calling a system service, which is a routine in the operating
system, generally exported by the Executive, and is not part of your executable.
Disassembling the operating system isn't really going to get you anything, but
since we're on the topic, here's how it all works:

NTDLL is used to call into the operating system, which is (generally) in the
address range (0x80000000-0xFFFFFFFF). The operating system addresses are not
accessible in user-mode; therefore a special protected mechanism (using a CPU
instruction) is used to control the transition from user-mode to kernel-mode.
NTDLL loads the system service number into the EAX register, then copies the
address the processor-specific kernel-mode transition code on the Kernel-User
shared page (0x7FFE0000 + 0x300) into the EDX register, then calls through the
EDX register.

MOV EAX, Service Number
MOV EDX, MM_SHARED_USER_DATA_VA + UsSystemCall
CALL EDX
RET n

The processor-specific kernel-mode transition code depends upon whether the CPU
is Intel, AMD or Pentium2 and earlier (Win2K and earlier). INT 2E vectors
through the IDT (entry number 0x2E), while SYSCALL and SYSENTER vector through
model-specific registers that are initialized at system boot time.

Win2K and earlier:
LEA EDX, [ESP+4]
INT 2E ; Ends up calling KiSystemService
RET

WinXP and later (Intel):
MOV EDX, ESP
SYSENTER ; Ends up calling KiFastCallEntry, which then calls
KiSystemService
RET

AMD K6 and later
MOV EDX, ESP
SYSCALL ; Ends up calling KiSystemCall, which then calls
KiSystemService
RET

KiSystemService uses the system service number(in EAX) as an index into the
system service dispatch table (actually, there are up to 4), which contains the
address of the routine in the operating system to call. This prevents an
application from calling any random address in the system; an application can
only call those routines that are listed in the system service dispatch table.

This is probably way more than you wanted to know, but once I get on a roll,
it's hard to stop.

-Brian

Brian Catlin, Sannas Consulting 310-944-9492
Windows Network, Video, WDM Device Driver Training & Consulting
See WWW.AZIUS.COM.bad for courses and scheduling
REMOVE .BAD FROM EMAIL AND WEB ADDRESS

Reply to author Forward