Debugging bug check 0x3F and 0xD8 via kernel mini-dumps.
Olegas
I would like to ask a few questions about whether or not it is possible to
get the offending driver’s name from kernel mini-dumps (88 KB).
Here is the situation:
We’ve encountered reports of BSOD with bug check 0x3F. Subsequently, we’ve
enabled PTE tracking as described in http://support.microsoft.com/kb/256004
After enabling tracking, the issue reoccurred and we got another 88 KB
mini-dump with the following output:
0: kd> !analyze -v
*******************************************************************************
*
*
* Bugcheck Analysis
*
*
*
*******************************************************************************
DRIVER_USED_EXCESSIVE_PTES (d8)
No System PTEs left. Usually caused by a driver not cleaning up
properly. If non-null, Parameter 1 shows the name of the driver
who is consuming the most PTEs. The calling stack also shows the name of
the driver which bugchecked. Both drivers need to be fixed and/or the number
of PTEs increased.
When possible, the guilty driver's name (Unicode string) is printed on
the bugcheck screen and saved in KiBugCheckDriver.
Arguments:
Arg1: 864a3130, If non-null, the guilty driver's name (Unicode string).
Arg2: 0000a0e0, If parameter 1 non-null, the number of PTEs used by the
guilty driver.
Arg3: 00000d3e, Total free system PTEs
Arg4: 0000cd16, Total system PTEs
Debugging Details:
------------------
CUSTOMER_CRASH_COUNT: 1
DEFAULT_BUCKET_ID: DRIVER_FAULT
BUGCHECK_STR: 0xD8
PROCESS_NAME: EXEName
LAST_CONTROL_TRANSFER: from 8051e00f to 804f9deb
STACK_TEXT:
eb2ce628 8051e00f 000000d8 864a3130 0000a0e0 nt!KeBugCheckEx+0x1b
eb2ce648 80508498 0000017a 0000a0e0 eb2cec10 nt!MiIssueNoPtesBugcheck+0x49
eb2ce690 8050876c 85f499c8 00000000 00000001
nt!MmMapLockedPagesSpecifyCache+0x11a
eb2ce6b0 f6f5527f 85f499c8 00000000 00000000 nt!MmMapLockedPages+0x18
WARNING: Stack unwind information not available. Following frames may be
wrong.
eb2ce6c8 f6f55052 8525fb18 199ffce8 18c00000 windrvr6+0x2527f
eb2ce6f0 f6f557e2 8517c0d8 85efc0a0 00000001 windrvr6+0x25052
eb2ce748 f6f56c3e 8517c0d8 85efc0a0 85f07f90 windrvr6+0x257e2
eb2ce794 f6f55e11 00000003 00000088 00000001 windrvr6+0x26c3e
eb2ce7d4 f6f3c9e8 85f07f90 85f07f90 00000000 windrvr6+0x25e11
eb2cebdc f6f3ce37 00000001 85f07f90 85f07f90 windrvr6+0xc9e8
eb2cec40 804ef095 8616ad58 9538260f 806e4410 windrvr6+0xce37
eb2cec50 8057e70a 853e3edc 85f07f90 853e3e48 nt!IopfCallDriver+0x31
eb2cec64 8057f56d 8616ad58 853e3e48 85f07f90 nt!IopSynchronousServiceTail+0x60
eb2ced00 805780c2 00000634 00000000 00000000 nt!IopXxxControlFile+0x5c5
eb2ced34 8054086c 00000634 00000000 00000000 nt!NtDeviceIoControlFile+0x2a
eb2ced34 7c90eb94 00000634 00000000 00000000 nt!KiFastCallEntry+0xfc
199ffc80 00000000 00000000 00000000 00000000 0x7c90eb94
STACK_COMMAND: kb
FOLLOWUP_IP:
windrvr6+2527f
f6f5527f ?? ???
SYMBOL_STACK_INDEX: 4
SYMBOL_NAME: windrvr6+2527f
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: windrvr6
IMAGE_NAME: windrvr6.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 4357a6a8
FAILURE_BUCKET_ID: 0xD8_windrvr6+2527f
BUCKET_ID: 0xD8_windrvr6+2527f
Followup: MachineOwner
---------
0: kd> lmvm windrvr6
start end module name
f6f30000 f6f5eb60 windrvr6 T (no symbols)
Loaded symbol image file: windrvr6.sys
Image path: windrvr6.sys
Image name: windrvr6.sys
Timestamp: Thu Oct 20 09:16:08 2005 (4357A6A8)
CheckSum: 00039536
ImageSize: 0002EB60
Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0
I don’t know if !vm is supposed to work well on a mini-dump, but here is the
output:
0: kd> !vm
*** Virtual Memory Usage ***
GetUlongFromAddress: unable to read from 80561108
Physical Memory: 0 ( 0 Kb)
GetUlongFromAddress: unable to read from 80560c40
************ NO PAGING FILE *********************
80560b60: Unable to get paged pool info
GetUlongPtrFromAddress: unable to read from 80550990
GetUlongPtrFromAddress: unable to read from 80560f2c
GetPointerFromAddress: unable to read from 80560c04
GetPointerFromAddress: unable to read from 80554c48
GetUlongFromAddress: unable to read from 80550918
GetUlongFromAddress: unable to read from 80550928
GetUlongFromAddress: unable to read from 805610fc
GetUlongFromAddress: unable to read from 805610bc
GetUlongPtrFromAddress: unable to read from 80553280
GetUlongPtrFromAddress: unable to read from 80554cc0
Available Pages: 0 ( 0 Kb)
ResAvail Pages: 0 ( 0 Kb)
********** Running out of physical memory **********
Locked IO Pages: 0 ( 0 Kb)
Free System PTEs: 3390 ( 13560 Kb)
******* 9 system PTE allocations have failed ******
Free NP PTEs: 32455 ( 129820 Kb)
Free Special NP: 0 ( 0 Kb)
Modified Pages: 0 ( 0 Kb)
Modified PF Pages: 0 ( 0 Kb)
80563c20: Unable to get pool descriptor
GetUlongFromAddress: unable to read from 805512b8
NonPagedPool Usage: 0 ( 0 Kb)
NonPagedPool Max: 0 ( 0 Kb)
GetUlongFromAddress: unable to read from 805512b4
PagedPool Usage: 0 ( 0 Kb)
PagedPool Maximum: 0 ( 0 Kb)
GetUlongFromAddress: unable to read from 80564c48
Shared Commit: 0 ( 0 Kb)
Special Pool: 0 ( 0 Kb)
Shared Process: 3358 ( 13432 Kb)
PagedPool Commit: 24616 ( 98464 Kb)
Driver Commit: 1981 ( 7924 Kb)
Committed pages: 300914 ( 1203656 Kb)
Commit limit: 0 ( 0 Kb)
********** Number of committed pages is near limit ********
Unable to read/NULL value _LIST_ENTRY @ 805627b8
ProcessCommitUsage could not be calculated
According to http://msdn2.microsoft.com/en-us/library/ms795903.aspx, by
taking the first parameter (which is a pointer) I should be able to find the
name for offending driver.
0: kd> dc 864a3130 L1
864a3130 864ae008 ..J.
0: kd> db 864ae008
864ae008 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
864ae018 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
My questions are:
Did I use correct set of commands in my attempt to locate driver name?
Given the output above, is it reasonable to believe that kernel mini-dumps
do not actually store enough memory pages for me to locate the name?
If so, does it mean that I have to change dump type collection to “Complete
Memory Dump” in order to locate this information?
Is there another trick to dump contents for (PUNICODE_STRING)
KiBugCheckDriver?
Thank you in advance for your feedback,
Olegas
Jeffrey Tan[MSFT]
What you got is called "Small Memory Dump" whose content is documented in
the windbg help: Debuggers->Crash Dump Files->Kernel-Mode Dump
Files->Varieties of Kernel-Mode Dump Files->Small Memory Dump.
As you can see, most of the memory pages are not contained in the memory
dump for size reason, so I do not believe "!vm" will work.
I think you already got the offending driver name in the "!analyze -v"
output; it is windrvr6, isn't it?
KiBugCheckDriver is not a function; it is a global variable which points to
UNICODE_STRING containing the driver name:
lkd> x /v nt!KiBugCheckDriver
prv global 808a6530 4 nt!KiBugCheckDriver = 0x00000000
"db" command is used to display the ASCII string. UNICODE_STRING is a data
structure which contains a pointer to the unicode string. Here is its
definition:
typedef struct _UNICODE_STRING {
USHORT Length;
USHORT MaximumLength;
PWSTR Buffer;
} UNICODE_STRING;
There is a "!ustr" extension command can be used to parse and display the
UNICODE_STRING. Please refer to the windbg help document for the details.
Hope this helps.
Best regards,
Jeffrey Tan
Microsoft Online Community Support
=========================================
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
msd...@microsoft.com.
This posting is provided "AS IS" with no warranties, and confers no rights.
Olegas
I really appreciate your feedback and tips. I do not deal with kernel side
of things very often. I made a wrong assumption based on the dump size (88 KB
> 64 KB) that I had a kernel memory dump, not small memory dump.
I want to clarify my earlier post - by “offending driver” I meant the one
that consumed the most PTEs. If I understand the following article correctly:
http://msdn2.microsoft.com/en-us/library/ms795903.aspx, I should be looking
for two drivers:
1. one that consumed the most PTEs
2. one that tripped and caused the bug check. (in my case it is windrvr6.sys)
I assume 1 and 2 can be the same driver. We will look into updating
windrvr6.sys, but I’m really interested in locating which driver consumed the
most PTEs.
0: kd> x /v nt!KiBugCheckDriver
pub global 8055c060 0 nt!KiBugCheckDriver = <no type information>
0: kd> db 8055c060
8055c060 5c 31 4a 86 00 00 00 00-00 00 00 00 00 00 00 00 \1J.............
8055c070 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
8055c080 d8 00 00 00 30 31 4a 86-e0 a0 00 00 3e 0d 00 00 ....01J.....>...
8055c090 16 cd 00 00 00 60 00 e1-00 00 00 00 00 00 00 00 .....`..........
8055c0a0 01 00 00 00 f4 2c f1 eb-18 53 00 00 01 00 04 00 .....,...S......
8055c0b0 00 00 00 00 b4 c0 55 80-b4 c0 55 80 00 00 00 00 ......U...U.....
8055c0c0 00 00 00 00 70 9e 59 86-40 90 59 86 00 00 00 00 ....p.Y.@.Y.....
8055c0d0 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0: kd> !ustr 8055c060
String(12636,34378) nt!KiBugCheckDriver+00000000
Given the output above, would you recommend using “Complete Memory Dump”
type to track down which driver does not clean up its memory use?
Thank you,
Olegas
Jeffrey Tan[MSFT]
Thanks for your feedback.
Oh, yes, it seems what you have is a "Kernel Memory Dump".
As the MSDN link pointed out, the nt!KiBugCheckDriver is a global pointer
of type "PUNICODE_STRING", so we should first find the content in this
pointer and then dump the unicode string:
http://msdn2.microsoft.com/en-us/library/ms795903.aspx
You may use "dd nt!KiBugCheckDriver l1" to dereference the pointer and then
dump its content:
lkd> dd nt!KiBugCheckDriver l1
808a6530 [dererenced address]
lkd> !ustr [dererenced address]
In the "db" output, the [dererenced address] should be "864a315c", so you
may use "!ustr 864a315c". I hope this will dump out the real culprit driver
name.
Thanks.
Best regards,
Jeffrey Tan
Microsoft Online Community Support
=========================================
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
msd...@microsoft.com.
This posting is provided "AS IS" with no warranties, and confers no rights.
Olegas
Thank you very much for the follow up. You have answered all of my questions!
0: kd> dd nt!KiBugCheckDriver l1
8055c060 864a315c
0: kd> db 864a315c
864a315c 18 00 18 00 7c 31 4a 86-00 40 10 09 06 00 00 00 ....|1J..@......
864a316c ff ff ff ff 4c b8 01 00-fe ff ff ff 00 00 00 00 ....L...........
864a317c 56 00 49 00 44 00 45 00-4f 00 50 00 52 00 54 00 V.I.D.E.O.P.R.T.
864a318c 2e 00 53 00 59 00 53 00-00 00 00 00 0e 00 01 00 ..S.Y.S.........
864a319c 4d 6d 49 6e 01 00 03 0a-49 6f 20 20 52 00 61 00 MmIn....Io R.a.
864a31ac 73 00 6c 00 32 00 74 00-70 00 00 00 03 00 08 0a s.l.2.t.p.......
864a31bc 4e 74 66 72 70 2a 4a 86-00 32 4a 86 68 bf 1c 85 Ntfrp*J..2J.h...
864a31cc 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0: kd> !ustr 864a315c
String(24,24) at 864a315c: VIDEOPRT.SYS
0: kd> lmvm VIDEOPRT
start end module name
f71cb000 f71de780 VIDEOPRT (deferred)
Mapped memory image file:
C:\symcache\VIDEOPRT.SYS\41107D0813780\VIDEOPRT.SYS
Image path: VIDEOPRT.SYS
Image name: VIDEOPRT.SYS
Timestamp: Wed Aug 04 01:07:04 2004 (41107D08)
CheckSum: 0001B84C
ImageSize: 00013780
File version: 5.1.2600.2180
Product version: 5.1.2600.2180
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 3.4 Driver
File date: 00000000.00000000
Translations: 0000.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operating System
InternalName: videoprt.sys
OriginalFilename: videoprt.sys
ProductVersion: 5.1.2600.2180
FileVersion: 5.1.2600.2180 (xpsp_sp2_rtm.040803-2158)
FileDescription: Video Port Driver
LegalCopyright: © Microsoft Corporation. All rights reserved.
I’m not familiar with intricate details of interface between video port
driver and video miniport driver, but the data that you helped me pull from
the dump is valuable. We’ll proceed with upgrading video driver to see if it
resolves the issue.
Once again, thank you very much for thorough explanation!
Olegas
Jeffrey Tan[MSFT]
Glad to see my reply can help you. If you need further help, please feel
free to post, thanks.
Best regards,
Jeffrey Tan
Microsoft Online Community Support
=========================================
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
msd...@microsoft.com.
This posting is provided "AS IS" with no warranties, and confers no rights.