Unmanaged code memory leak?
rick
virtual bytes continuously increase but # Bytes in all Heaps and # of
current logical Threads remain relatively flat.I see quite a few
objects in gen 2 but I don't know what to do after that. The service
is a C# app and multithreaded (3.5 framework). What should I look at
next?
:000> !finalizequeue
SyncBlocks to be cleaned up: 0
MTA Interfaces to be released: 0
STA Interfaces to be released: 0
----------------------------------
generation 0 has 387 finalizable objects (095cb8d8->095cbee4)
generation 1 has 0 finalizable objects (095cb8d8->095cb8d8)
generation 2 has 352 finalizable objects (095cb358->095cb8d8)
Ready for finalization 0 objects (095cbee4->095cbee4)
Statistics:
MT Count TotalSize Class Name
7a78a580 1 20
Microsoft.Win32.SafeHandles.SafeEventLogWriteHandle
7a7739f4 1 20 System.Net.SafeLocalFree
7a75fb30 1 20
Microsoft.Win32.SafeHandles.SafeProcessHandle
79116758 1 20
Microsoft.Win32.SafeHandles.SafeTokenHandle
6540122c 1 20 Bid+AutoInit
08274e88 1 20
Microsoft.Win32.SafeHandles.SafeFileMapViewHandle
08274da0 1 20
Microsoft.Win32.SafeHandles.SafeFileMappingHandle
08273808 1 20
Microsoft.Win32.SafeHandles.SafeProcessHandle
6540ee5c 1 24 System.Data.OleDb.ChapterHandle
6540d508 1 24 System.Data.OleDb.OleDbServicesWrapper
7a7557f0 1 32 System.ComponentModel.Container
7a7932c8 1 36 System.Net.AutoWebProxyScriptWrapper
65404cdc 1 36 System.Data.SqlClient.SNILoadHandle
7a76d204 2 40 System.Net.SafeCloseSocket
+InnerSafeCloseSocket
7a762308 2 40
Microsoft.Win32.SafeHandles.SafeFileMapViewHandle
7a762248 2 40
Microsoft.Win32.SafeHandles.SafeFileMappingHandle
7910b630 2 40
System.Security.Cryptography.SafeProvHandle
791037c0 2 40
Microsoft.Win32.SafeHandles.SafeFileMappingHandle
79103764 2 40
Microsoft.Win32.SafeHandles.SafeViewOfFileHandle
67b33d94 2 40 System.Transactions.SafeIUnknown
7a774674 2 48 System.Net.SafeRegistryHandle
7a774440 2 56 System.Net.SafeCloseSocketAndEvent
7a7a1a3c 1 88 System.Diagnostics.EventLog
00693d6c 1 100 KPIService.KPICollector
7910a5c4 2 120
System.Runtime.Remoting.Contexts.Context
6540f27c 6 144 System.Data.OleDb.RowHandleBuffer
6540f0ac 6 144 System.Data.OleDb.DualCoTaskMem
6540e9f8 6 168 System.Data.OleDb.PropertyIDSet
653fdd8c 3 168 System.Data.SqlClient.SqlConnection
79101444 10 200
Microsoft.Win32.SafeHandles.SafeFileHandle
79108ba4 5 220 System.Threading.ReaderWriterLock
6540d8f4 12 240 System.Data.OleDb.DataSourceWrapper
6540b490 6 264 System.Data.OleDb.OleDbConnection
79106384 14 280
Microsoft.Win32.SafeHandles.SafeRegistryHandle
6540d9c8 12 288 System.Data.OleDb.SessionWrapper
65407d48 1 296 System.Data.DataTable
79112884 13 312 System.Threading.TimerBase
6540e53c 6 312 System.Data.OleDb.RowBinding
790fe704 6 336 System.Threading.Thread
6540dfb0 19 456 System.Data.OleDb.DBPropSet
654050b0 23 460 System.Data.SqlClient.SNIPacket
65404f94 23 552 System.Data.SqlClient.SNIHandle
654044a8 12 624
System.Data.ProviderBase.DbConnectionPool+PoolWaitHandles
653fe188 5 660 System.Data.SqlClient.SqlCommand
79116b38 9 720 System.IO.FileStream
79112728 47 940
Microsoft.Win32.SafeHandles.SafeWaitHandle
6540d314 40 960 System.Data.OleDb.StringMemHandle
6540b7fc 13 1092 System.Data.OleDb.OleDbCommand
08271acc 10 1200 System.Diagnostics.PerformanceCounter
79104c38 127 2032 System.WeakReference
7a7614c0 20 2400 System.Diagnostics.PerformanceCounter
654088b4 26 3848 System.Data.DataColumn
7a7673f4 224 35840 System.Diagnostics.Process
Total 739 objects
0:000> !gchandles
GC Handle Statistics:
Strong Handles: 125
Pinned Handles: 63
Async Pinned Handles: 0
Ref Count Handles: 0
Weak Long Handles: 59
Weak Short Handles: 39
Other Handles: 0
Statistics:
MT Count TotalSize Class Name
79100f58 1 12
System.Security.Permissions.SecurityPermission
654011e4 1 12 Bid+BindingCookie
790ff734 1 20 System.RuntimeType
79194c44 1 24 System.Security.PermissionListSet
790feba4 1 28 System.SharedStatics
67a317d4 1 32 System.ServiceProcess.NativeMethods
+ServiceControlCallbackEx
67a316b4 1 32 System.ServiceProcess.NativeMethods
+ServiceMainCallback
654010e8 1 32 Bid+CtrlCB
7a762dac 1 36 System.Diagnostics.TraceSource
790fd0f0 3 36 System.Object
7a755b4c 1 40 System.Diagnostics.BooleanSwitch
67b33d94 2 40 System.Transactions.SafeIUnknown
791132b4 1 48 System.Security.NamedPermissionSet
79116fac 2 64 System.Reflection.Emit.AssemblyBuilder
790fe17c 1 72 System.ExecutionEngineException
790fe0e0 1 72 System.StackOverflowException
790fe044 1 72 System.OutOfMemoryException
6540b7fc 1 84 System.Data.OleDb.OleDbCommand
7910a2fc 2 88 System.Runtime.Remoting.Identity
6540b490 2 88 System.Data.OleDb.OleDbConnection
7910a5c4 2 120
System.Runtime.Remoting.Contexts.Context
653fe76c 1 136 System.Data.SqlClient.SqlDataReader
7a75afa0 7 140 System.Net.TimerThread+TimerQueue
7a76b084 4 144 System.Net.Logging+NclTraceSource
790fe284 2 144 System.Threading.ThreadAbortException
07d78eec 4 144 System.Net.Logging+NclTraceSource
07d79134 4 160 System.Diagnostics.SourceSwitch
7a763aac 5 200 System.Diagnostics.SourceSwitch
790fed00 2 200 System.AppDomain
79121168 4 224 System.Reflection.Emit.ModuleBuilder
79112958 13 260 System.Threading._TimerCallback
7910a4bc 4 320 System.Runtime.Remoting.ServerIdentity
790fcc48 19 456 System.Reflection.Assembly
790fe704 12 672 System.Threading.Thread
791071c0 16 768 System.Reflection.Module
79100a18 46 1656 System.Security.PermissionSet
791083f4 54 4104 System.RuntimeType+RuntimeTypeCache
7912d8f8 61 68888 System.Object[]
Total 286 objects
0:000> !threads
ThreadCount: 10
UnstartedThread: 0
BackgroundThread: 7
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
PreEmptive GC Alloc
Lock
ID OSID ThreadOBJ State GC Context Domain
Count APT Exception
0 1 16bc 00161200 a020 Enabled 00000000:00000000
00157b98 0 MTA
2 2 1274 0016d9f8 b220 Enabled 00000000:00000000
00157b98 0 MTA (Finalizer)
3 4 179c 00186258 180b220 Enabled 00000000:00000000
00157b98 0 MTA (Threadpool Worker)
4 5 11e0 001a3d38 80a220 Enabled 00000000:00000000
00157b98 0 MTA (Threadpool Completion Port)
8 6 1200 001d39d0 200b020 Enabled 06d025fc:06d03fe8
00157b98 0 MTA
10 8 c84 043713d0 180b220 Enabled 00000000:00000000
00157b98 0 MTA (Threadpool Worker)
11 1c 15f4 043bed78 1220 Enabled 00000000:00000000
00157b98 0 Ukn
14 b fa0 08fb9e30 180b220 Enabled 00000000:00000000
00157b98 0 MTA (Threadpool Worker)
15 16 500 0955b8a8 200b220 Enabled 00000000:00000000
00157b98 1 MTA
22 13 eec 08e7b660 b020 Enabled 06d18d5c:06d19fe8
00157b98 0 MTA
Marc Sherman
http://blogs.msdn.com/tess/default.aspx
It's a great resource for debugging .NET issues using windbg and sos.
"rick" <rkha...@yahoo.com> wrote in message
news:8a12fd55-30ec-46bf...@l35g2000pra.googlegroups.com...
rick
Kjell Gunnar
To verify your suspicion do a !heap –s regularly to see if any
unmanaged heaps grows.
Also do !address –summary to see the type of memory that increases.
Good Luck
Kjell Gunnar
rick
Where can I find help on !heap? Search on Tess blog returns nothing
and I found http://msdn.microsoft.com/en-us/library/cc266933.aspx.
0:000> !address -summary
-------------------- Usage SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Pct(Busy) Usage
512a000 ( 83112) : 03.96% 29.31% : RegionUsageIsVAD
6eb09000 ( 1813540) : 86.48% 00.00% : RegionUsageFree
7012000 ( 114760) : 05.47% 40.47% : RegionUsageImage
1700000 ( 23552) : 01.12% 08.31% : RegionUsageStack
17000 ( 92) : 00.00% 00.03% : RegionUsageTeb
3c90000 ( 62016) : 02.96% 21.87% : RegionUsageHeap
0 ( 0) : 00.00% 00.00% : RegionUsagePageHeap
1000 ( 4) : 00.00% 00.00% : RegionUsagePeb
1000 ( 4) : 00.00% 00.00% :
RegionUsageProcessParametrs
2000 ( 8) : 00.00% 00.00% :
RegionUsageEnvironmentBlock
Tot: 7fff0000 (2097088 KB) Busy: 114e7000 (283548 KB)
-------------------- Type SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
6eb09000 ( 1813540) : 86.48% : <free>
7ad9000 ( 125796) : 06.00% : MEM_IMAGE
668000 ( 6560) : 00.31% : MEM_MAPPED
93a6000 ( 151192) : 07.21% : MEM_PRIVATE
-------------------- State SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
c77c000 ( 204272) : 09.74% : MEM_COMMIT
6eb09000 ( 1813540) : 86.48% : MEM_FREE
4d6b000 ( 79276) : 03.78% : MEM_RESERVE
Largest free region: Base 1100c000 - Size 39704000 (941072 KB)
0:000> !heap -s
Heap Flags Reserv Commit Virt Free List UCR Virt
Lock Fast
(k) (k) (k) (k) length blocks
cont. heap
-----------------------------------------------------------------------------
00150000 00000002 8192 6096 6756 2502 842 62 0
295 L
External fragmentation 41 % (842 free blocks)
00250000 00008000 64 12 12 10 1 1 0
0
00330000 00001002 64 40 40 1 1 1 0
0 L
00350000 00041002 256 16 16 3 1 1 0
0 L
003b0000 00001002 3136 2156 2336 629 116 19 0
0 L
External fragmentation 29 % (116 free blocks)
003d0000 00001002 1088 848 848 56 2 1 0
0 L
00990000 00000002 1024 20 20 3 1 1 0
0 L
00e70000 00001002 256 48 48 2 1 1 0
0 L
00fc0000 00001002 64 16 16 0 0 1 0
0 L
03440000 00001003 1280 684 684 3 1 1 0
bad
03480000 00001003 256 112 112 3 1 1 0
bad
034c0000 00001003 256 24 24 2 1 1 0
bad
03500000 00001003 256 4 4 2 1 1 0
bad
03540000 00001003 256 4 4 2 1 1 0
bad
03580000 00001003 256 4 4 2 1 1 0
bad
035c0000 00001003 256 4 4 2 1 1 0
bad
03600000 00001003 1280 832 864 611 35 3 0
bad
03640000 00001003 1280 40 40 11 1 2 0
bad
03680000 00001003 256 36 84 34 2 2 0
bad
036c0000 00001003 256 12 12 10 1 1 0
bad
03700000 00001003 256 4 4 2 1 1 0
bad
03740000 00001003 256 4 4 2 1 1 0
bad
03780000 00001003 256 4 4 2 1 1 0
bad
03da0000 00001002 64 48 48 40 2 1 0
0 L
03db0000 00011002 256 24 24 16 1 1 0
0 L
03df0000 00001002 64 16 16 6 1 1 0
0 L
03e00000 00001002 256 64 64 12 1 1 0
0 L
03e40000 00001002 64 12 12 4 1 1 0
0 L
03fb0000 00001003 7424 5380 5380 1 1 1 0
bad
04b50000 00001003 7424 3768 3768 1 1 1 0
bad
04ef0000 00001000 1024 1024 1024 495 1 0 0
0
045c0000 00001002 64 16 16 1 1 1 0
0 L
049a0000 00001003 7424 3836 3836 2 1 1 0
bad
07a50000 00001003 7424 5056 5056 3 1 1 0
bad
05aa0000 00001002 2048 632 960 328 31 28 0
0 L
06100000 00041002 256 52 52 4 2 1 0
0 L
09030000 00001003 7424 3540 3540 1 1 1 0
bad
-----------------------------------------------------------------------------
0:000> !help heap
-------------------------------------------------------------------------------
Documentation for heap not found
pat styles [microsoft]
Did you try searching for help on !heap in the debugger documentation?
There is a pretty extensive discussion of it in there.
.pat styles [microsoft]
"rick" <rkha...@yahoo.com> wrote in message
news:8d76492e-303e-44b8...@b25g2000prb.googlegroups.com...
Kjell Gunnar
For me your output don’t show a very big app is it just started ?.
But the interesting is how it evolves over time.
So let the app run for a while until you can see a very significant
increase in private bytes.
Then do the same commands again and look for differences.
You should then hopefully be able to see what type of memory that
contributes to the increase.
The RegionUsageHeap means unmanaged heaps only and the Commit column
out of !heap contributes to private bytes.
But again this will only help you to verify your suspicion about
unmanaged leak.
Since your app is so small (86.48% : <free>) I think you should
let it run for a couple of days and see if the private / virtual bytes
usage flatter out.
Here is a post on unmanaged heap analyzes and !heap usage:
http://groups.google.com/group/microsoft.public.windbg/browse_frm/thread/ba685a5d4ef3c073/6d3634948d8c3806?hl=en&q=Ivan+Brugiolo+heap+leak
Regards
Kjell Gunnar
On Jul 30, 4:14 pm, rick <rkhaer...@yahoo.com> wrote:
> Thanks Kjell!
>
> Where can I find help on !heap? Search on Tess blog returns nothing
> and I foundhttp://msdn.microsoft.com/en-us/library/cc266933.aspx.
> -----------------------------------------------------------------------------
> -----------------------------------------------------------------------------
> 0:000> !help heap
> -------------------------------------------------------------------------------
rick
Thanks for the reply! I am very new to Windbg so forgive me for my
lame questions and replies.
The reason this service concerns me is that I believe it consumes
memory to the point that it pulls down the Windows 2003 server. After
letting it run for days, eventually other applications stop processing
normally. Sometimes we can't even Remote Desktop in and reboot the
server gracefully.
Here is the !address -summary of the app after running for 1,330
minutes and after Process.PeakWorkingSet64 increased 517,320 Kb within
a minute to a total of 713,032 Kb. I know the code and I can't think
of anything that would/should increase the memory usage like that.
The stats below state the unmanaged memory (RegionUsageFree) is less
than 2 Mb (1655532)? The stats were captured with adplus.vbs -quiet -
hang -p 4672. I have noticed after running adplus Peak Mem Usage and
Mem Usage in the TaskManager doubles.
0:000> !address -summary
-------------------- Usage SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Pct(Busy) Usage
c9d5000 ( 206676) : 09.86% 46.81% : RegionUsageIsVAD
650bb000 ( 1655532) : 78.94% 00.00% : RegionUsageFree
734c000 ( 118064) : 05.63% 26.74% : RegionUsageImage
2000000 ( 32768) : 01.56% 07.42% : RegionUsageStack
20000 ( 128) : 00.01% 00.03% : RegionUsageTeb
51f0000 ( 83904) : 04.00% 19.00% : RegionUsageHeap
0 ( 0) : 00.00% 00.00% : RegionUsagePageHeap
1000 ( 4) : 00.00% 00.00% : RegionUsagePeb
1000 ( 4) : 00.00% 00.00% :
RegionUsageProcessParametrs
2000 ( 8) : 00.00% 00.00% :
RegionUsageEnvironmentBlock
Tot: 7fff0000 (2097088 KB) Busy: 1af35000 (441556 KB)
-------------------- Type SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
650bb000 ( 1655532) : 78.94% : <free>
7e13000 ( 129100) : 06.16% : MEM_IMAGE
668000 ( 6560) : 00.31% : MEM_MAPPED
12aba000 ( 305896) : 14.59% : MEM_PRIVATE
-------------------- State SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
104c0000 ( 267008) : 12.73% : MEM_COMMIT
650bb000 ( 1655532) : 78.94% : MEM_FREE
aa75000 ( 174548) : 08.32% : MEM_RESERVE
Largest free region: Base 1b83a000 - Size 220e6000 (557976 KB)
Kjell Gunnar
I don’t think its a unmanaged leak, the RegionUsageHeap shows only a
small growth,
but the RegionUsageIsVAD has a significant increase.
0:008> ? ((0xc9d5000-0x512a000) /0n1024) / 0n1024
Evaluate expression: 120 = 00000078
There can be a number of reasons for this, but since this is a managed
app you could check if the managed heap grows.
.loadby sos mscorwks
!eeheap –gc
will show the total size of the managed heap, check this and see if
it grows over time.
Regards
Kjell Gunnar
rick
I captured the !address -summary and !eeheap -gc. I noticed a couple
of objects in !dumpheap that had a hight count. All objects were from
a couple of dlls. I changed the classes to static and no longer see
them in !dumpheap but Private Bytes continues to grow.
8/4/2009 12:41 PM
0:000> !address -summary
-------------------- Usage SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Pct(Busy) Usage
505e000 ( 82296) : 03.92% 32.29% : RegionUsageIsVAD
7070e000 ( 1842232) : 87.85% 00.00% : RegionUsageFree
7012000 ( 114760) : 05.47% 45.03% : RegionUsageImage
1e00000 ( 30720) : 01.46% 12.05% : RegionUsageStack
1e000 ( 120) : 00.01% 00.05% : RegionUsageTeb
1a50000 ( 26944) : 01.28% 10.57% : RegionUsageHeap
0 ( 0) : 00.00% 00.00% : RegionUsagePageHeap
1000 ( 4) : 00.00% 00.00% : RegionUsagePeb
1000 ( 4) : 00.00% 00.00% :
RegionUsageProcessParametrs
2000 ( 8) : 00.00% 00.00% :
RegionUsageEnvironmentBlock
Tot: 7fff0000 (2097088 KB) Busy: 0f8e2000 (254856 KB)
-------------------- Type SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
7070e000 ( 1842232) : 87.85% : <free>
7ad9000 ( 125796) : 06.00% : MEM_IMAGE
678000 ( 6624) : 00.32% : MEM_MAPPED
7791000 ( 122436) : 05.84% : MEM_PRIVATE
-------------------- State SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
bc87000 ( 193052) : 09.21% : MEM_COMMIT
7070e000 ( 1842232) : 87.85% : MEM_FREE
3c5b000 ( 61804) : 02.95% : MEM_RESERVE
Largest free region: Base 1100c000 - Size 39704000 (941072 KB)
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x07ff96cc
generation 1 starts at 0x07fd4080
generation 2 starts at 0x00fe1000
ephemeral segment allocation context: none
segment begin allocated size
0017d1f8 7a733370 7a754b98 0x00021828(137256)
0016f850 790d8620 790f7d8c 0x0001f76c(128876)
00fe0000 00fe1000 01966af8 0x00985af8(9984760)
07e80000 07e81000 08077320 0x001f6320(2056992)
Large object heap starts at 0x01fe1000
segment begin allocated size
01fe0000 01fe1000 022353a0 0x002543a0(2442144)
Total Size 0xe1114c(14750028)
------------------------------
GC Heap Size 0xe1114c(14750028)
8/4/2009 2:25 PM
0:000> !address -summary
-------------------- Usage SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Pct(Busy) Usage
bb8d000 ( 192052) : 09.16% 50.74% : RegionUsageIsVAD
68e54000 ( 1718608) : 81.95% 00.00% : RegionUsageFree
7344000 ( 118032) : 05.63% 31.19% : RegionUsageImage
1700000 ( 23552) : 01.12% 06.22% : RegionUsageStack
17000 ( 92) : 00.00% 00.02% : RegionUsageTeb
2bb0000 ( 44736) : 02.13% 11.82% : RegionUsageHeap
0 ( 0) : 00.00% 00.00% : RegionUsagePageHeap
1000 ( 4) : 00.00% 00.00% : RegionUsagePeb
1000 ( 4) : 00.00% 00.00% :
RegionUsageProcessParametrs
2000 ( 8) : 00.00% 00.00% :
RegionUsageEnvironmentBlock
Tot: 7fff0000 (2097088 KB) Busy: 1719c000 (378480 KB)
-------------------- Type SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
68e54000 ( 1718608) : 81.95% : <free>
7e0b000 ( 129068) : 06.15% : MEM_IMAGE
678000 ( 6624) : 00.32% : MEM_MAPPED
ed19000 ( 242788) : 11.58% : MEM_PRIVATE
-------------------- State SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
bfc7000 ( 196380) : 09.36% : MEM_COMMIT
68e54000 ( 1718608) : 81.95% : MEM_FREE
b1d5000 ( 182100) : 08.68% : MEM_RESERVE
Largest free region: Base 1b83a000 - Size 2eed6000 (768856 KB)
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x114516a4
generation 1 starts at 0x113b4a54
generation 2 starts at 0x00fe1000
ephemeral segment allocation context: none
segment begin allocated size
0017d1f8 7a733370 7a754b98 0x00021828(137256)
0016f850 790d8620 790f7d8c 0x0001f76c(128876)
00fe0000 00fe1000 0171d4ec 0x0073c4ec(7587052)
07e80000 07e81000 0801ef30 0x0019df30(1695536)
11010000 11011000 114c56b0 0x004b46b0(4933296)
Large object heap starts at 0x01fe1000
segment begin allocated size
01fe0000 01fe1000 022353a0 0x002543a0(2442144)
Total Size 0x1023e00(16924160)
------------------------------
GC Heap Size 0x1023e00(16924160)
*********************************************************************
8/4/2009 4:55 PM
0:000> !address -summary
-------------------- Usage SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Pct(Busy) Usage
bbac000 ( 192176) : 09.16% 49.95% : RegionUsageIsVAD
68831000 ( 1712324) : 81.65% 00.00% : RegionUsageFree
7344000 ( 118032) : 05.63% 30.68% : RegionUsageImage
1b00000 ( 27648) : 01.32% 07.19% : RegionUsageStack
1b000 ( 108) : 00.01% 00.03% : RegionUsageTeb
2db0000 ( 46784) : 02.23% 12.16% : RegionUsageHeap
0 ( 0) : 00.00% 00.00% : RegionUsagePageHeap
1000 ( 4) : 00.00% 00.00% : RegionUsagePeb
1000 ( 4) : 00.00% 00.00% :
RegionUsageProcessParametrs
2000 ( 8) : 00.00% 00.00% :
RegionUsageEnvironmentBlock
Tot: 7fff0000 (2097088 KB) Busy: 177bf000 (384764 KB)
-------------------- Type SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
68831000 ( 1712324) : 81.65% : <free>
7e0b000 ( 129068) : 06.15% : MEM_IMAGE
678000 ( 6624) : 00.32% : MEM_MAPPED
f33c000 ( 249072) : 11.88% : MEM_PRIVATE
-------------------- State SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
cb40000 ( 208128) : 09.92% : MEM_COMMIT
68831000 ( 1712324) : 81.65% : MEM_FREE
ac7f000 ( 176636) : 08.42% : MEM_RESERVE
Largest free region: Base 1b83a000 - Size 2eed6000 (768856 KB)
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x1142cb2c
generation 1 starts at 0x113feb28
generation 2 starts at 0x00fe1000
ephemeral segment allocation context: none
segment begin allocated size
0017d1f8 7a733370 7a754b98 0x00021828(137256)
0016f850 790d8620 790f7d8c 0x0001f76c(128876)
00fe0000 00fe1000 0191a784 0x00939784(9672580)
07e80000 07e81000 0801ef30 0x0019df30(1695536)
11010000 11011000 114a6b38 0x00495b38(4807480)
Large object heap starts at 0x01fe1000
segment begin allocated size
01fe0000 01fe1000 02323620 0x00342620(3417632)
Total Size 0x12f07a0(19859360)
------------------------------
GC Heap Size 0x12f07a0(19859360)
8/5/2009 5:50 AM
0:000> !address -summary
-------------------- Usage SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Pct(Busy) Usage
c0b6000 ( 197336) : 09.41% 48.30% : RegionUsageIsVAD
670e9000 ( 1688484) : 80.52% 00.00% : RegionUsageFree
7344000 ( 118032) : 05.63% 28.89% : RegionUsageImage
1900000 ( 25600) : 01.22% 06.27% : RegionUsageStack
19000 ( 100) : 00.00% 00.02% : RegionUsageTeb
41f0000 ( 67520) : 03.22% 16.52% : RegionUsageHeap
0 ( 0) : 00.00% 00.00% : RegionUsagePageHeap
1000 ( 4) : 00.00% 00.00% : RegionUsagePeb
1000 ( 4) : 00.00% 00.00% :
RegionUsageProcessParametrs
2000 ( 8) : 00.00% 00.00% :
RegionUsageEnvironmentBlock
Tot: 7fff0000 (2097088 KB) Busy: 18f07000 (408604 KB)
-------------------- Type SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
670e9000 ( 1688484) : 80.52% : <free>
7e0b000 ( 129068) : 06.15% : MEM_IMAGE
678000 ( 6624) : 00.32% : MEM_MAPPED
10a84000 ( 272912) : 13.01% : MEM_PRIVATE
-------------------- State SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
e835000 ( 237780) : 11.34% : MEM_COMMIT
670e9000 ( 1688484) : 80.52% : MEM_FREE
a6d2000 ( 170824) : 08.15% : MEM_RESERVE
Largest free region: Base 1b83a000 - Size 2eed6000 (768856 KB)
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x14d74234
generation 1 starts at 0x14d5c098
generation 2 starts at 0x00fe1000
ephemeral segment allocation context: none
segment begin allocated size
0017d1f8 7a733370 7a754b98 0x00021828(137256)
0016f850 790d8620 790f7d8c 0x0001f76c(128876)
00fe0000 00fe1000 01fd1898 0x00ff0898(16713880)
07e80000 07e81000 083d37c0 0x005527c0(5580736)
149b0000 149b1000 14e2545c 0x0047445c(4670556)
Large object heap starts at 0x01fe1000
segment begin allocated size
01fe0000 01fe1000 02323620 0x00342620(3417632)
Total Size 0x1d3aa68(30648936)
------------------------------
GC Heap Size 0x1d3aa68(30648936)
Kjell Gunnar
If you compare your first and last printouts there is aprox 29 hours
between.
There is a notable growth in both unmanaged (RegionUsageHeap) and
managed (!eeheap) heaps.
This is a pure .NET app, isn’t it ? Therefore it is likely that the
root cause a managed issue.
I’m not too familiar with managed memory leak hunting, but here is
some links:
http://msdn.microsoft.com/en-us/library/ms954591.aspx
http://msdn.microsoft.com/nb-no/magazine/cc163528(en-us).aspx
http://blogs.msdn.com/maoni/archive/2006/11/08/correlating-the-output-of-eeheap-gc-and-address.aspx
But let the app run for a couple of days to see if the private bytes
increase stops.
If you include the WinDbg .time command in your printouts, we can see
how long the process has been alive.
Regards and good luck
Kjell Gunnar
rick
Thanks for the links. I will check it out. I will let it run until
tomorrow morning. Here is where it is at currently according to
Process Explorer.
Private Bytes = 130 Mb
Peak Private Bytes = 221 Mb.
0:000> .TIME
Debug session time: Thu Aug 6 10:33:35.595 2009 (GMT-7)
System Uptime: 4 days 18:04:46.375
Process Uptime: 1 days 1:39:52.440
Kernel time: 0 days 0:55:36.750
User time: 0 days 10:24:06.750
0:000> !address -summary
-------------------- Usage SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Pct(Busy) Usage
b2d8000 ( 183136) : 08.73% 42.97% : RegionUsageIsVAD
65fb8000 ( 1670880) : 79.68% 00.00% : RegionUsageFree
734c000 ( 118064) : 05.63% 27.70% : RegionUsageImage
2000000 ( 32768) : 01.56% 07.69% : RegionUsageStack
20000 ( 128) : 00.01% 00.03% : RegionUsageTeb
59f0000 ( 92096) : 04.39% 21.61% : RegionUsageHeap
0 ( 0) : 00.00% 00.00% : RegionUsagePageHeap
1000 ( 4) : 00.00% 00.00% : RegionUsagePeb
1000 ( 4) : 00.00% 00.00% :
RegionUsageProcessParametrs
2000 ( 8) : 00.00% 00.00% :
RegionUsageEnvironmentBlock
Tot: 7fff0000 (2097088 KB) Busy: 1a038000 (426208 KB)
-------------------- Type SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
65fb8000 ( 1670880) : 79.68% : <free>
7e13000 ( 129100) : 06.16% : MEM_IMAGE
678000 ( 6624) : 00.32% : MEM_MAPPED
11bad000 ( 290484) : 13.85% : MEM_PRIVATE
-------------------- State SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
e955000 ( 238932) : 11.39% : MEM_COMMIT
65fb8000 ( 1670880) : 79.68% : MEM_FREE
b6e3000 ( 187276) : 08.93% : MEM_RESERVE
Largest free region: Base 1b83a000 - Size 2eed6000 (768856 KB)
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x162122e0
generation 1 starts at 0x161e7cfc
generation 2 starts at 0x00fe1000
ephemeral segment allocation context: none
segment begin allocated size
0017d1f8 7a733370 7a754b98 0x00021828(137256)
0016f800 790d8620 790f7d8c 0x0001f76c(128876)
00fe0000 00fe1000 01c24e58 0x00c43e58(12861016)
16010000 16011000 16260e4c 0x0024fe4c(2424396)
Large object heap starts at 0x01fe1000
segment begin allocated size
01fe0000 01fe1000 02277d78 0x00296d78(2715000)
Total Size 0x116b9b0(18266544)
------------------------------
GC Heap Size 0x116b9b0(18266544)
rick
Yes, the app is all C# .NET 3.5 framework (four projects with two DLLs
and one service). It looks like I might have the fixed the managed
side (GS Heap Size?). The RegionUsageHeap continues to rise. Your
http://msdn.microsoft.com/en-us/library/ms954591.aspx link seems to
lean that way. I stumbled across
http://blogs.msdn.com/johan/archive/2008/02/21/walkthrough-troubleshooting-a-native-memory-leak.aspx
and downloaded the DebugDiag app.
I get these three warnings.
1) mscorwks.dll is responsible for 16.30 MBytes worth of outstanding
allocations. The following are the top 2 memory consuming functions:
mscorwks!EEVirtualAlloc+104: 16.00 MBytes worth of outstanding
allocations.
mscorwks!EEHeapAlloc+12d: 307.08 KBytes worth of outstanding
allocations.
2) WARNING - DebugDiag was not able to locate debug symbols for
oran9.dll, so the reported function name(s) may not be accurate.
oran9.dll is responsible for 551.85 KBytes worth of outstanding
allocations. The following are the top 2 memory consuming functions:
oran9!nsgblini+508: 215.02 KBytes worth of outstanding allocations.
oran9!nsbfree+1fa: 111.18 KBytes worth of outstanding allocations.
3) Detected symptoms of high fragmentation in the following heaps in
KPIService.exe__PID__5168__Date__08_07_2009__Time_04_20_07AM__100__Leak
Dump - Leak Rule Expired.dmp
0x003b0000 (msvcrt!_crtheap - 82.50% Fragmented)
0:000> .TIME
Debug session time: Thu Aug 6 10:33:35.595 2009 (GMT-7)
System Uptime: 4 days 18:04:46.375
Process Uptime: 1 days 1:39:52.440
Kernel time: 0 days 0:55:36.750
User time: 0 days 10:24:06.750
0:000> !address -summary
-------------------- Usage SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Pct(Busy) Usage
b2d8000 ( 183136) : 08.73% 42.97% : RegionUsageIsVAD
65fb8000 ( 1670880) : 79.68% 00.00% : RegionUsageFree
734c000 ( 118064) : 05.63% 27.70% : RegionUsageImage
2000000 ( 32768) : 01.56% 07.69% : RegionUsageStack
20000 ( 128) : 00.01% 00.03% : RegionUsageTeb
59f0000 ( 92096) : 04.39% 21.61% : RegionUsageHeap
0 ( 0) : 00.00% 00.00% : RegionUsagePageHeap
1000 ( 4) : 00.00% 00.00% : RegionUsagePeb
1000 ( 4) : 00.00% 00.00% :
RegionUsageProcessParametrs
2000 ( 8) : 00.00% 00.00% :
RegionUsageEnvironmentBlock
Tot: 7fff0000 (2097088 KB) Busy: 1a038000 (426208 KB)
-------------------- Type SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
65fb8000 ( 1670880) : 79.68% : <free>
7e13000 ( 129100) : 06.16% : MEM_IMAGE
678000 ( 6624) : 00.32% : MEM_MAPPED
11bad000 ( 290484) : 13.85% : MEM_PRIVATE
-------------------- State SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
e955000 ( 238932) : 11.39% : MEM_COMMIT
65fb8000 ( 1670880) : 79.68% : MEM_FREE
b6e3000 ( 187276) : 08.93% : MEM_RESERVE
Largest free region: Base 1b83a000 - Size 2eed6000 (768856 KB)
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x162122e0
generation 1 starts at 0x161e7cfc
generation 2 starts at 0x00fe1000
ephemeral segment allocation context: none
segment begin allocated size
0017d1f8 7a733370 7a754b98 0x00021828(137256)
0016f800 790d8620 790f7d8c 0x0001f76c(128876)
00fe0000 00fe1000 01c24e58 0x00c43e58(12861016)
16010000 16011000 16260e4c 0x0024fe4c(2424396)
Large object heap starts at 0x01fe1000
segment begin allocated size
01fe0000 01fe1000 02277d78 0x00296d78(2715000)
Total Size 0x116b9b0(18266544)
------------------------------
GC Heap Size 0x116b9b0(18266544)
0:000> .time
Debug session time: Fri Aug 7 03:16:08.276 2009 (GMT-7)
System Uptime: 5 days 10:47:19.218
Process Uptime: 1 days 18:22:25.121
Kernel time: 0 days 1:30:55.406
User time: 0 days 18:20:57.796
0:000> !address -summary
-------------------- Usage SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Pct(Busy) Usage
b2db000 ( 183148) : 08.73% 41.67% : RegionUsageIsVAD
652b4000 ( 1657552) : 79.04% 00.00% : RegionUsageFree
734c000 ( 118064) : 05.63% 26.86% : RegionUsageImage
2100000 ( 33792) : 01.61% 07.69% : RegionUsageStack
21000 ( 132) : 00.01% 00.03% : RegionUsageTeb
65f0000 ( 104384) : 04.98% 23.75% : RegionUsageHeap
0 ( 0) : 00.00% 00.00% : RegionUsagePageHeap
1000 ( 4) : 00.00% 00.00% : RegionUsagePeb
1000 ( 4) : 00.00% 00.00% :
RegionUsageProcessParametrs
2000 ( 8) : 00.00% 00.00% :
RegionUsageEnvironmentBlock
Tot: 7fff0000 (2097088 KB) Busy: 1ad3c000 (439536 KB)
-------------------- Type SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
652b4000 ( 1657552) : 79.04% : <free>
7e13000 ( 129100) : 06.16% : MEM_IMAGE
678000 ( 6624) : 00.32% : MEM_MAPPED
128b1000 ( 303812) : 14.49% : MEM_PRIVATE
-------------------- State SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
fec6000 ( 260888) : 12.44% : MEM_COMMIT
652b4000 ( 1657552) : 79.04% : MEM_FREE
ae76000 ( 178648) : 08.52% : MEM_RESERVE
Largest free region: Base 1b83a000 - Size 2eed6000 (768856 KB)
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x163efe5c
generation 1 starts at 0x16339068
generation 2 starts at 0x00fe1000
ephemeral segment allocation context: none
segment begin allocated size
0017d1f8 7a733370 7a754b98 0x00021828(137256)
0016f800 790d8620 790f7d8c 0x0001f76c(128876)
00fe0000 00fe1000 0189fbd8 0x008bebd8(9169880)
16010000 16011000 164de784 0x004cd784(5035908)
Large object heap starts at 0x01fe1000
segment begin allocated size
01fe0000 01fe1000 02277d78 0x00296d78(2715000)
Total Size 0x1064068(17186920)
------------------------------
GC Heap Size 0x1064068(17186920)
0:000> .time
Debug session time: Fri Aug 7 08:49:15.325 2009 (GMT-7)
System Uptime: 5 days 16:20:26.093
Process Uptime: 1 days 23:55:32.169
Kernel time: 0 days 1:43:42.218
User time: 0 days 21:02:35.218
0:000> !address -summary
-------------------- Usage SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Pct(Busy) Usage
b2e3000 ( 183180) : 08.73% 39.99% : RegionUsageIsVAD
640aa000 ( 1639080) : 78.16% 00.00% : RegionUsageFree
734c000 ( 118064) : 05.63% 25.78% : RegionUsageImage
2300000 ( 35840) : 01.71% 07.83% : RegionUsageStack
23000 ( 140) : 00.01% 00.03% : RegionUsageTeb
75f0000 ( 120768) : 05.76% 26.37% : RegionUsageHeap
0 ( 0) : 00.00% 00.00% : RegionUsagePageHeap
1000 ( 4) : 00.00% 00.00% : RegionUsagePeb
1000 ( 4) : 00.00% 00.00% :
RegionUsageProcessParametrs
2000 ( 8) : 00.00% 00.00% :
RegionUsageEnvironmentBlock
Tot: 7fff0000 (2097088 KB) Busy: 1bf46000 (458008 KB)
-------------------- Type SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
640aa000 ( 1639080) : 78.16% : <free>
7e13000 ( 129100) : 06.16% : MEM_IMAGE
678000 ( 6624) : 00.32% : MEM_MAPPED
13abb000 ( 322284) : 15.37% : MEM_PRIVATE
-------------------- State SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
1061c000 ( 268400) : 12.80% : MEM_COMMIT
640aa000 ( 1639080) : 78.16% : MEM_FREE
b92a000 ( 189608) : 09.04% : MEM_RESERVE
Largest free region: Base 1b83a000 - Size 2eed6000 (768856 KB)
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x1787fed4
generation 1 starts at 0x17811000
generation 2 starts at 0x00fe1000
ephemeral segment allocation context: none
segment begin allocated size
0017d1f8 7a733370 7a754b98 0x00021828(137256)
0016f800 790d8620 790f7d8c 0x0001f76c(128876)
00fe0000 00fe1000 01d502bc 0x00d6f2bc(14086844)
17810000 17811000 178d5e10 0x000c4e10(806416)
Large object heap starts at 0x01fe1000
segment begin allocated size
01fe0000 01fe1000 02277d78 0x00296d78(2715000)
Total Size 0x110bdd8(17874392)
------------------------------
GC Heap Size 0x110bdd8(17874392)
Kjell Gunnar
Yes we can now see that the managed heap size is jumping a bit up and
down and that’s perfectly normal I think.
However your native heap seems to be constantly growing:
59f0000 ( 92096) : 04.39% 21.61% : RegionUsageHeap
65f0000 ( 104384) : 04.98% 23.75% : RegionUsageHeap
75f0000 ( 120768) : 05.76% 26.37% : RegionUsageHeap
I wonder if your above measurements were done while the DebugDiag was
active and monitored your apps for leaks?
If so you need to examine the leak analyze output carefully and try to
find which heaps that
contribute most to the RegionUsageHeap increase.
The DebugDiag is a great tool for native leaks, but I don’t think it
can contribute much on managed leaks.
My own experience is that while DebugDiag monitors an app for leak,
it injects a dll into its address space which makes its own heap for
collecting statistics.
This heap grows and is therefore contributing to increase in native
heap usage.
Good luck
Kjell Gunnar
On Aug 7, 10:09 pm, rick <rkhaer...@yahoo.com> wrote:
> Kjell,
>
> Yes, the app is all C# .NET 3.5 framework (four projects with two DLLs
> and one service). It looks like I might have the fixed the managed
> side (GS Heap Size?). The RegionUsageHeap continues to rise. Yourhttp://msdn.microsoft.com/en-us/library/ms954591.aspxlink seems to
> lean that way. I stumbled acrosshttp://blogs.msdn.com/johan/archive/2008/02/21/walkthrough-troublesho...