I don't know what causes this problem but I've always wondered about the
threads that end not being cleaned up. I use thread.start to start them
but nothing to end them. I've been told VB will clean these up.
One thing that didn't change over from 2003 to 2005 was
Thread.ApartmentState = ApartmentState.STA. 2005 says it obsolete but
the program runs so I ignored it's warning. Could this be causing a
problem?
Any ideas?
Each time you spin up a new thread, it's going to take up at least 1 meg of
fully committed virtual memory. You can read all the gory details about this
at:
http://www.bluebytesoftware.com/blog/PermaLink,guid,733d7537-f982-4886-af62-66bed0f97ab5.aspx
You really, really need to use a different model is you want your
application to scale in any meaningful way. Async I/O is really your only
long term solution.
Typically, you don't need to shutdown threads - they just go away when their
top level method exits. If you poke at your application, use a Performance
Counter to tell how many threads you've got running. If the number is very
high, then something is keeping the threads heald open. Do NOT (don't!
really!) just call "Kill Thread" on the open threads if this is the case -
figure out what the problem is and fix it.
You don't need to worry about the Apartment state stuff - it's not relevant
for a socket app.
--
Chris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP
http://www.coversant.com/blogs/cmullins
"cj" <c...@nospam.nospam> wrote in message
news:OJHszDsp...@TK2MSFTNGP05.phx.gbl...
Hope this helps. Please feel free to let us know if you have any other
questions or concerns.
Have a good day!
Best regards,
Charles Wang
Microsoft Online Community Support
=====================================================
Get notification to my posts through email? Please refer to:
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications
If you are using Outlook Express, please make sure you clear the check box
"Tools/Options/Read: Get 300 headers at a time" to see your reply promptly.
Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
======================================================
When responding to posts, please "Reply to Group" via
your newsreader so that others may learn and benefit
from this issue.
======================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
======================================================
I'd like to see some documentation on the statement that each new thread
consumes 1MB virtual memory. If you can point me to an article I'd
appreciate it.
Anyway, the original reply was:
Well, handling a request with a new thread is a pretty bad way to ago.
Each time you spin up a new thread, it's going to take up at least 1 meg of
fully committed virtual memory. You can read all the gory details about this
at:
http://www.bluebytesoftware.com/blog/PermaLink,guid,733d7537-f982-488...
You really, really need to use a different model is you want your
application to scale in any meaningful way. Async I/O is really your only
long term solution.
Typically, you don't need to shutdown threads - they just go away when their
top level method exits. If you poke at your application, use a Performance
Counter to tell how many threads you've got running. If the number is very
high, then something is keeping the threads heald open. Do NOT (don't!
really!) just call "Kill Thread" on the open threads if this is the case -
figure out what the problem is and fix it.
You don't need to worry about the Apartment state stuff - it's not relevant
for a socket app.
--
Chris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP
http://www.coversant.com/blogs/cmullins
"cj" <c...@nospam.nospam> wrote in message
news:uKMXLy2p...@TK2MSFTNGP06.phx.gbl...
Open Process Explorer, click View->Show Lower Panel, select Lower Panel
View-> Handles.
You may check if the thread number is growing and the handles are not
closed.
Dim oSession As SessionClass
Dim oThread As Thread
oThread = New Thread(AddressOf oSession.ThreadMain)
oThread.Start()
It's my understanding that when the sub ThreadMain ends the tread also
ends. This is true right? I have not heard of closing a thread handle.
The first thing that ThreadMain does is increment a counter and the last
thing is decrement the counter. The counter is another class which uses
synclock to keep the threads from conflicting with eachother. This
counter is read an displayed by a timer in the main program to show how
many threads are running at any given time. It usually runs between 1
and 3. Part of the processing being done by each thread is contacting a
sql server and if the sql server is having problems processing by this
program backs up and I've seen it with over 100 threads open before. It
handles it fine as the sql request time out this program returns and
error response to the TCP/IP client that the thread is handling and the
sub/tread ends. It just takes much longer to timeout than to get a
response from SQL most of the time so during those times the concurrent
thread count will be much higher.
Dim oSession As SessionClass
Dim oThread As Thread
oThread = New Thread(AddressOf oSession.ThreadMain)
oThread.Start()
It's my understanding that when the sub ThreadMain ends the tread also
ends. This is true right? I have not heard of closing a thread handle.
The first thing that ThreadMain does is increment a counter and the last
thing is decrement the counter. The counter is another class which uses
synclock to keep the threads from conflicting with eachother. This
counter is read an displayed by a timer in the main program to show how
many threads are running at any given time. It usually runs between 1
and 3. Part of the processing being done by each thread is contacting a
sql server and if the sql server is having problems processing by this
program backs up and I've seen it with over 100 threads open before. It
handles it fine as the sql request time out this program returns and
error response to the TCP/IP client that the thread is handling and the
sub/tread ends. It just takes much longer to timeout than to get a
response from SQL most of the time so during those times the concurrent
thread count will be much higher.
Dim oSession As SessionClass
Dim oThread As Thread
oThread = New Thread(AddressOf oSession.ThreadMain)
oThread.Start()
It's my understanding that when the sub ThreadMain ends the tread also
ends. This is true right? I have not heard of closing a thread handle.
The first thing that ThreadMain does is increment a counter and the last
thing is decrement the counter. The counter is another class which uses
synclock to keep the threads from conflicting with eachother. This
counter is read an displayed by a timer in the main program to show how
many threads are running at any given time. It usually runs between 1
and 3. Part of the processing being done by each thread is contacting a
sql server and if the sql server is having problems processing by this
program backs up and I've seen it with over 100 threads open before. It
handles it fine as the sql request time out this program returns and
error response to the TCP/IP client that the thread is handling and the
sub/tread ends. It just takes much longer to timeout than to get a
response from SQL most of the time so during those times the concurrent
thread count will be much higher.
The 1 meg of fully committed virtual memory the threads use should not
be kept after the threads end according to what I've seen/read. And if
I'm right then letting each thread have a meg is no problem.
I recommend that you run perfmon.exe to monitor the counter "Thread Count"
and "Virtual Bytes" of your process to see if this issue was caused by
thread explosion.
Hope this helps. If you have any other questions or concerns, please feel
free to let me know.
I go to the bottom and right click then add counters and can't figure
out what to add. I think I need to set performance object to .net clr
memory but then which items under that do I monitor? I can't find
anywhere to monitor how many threads are running. In task manager I am
monitoring threads running and virtual memory.
I don't have a clue what I'm looking at in perfmon.
For the CLR memory, you can select it from "Performance object" list, but I
am not sure if it is a helpful counter since the thread objects are
unmanaged.
Thanks
I've been running perfmon all day and this is what I have now.
Threadcount has fluctuated normally. I see no upward trend at all.
Virtual Bytes seems to be stepping up every few hours. I don't knwo
what to make of private bytes and working set as they are going up at
about a 45 degree angle--(they seem to indicate a problem but I don't
know what). This program has been running for over a year now without
any problems but the pc is scheduled to reboot weekly. We never looked
at the memory until now. The scheduled reboot got turned off and it
went two weeks and someone noticed an error on the screen that said we
were running out of virtual memory. That's what got us started looking
into this. The graph is set to update every 5 minutes.
Thanks for your help
http://img167.imageshack.us/my.php?image=mozscreenshot2mq9.jpg
Also, you may refer to the section Step 2: Identify Workload of this
article:
Performance Testing for Application Blocks
http://msdn2.microsoft.com/en-us/library/ms998241.aspx
NOTE: This issue is memory related. To track the root cause, memory dump
analysis may be required, however this work can only be done by CSS. Our
managed newsgroup is focused on break/fix issues that are neither urgent
nor complex. If the issue is urgent to your business, it is recommended
that you contact Microsoft Customer Support Services (CSS) via telephone so
that a dedicated Support Professional can assist you in a more efficient
manner. Please be advised that contacting phone support will be a charged
call.
To obtain the phone numbers for specific technology request please take a
look at the web site listed below.
http://support.microsoft.com/default.aspx?scid=fh;EN-US;PHONENUMBERS
If you are outside the US please see http://support.microsoft.com for
regional support phone numbers.
Best regards,
Charles Wang
Microsoft Online Community Support
=====================================================
In .Net, Memory is managed, but filehandles, COM Objects and other things
are not.
--
Rory
Do you see anything wrong with this class? It called by all my threads
whenever they need to log a transaction.
Public Class MyStringLogger
Private Shared m_loglock As New Object
Public Shared Sub Write(ByVal fileName As String, ByVal strToWrite
As String)
SyncLock (m_loglock)
Try
Dim sw As New System.io.StreamWriter(fileName, True)
sw.WriteLine(strToWrite)
sw.Close()
Catch
'if it fails forget it--not really that important
anyway
End Try
End SyncLock
End Sub
End Class
Do I need sw.dispose() after sw.close()?
I've always wondered how to ensure it's closed in case an error occurs
in opening the file or writing to it etc. In the case above if it fails
to write the file it would be left open I suppose--then again since it
is declared inside the try catch block it should cease to exist when
that section of code is done executing. Right? The network might be
down and the file can't be written to so I have to plan for this stuff.
For managed and unmanaged code interoperability, you can refer to:
An Overview of Managed/Unmanaged Code Interoperability
http://msdn2.microsoft.com/en-us/library/ms973872.aspx
Unmanaged code may cause memory leak however even managed code may also
probably cause the issue which appears to have a memory leak. I recommend
that you refer to this article:
How to identify memory leaks in the common language runtime
http://support.microsoft.com/kb/318263
You may refer to the KB article and check if there is similar issue in your
code.
Please feel free to let me know if you have any other questions or
concerns. I am glad to work with you for further research.
Best regards,
Charles Wang
Microsoft Online Community Support
=====================================================
Get notification to my posts through email? Please refer to:
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications
If you are using Outlook Express, please make sure you clear the check box
"Tools/Options/Read: Get 300 headers at a time" to see your reply promptly.
Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
======================================================
No. sw.Close calls dispose internally. In fact, the implementation
is:
Public Sub Close ()
Me.Dispose(True)
GC.SupressFinalize(Me)
End Sub
CJ, you might want to look at VB.NET's USING statement. It makes a
lot of this stuff much easier, since it guarentees that Dispose will
be called - even if you exit via an exception. I would probably
implement your above block somehting like:
SyncLock (m_loglock)
Try
Using sw As New StreamWriter(fileName, True)
sw.WriteLine(strToWrite)
End Using ' automatically closed
Catch
End Try
End SyncLock
Anyway, HTH
--
Tom Shelton
Not sure if you noticed the description in the KB article 318263:
=============================================
Additionally, a project may only appear to have a memory leak. This
condition can occur if many large objects (such as DataTable objects) are
declared and then added to a collection (such as a DataSet). The resources
that these objects own may never be released, and the resources are left
alive for the whole run of the program. This appears to be a leak, but
actually it is just a symptom of the way that memory is being allocated in
the program.
=============================================
All objects that implement IDispose interface have a Dispose method.
Dataset implement IDispose interface, so you can call explicitly call it to
release resources. Otherwise, you will wait for GC releasing the objects.,
however we do not know when GC collect the memory. For those temorporay
objects, such as returned large objects, please explicityly call Dispose
method to release resources after you use them.
Just feel free to post back if you encounter any problems in future. Have a
good day!
Have a good day!
Charles Wang
Microsoft Online Community Support
======================================================
I can't say that I've seen any improvement. Private Bytes and Working
Set climb steadily during the day when the system is busy. Virtual
Bytes slowly climbs. It'll remain flat for hours then jump up. Threads
have shown no signs of increase. I'm using dispose for everything that
offers it now. I haven't had time to look into Using yet.
Thanks for checking,
cj
Please feel free to post back once you collect the data. I am glad to work
with you for further assistance.
Have a nice day!
Best regards,
Charles Wang
Microsoft Online Community Support
=====================================================
Get notification to my posts through email? Please refer to:
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications
If you are using Outlook Express, please make sure you clear the check box
"Tools/Options/Read: Get 300 headers at a time" to see your reply promptly.
Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
I added the counters yesterday and it's been almost 24 hours now.
# Bytes in All Heaps fluctuates but has shown no signs of increase
Last 585284, Avg 565394, Min 399088, Max 860336
Large Object Heap Size hasn't changed since it started
Last 33816, Avg 33816, Min 33816, Max 33816
% Time in GC has fluctuated but is very small
Last .002, Avg .004, Min .001, Max .016
Private bytes continues to climb whever the system is active
Last 117538816, Avg 88500153, Min 25710592, Max 120004608
Working Set also continues to climb at the same rate as private bytes
Last 119078912, Avg 89172462, Min 25686016, Max 119353344
Virtual Bytest continues to climb in a stair step fashion--jumps up
every few hours.
Last 294690816, Avg 254321252, Min 161128448, Max 300130304
Thread Count shows no signs of increase
Last 16, Avg 15, Min 13, Max 18
I still need to look at Using. I've been busy on other things. This
doesn't cause a problem unless it goes over a week and normally it's
rebooted weekly. But I'm very curious as to what is going on and I
really appreciate you help.
Charles Wang[MSFT] wrote:
> Hi cj,
> In this case, I recommend that you also add the following counters to
> monitor your .NET application to see if the CLR memory is being used highly:
> .NET CLR Memory\# Bytes in All Heaps
> .NET CLR Memory\Large Object Heap Size
> .NET CLR Memory\% Time in GC
Please feel free to let me know if you have any other questions or
concerns.
Best regards,
Public Class MyStringLogger
Private Shared m_loglock As New Object
Public Shared Sub Write(ByVal fileName As String, ByVal
strToWrite As String)
SyncLock (m_loglock)
'remark out this try catch block to prevent all writing
to disk
Try
Dim sw As New System.io.StreamWriter(fileName, True)
sw.WriteLine(strToWrite)
sw.Close()
sw.Dispose()
Catch
End Try
End SyncLock
End Sub
End Class
My SQL connections are closed and disposed as soon as they are no longer
needed.
I've contacted Catalyst the COM provider and they advised me to use an
uninitialize method which I now use. They do not provide a dispose method.
If your SQL connections are closed and disposed as soon as they are no
longer needed, they won't be leaked. For the COM objects or references, you
may check if they provide a method to release themselves. Dispose method is
just a .NET class who implemented IDispose interface, however for those
unmanaged objects, if they were not wrapped in a .NET class which
implemented IDispose interface, you need to use their self-provided release
function to release themselves.
From the perfmon monitor result, we can see that this is a unmanged
resources leakage issue. The managed resources were controlled very well in
your application. I do not know what the COM component you used was, but
there might be another possible reason, if the COM component itself has
memory leaks, it is hard for you to correct it in your .NET application.
I recommend that you work with your COM developers to identify which COM
objects and references need to be released in what way.
Hope this helps. If you have any other questions or concerns, please feel
free to post back. I am glad to work with you for further assistance.
I'd already spoken with the com vendor and he told me to make sure I
called .Uninitialize. I am but that doesn't help.
I don't like knowing it's using increasing amounts of memory and I can't
explain it but it runs incredibly well as long as the program is
restarted at least once a week. Thankfully that isn't a problem as when
we first wrote the program we had worried about the ability of the pc to
run a multi threaded app for weeks at a time w/o a reboot and the
program was written to reboot the pc on selected days at 2:00am. This
"memory leak" might have been occurring all along but nobody noticed
until the automatic reboot was turned off and it went just over 2 weeks
an gave the memory error.
I appreciate your help but I don't know what more we can do. As I have
time I'll keep trying things.
Since this issue is complex and may need dump analysis for tracking the
root cause, I would like to recommend a senior support channel for you.
Effectively and immediately you can contact Microsoft Customer Support
Services (CSS) via telephone so that a dedicated Support Professional can
assist you in a more efficient manner. Please be advised that contacting
phone support will be a charged call.
To obtain the phone numbers for specific technology request please take a
look at the web site listed below.
http://support.microsoft.com/default.aspx?scid=fh;EN-US;PHONENUMBERS
If you are outside the US please see http://support.microsoft.com for
regional support phone numbers.
If you have any other questions or concerns, please feel free to let me
know.
Have a good day!