v1.1 Memory Leak w/ Multiple Threads (test code included)

[Brian Adkins]

I've encountered an unusual memory leak in v1.1 (therefore probably v1.0
also) that involves multiple threads and a db connection.

If the program is modified to be single threaded, OR the db connection is
removed, the leak disappears. What's strange about the db connection prereq
is that it doesn't even have to be used. If you examine the code below,
simply opening a connection even if it's immediately closed and unused
thereafter is enough to cause the memory leak.

Compile the program below with v1.1 and fire up perfmon and look at the gen
0, 1, 2 and large object heaps. All you need to do is supply a valid
connection string for your SQL server; otherwise, the code is entirely self
contained and will run as is.

Is this the best place to submit a reproducible error to Microsoft, or is
there a better forum?

Brian Adkins

--- snip ---
using System;
using System.Collections;
using System.Data.SqlClient;
using System.Net;
using System.Text;
using System.Threading;

/// <summary>
/// This class demonstrates an interesting bug in the
/// .NET memory management system.
///
/// To cause a memory leak in this program, there are
/// two prerequisites:
/// 1) An SqlConnection (or OleDbConnection) must be opened.
/// It can be immediately closed and unused afterward!
/// 2) The program must be multi-threaded.
///
/// If either of the above two prerequisites are not met,
/// memory does not leak!
/// </summary>
class MemLeak
{
private const string CONNECTION_STRING =
"data source=<your source>;initial catalog=<your catalog>;user
id=<your user>;" +
"password=<your pswd>;persist security info=True;" +
"workstation id=<your workstation>;packet size=4096";

private const int NUM_THREADS = 3;
private const int SLEEP_MS = 250;

private Queue _pageQueue = new Queue();
private Thread[] _threads;
private string _url;

[STAThread]
static void Main (string[] args)
{
if (args.Length != 1)
{
Console.WriteLine("You must supply a URL as an argument.");
return;
}
#if true
// Simply change the conditional compilation expression from
// true to false to eliminate the memory leak!
SqlConnection conn = new SqlConnection(CONNECTION_STRING);
conn.Open();
conn.Close();
#endif

MemLeak x = new MemLeak(args[0]);
x.Init();
}

public MemLeak (string url)
{
_url = url;
}

public void Init ()
{
// Modify the program to not use multiple threads for adding
// to the queue, and the memory leak disappears!
CreateThreads(NUM_THREADS);

while (true)
{
try
{
string page = DequeuePageElement();
Console.WriteLine("Page length is {0}", page.Length);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Console.WriteLine(ex.StackTrace);
}
}
}

private void CreateThreads (int numThreads)
{
_threads = new Thread[numThreads];

for (int i = 0; i < _threads.Length; ++i)
{
_threads[i] = new Thread(new ThreadStart(ProducerThread));
_threads[i].Start();
}
}

private string DequeuePageElement ()
{
while (true)
{
lock (this)
{
if (_pageQueue.Count > 0)
{
return (string) _pageQueue.Dequeue();
}
}

Thread.Sleep(SLEEP_MS);
}
}

private void ProducerThread ()
{
UTF8Encoding encoding = new UTF8Encoding();
WebClient webClient = new WebClient();
string page;

while (true)
{
try
{
byte[] bytes = webClient.DownloadData(_url);
page = encoding.GetString(bytes);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Console.WriteLine(ex.StackTrace);
page = "";
Thread.Sleep(SLEEP_MS);
}

lock (this)
{
_pageQueue.Enqueue(page);
}
}
}
}
--- snip ---

[Brian Adkins]

Actually, there appear to be 3 prerequisites:

1) Open an SqlConnection at some point in the program - even if it's closed
immediately and unused.
2) Program is multi-threaded
3) WebClient is used - I tried removing this prereq and producing random
strings to simulate pulling web pages and there is no leak

Brian Adkins

[Chris Cummings]

Change the connection string to turn of connection pooling
and see if it still leaks.

>.
>

[Brian Adkins]

Chris, thanks for the reply. I'll try your suggestion of changing
connection pooling this evening, but I'd be surprised if that helps. I'm
curious about what you're suspecting since the connection is not actively
used in the program, it's simply created, opened, closed and never used
again.

"Chris Cummings" <chr...@jetsonsystems.com> wrote in message
news:0da001c32f66$3cf0f5a0$a501...@phx.gbl...

[Chris Cummings]

I thought maybe a side effect of opening the connection
causes the pooling code to start creating more connections
(i.e. filling up the pool) or starts a connection
monitoring thread that may allocate some memory.

>.
>

[Brian Adkins]

Ah. Well even if it did immediately fill up the pool of connections, that
would be a fixed amount of memory, and since there is no database activity
after the initial conn.Open(); conn.Close(); I'd be very surprised if a
pooling side effect was contributing to this, but it's a pretty strange
scenario to begin with.

"Chris Cummings" <chr...@jetsonsystems.com> wrote in message

news:090701c32f74$437b1f00$a001...@phx.gbl...

[Brian Adkins]

As a correction to my post on 6/9/03, this isn't related only to large
objects. If the web pages returned are over the large object threshhold,
then the large object heap increases almost monotonically. If the web pages
returned are under the threshhold, then the gen 2 heap increases almost
monotonically.

To increase the memory consumption rate, supply a URL to a web site
containing a large front page.

Also, as a note, the leak is not related to the Queue growing boundlessly.
The original program limited the number of elements in the Queue, but I
removed that to make the test case smaller. The main thread (consumer) has
no trouble keeping up with at least 3 producer threads.

Brian Adkins

[Russ Bishop]

I assume a forced call to gc.collect doesn't clean things up.

-- russ

"Brian Adkins" <msn...@lojic.com> wrote in message
news:ef9gTVwL...@TK2MSFTNGP11.phx.gbl...

[Brian Adkins]

No, calling GC.Collect() does not free VM. And what good is having
automatic memory management if you have to manually call a garbage collector
method?

Having said that, I did find that calling System.GC.GetTotalMemory(true) in
the main loop of my program does free virtual memory. However, I just
encountered a 14 MB web page that my program tried to download and it
quickly consumed about a gig of VM because it happened during one iteration.
Looks like the WebClient.DownloadData() method is consuming a lot of buffers
that the GC doesn't free.

Now I suppose someone will suggest that I spin off a thread to make the call
to GetTotalMemory() !! So much for automatic memory management.

I'm surprised others haven't been bit by this yet.

"Russ Bishop" <nowhere> wrote in message
news:OCclcR9M...@TK2MSFTNGP11.phx.gbl...

[Russ Bishop]

Sometimes working around the problem helps you determine where exactly the
problem is, which leads to a faster resolution.

-- russ

"Brian Adkins" <msn...@lojic.com> wrote in message

news:e9rlaiTN...@TK2MSFTNGP10.phx.gbl...

[Mark Pearce]

Hi Brian,

Make sure that you're looking at real memory usage, rather than at memory
allocation (the working set). I very much doubt that the call to
GC.GetTotalMemory is reducing the memory used - more likely, it's showing
you the real memory used rather than the memory allocated.

You can reduce the working set programmatically if you wish. See:
http://tinyurl.com/4c46

Does this trick affect performance?
http://tinyurl.com/4c48

HTH,

Mark
--
Author of "Comprehensive VB .NET Debugging"
http://www.apress.com/book/bookDisplay.html?bID=128

"Brian Adkins" <msn...@lojic.com> wrote in message

news:e9rlaiTN...@TK2MSFTNGP10.phx.gbl...

[Brian Adkins]

Mark,
I am looking at virtual memory, and the GC.GetTotalMemory call definitely
results in a garbage collection and reduction in virtual memory for the
process. Without the GC.GetTotalMemory call, the program eventually crashes
due to running out of virtual memory. It's very easy to see this for
yourself by running the supplied code.

You may be thinking of how it is suppose to work instead of how it is
currently broken in v1.1.

A bug resulting in the garbage collector not running will affect virtual
memory, not real memory.

Brian

"Mark Pearce" <ev...@bay.com> wrote in message
news:eUOveAAO...@TK2MSFTNGP10.phx.gbl...

[Willy Denoyette [MVP]]

Brian,
Try changing the STAThread attribute to MTAThread.
The leak you are observing is a handle leak (event kernel sync object - see Perfmon handles count), this leak seems to be related to
the threading model used, when a db connection is created in an STA thead you will observe event handle leaks, when a connection is
made in an MTA or unitialized thread no leak occurs.
This looks like a bug.

Willy.

"Brian Adkins" <msn...@lojic.com> wrote in message news:ef9gTVwL...@TK2MSFTNGP11.phx.gbl...

[Brian Adkins]

I'll check it out, but I don't think it's a handle leak. From all
appearances, the garbage collector is simply not running under certain
circumstances. The memory consumption is proportional to the size of new
objects being created.

"Willy Denoyette [MVP]" <willy.d...@skynet.be> wrote in message
news:eXa3ObcP...@tk2msftngp13.phx.gbl...

[Anthony Moore]

This definitely looks like a problem somewhere, and the repro you have
provided is very clean. I'll try to get this issue escalated so we can
determine the best work around and get it fixed in the product if required.
We can't make any sort of guarentee on a public forum in the absense of a
support contract, but the information you have provided definitely
increases the changes of a good outcome. Also, a fix in the product does
not usually mean a fix in a service pack, so it can be years before you see
the result.

--------------------
> From: "Brian Adkins" <msn...@lojic.com>
> Subject: v1.1 Memory Leak w/ Multiple Threads (test code included)
> Date: Tue, 10 Jun 2003 00:04:12 -0400
> Lines: 159
> X-Priority: 3
> X-MSMail-Priority: Normal
> X-Newsreader: Microsoft Outlook Express 6.00.2800.1158
> X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165
> Message-ID: <ef9gTVwL...@TK2MSFTNGP11.phx.gbl>
> Newsgroups:
microsoft.public.dotnet.framework.clr,microsoft.public.dotnet.framework.perf
ormance
> NNTP-Posting-Host: rdu74-152-190.nc.rr.com 24.74.152.190
> Path: cpmsftngxa06.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP11.phx.gbl
> Xref: cpmsftngxa06.phx.gbl
microsoft.public.dotnet.framework.performance:4325
microsoft.public.dotnet.framework.clr:6955
> X-Tomcat-NG: microsoft.public.dotnet.framework.clr

--

This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

Note: For the benefit of the community-at-large, all responses to this
message are best directed to the newsgroup/thread from which they
originated.

[JM Servera]

Hi Brian, try to profile your application with allocation profiler and see
what it happens.
I've had a similar problem some months ago and had to stop the project
because the allocation profiler showed us that it was an unmanaged memory
leak, but an allocation profiler may help you to find some bugs in your
program that may leave references to your objects.
You can donwload allocation profiler in:
http://www.gotdotnet.com/Community/UserSamples/Details.aspx?SampleGuid=36a3e666-6877-4c26-b62d-bfd7cb3154ac

"Brian Adkins" <msn...@lojic.com> escribió en el mensaje
news:OFxShobR...@tk2msftngp13.phx.gbl...

[Aaron Barth [MSFT]]

Hello,

Here is a brief summary of this problem and ways to correct it.

PROBLEM: Memory leak in C# Console app when using a SqlClient SqlConnection.
FACTORS for a repro:
1. Console's Main() function must be decorated with [STAThread] attribute.
2. SqlClient's connection Pooling must be enabled.
3. Main thread blocks and does not pump COM messages.

So given the factors for the repro, here is some information on what I
believe is happening.

1. Main thread does it's thing. (creating threads, and dequeing/enqueuing
etc.) in a tight while loop.
2. The Finalizer thread wakes up and tries to cleanup some COM object
wrappers.
3. The Finalizer thread tries to cleanup a COM+ object that was created on
the main thread due to SQL Connection Pooling.
4. Since this Finalizer is blocked trying to GetToSTA() [thread 0 in this
case] other finalizations will be blocked.
a. This will manifest itself as a managed memory leak since
Finalization cannot occur for any Finalizable objects.

POSSIBLE workarounds listed in previous posts.

1. Change [STAThread] to [MTAThread]. This should work.
2. Brian mentioned that he added a call to System.GC.GetTotalMemory(true)
on the main thread. This cleaned up the memory.
More Information: This cleaned up the memory because this call allowed
the main thread to pump messages and then we would wait until all
Finalizations completed.
3. Someone suggested to remove the Connection Pooling option from the
connection string. This should also work for this specific case. However
if you require connection pooling this doesn't help you :-)

My recommendation would be to change the [STAThread] attribute to
[MTAThread] and the test application should work correctly.

Thanks,

Aaron
--------------------
>X-Tomcat-ID: 794366794
>References: <ef9gTVwL...@TK2MSFTNGP11.phx.gbl>
>MIME-Version: 1.0
>Content-Type: text/plain
>Content-Transfer-Encoding: 7bit
>From: amo...@online.microsoft.com (Anthony Moore)
>Organization: Microsoft
>Date: Thu, 10 Jul 2003 22:59:12 GMT
>Subject: RE: v1.1 Memory Leak w/ Multiple Threads (test code included)
>X-Tomcat-NG: microsoft.public.dotnet.framework.clr
>Message-ID: <mvuckbzR...@cpmsftngxa06.phx.gbl>
>Newsgroups: microsoft.public.dotnet.framework.clr
>Lines: 165
>Path: cpmsftngxa06.phx.gbl
>Xref: cpmsftngxa06.phx.gbl microsoft.public.dotnet.framework.clr:7274
>NNTP-Posting-Host: TOMCATIMPORT2 10.201.218.182