Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

How to use memory commands to detect memory leak in tcl script.

586 views
Skip to first unread message

Rohan Yadav

unread,
Jan 23, 2018, 1:00:05 PM1/23/18
to
Hello all,
In my organization we are currently running 2000+ tcl script simultaneously to perform the business function. Currently we are experiencing memory outage each day. We decided to check if tcl scripts are leaking memory or not.
Is there a way from memory command from which I can see if tcl scripts are leaking memory or not. I tried running memory active command together with memory tag and it didn't showed anything useful. Does anybody knows how to use memory command to detect memory leak.
P.S. We cannot run each and every single script in isolation with valgrind due to complication and setup required to run them.

Don Porter

unread,
Jan 23, 2018, 2:08:29 PM1/23/18
to
On 01/23/2018 01:00 PM, Rohan Yadav wrote:
> In my organization we are currently running 2000+ tcl script simultaneously...

Does that mean 2000 Tcl_Interp's concurrently running, each with a
single script? Or something else?

If it is 2000 concurrent interp's, what is the concurrency mechanism?
Separate processes? Separate threads in one process? Something else?

--
| Don Porter Applied and Computational Mathematics Division |
| donald...@nist.gov Information Technology Laboratory |
| http://math.nist.gov/~DPorter/ NIST |
|______________________________________________________________________|

rohandr...@gmail.com

unread,
Jan 23, 2018, 2:18:10 PM1/23/18
to
We have by default 10 threads(customizable with config option) inside a process to execute the tcl scripts. 10 threads have there own interpreter. So at a time 10 interpreter's are running simultaneously.

Don Porter

unread,
Jan 23, 2018, 2:59:16 PM1/23/18
to
On 01/23/2018 02:18 PM, rohandr...@gmail.com wrote:
> We have ... 10 threads inside a process [each with its] own interpreter.

>>> we are experiencing memory outage each day.

What are the symptoms of a "memory outage" precisely?

When you monitor the memory usage of your 10-thread, Tcl-intepreting
program, do you see unexplained memory growth?

What are the tools available on your platform to examine memory usage?

Does your program make use of any C-coded extensions that you have
developed for yourselves? Or is it just Tcl + Thread + scripts?

What are the versions of the software components in use?

Rohan Yadav

unread,
Jan 23, 2018, 3:48:48 PM1/23/18
to
Our software uses C-coded extension to execute the tcl script. It's been running on windows platform and we are using tcl version 8.5.2.
We ran our software with Dr. Memory tool(similar to valgrind for windows platform) and analyzed if there are any memory leak present in our codebase and it didn't showed anything suspicious in our codebase.

Typical memory outage scenrario includes, client had to restart the process each day. It reaches maximum memory before end of the day, and our process crashes due to reaching maximum memory limit. And yes we do see unexplained memory growth each day and thats what we are trying to figure it out.

We tried with disabling multiple threads, and running it on single thread but still problem persist.
One of our tcl developer looked into most runned tcl script for client, and he found out that there was some problem with the tcl code. Basically we were not feeing up the memory for tcl script.(For tdom package, client were deleting the dom object.)

So our purpose is to find the script that has these kind of leaks without manually gpoing through each and every script.
We though that using memory command inside the tcl script would give some useful information, but that haven't been the case.

Is there is a way to find the leaks using memory command inside the tcl script.

Gerald Lester

unread,
Jan 23, 2018, 7:13:36 PM1/23/18
to
When the interp is first cut:
1) Record all of the namespaces
2) Record all of the global variables in the namespaces
3) Record the size of all arrays
4) Record the size of all dictionaries

When the interp is finish running a script compare the current
namespaces, globals, array size and dictionary sizes -- if they have
increased you have a leak.

The easiest way to fix this is to just cut a new interp each time and
not reuse them.

--
+----------------------------------------------------------------------+
| Gerald W. Lester, President, KNG Consulting LLC |
| Email: Gerald...@kng-consulting.net |
+----------------------------------------------------------------------+

Rohan Yadav

unread,
Jan 23, 2018, 8:32:07 PM1/23/18
to
Hi Gerald,
Is there is a way to record all the above mentioned data. I was hoping that memory active command would have given me this info, but thats not the case.

Rich

unread,
Jan 23, 2018, 9:51:56 PM1/23/18
to
Rohan Yadav <rohandr...@gmail.com> wrote:
> On Tuesday, 23 January 2018 16:13:36 UTC-8, Gerald Lester wrote:
>> When the interp is first cut:
>> 1) Record all of the namespaces
>> 2) Record all of the global variables in the namespaces
>> 3) Record the size of all arrays
>> 4) Record the size of all dictionaries
>>
>> When the interp is finish running a script compare the current
>> namespaces, globals, array size and dictionary sizes -- if they have
>> increased you have a leak.
>>
>> The easiest way to fix this is to just cut a new interp each time and
>> not reuse them.
>>
>
> Hi Gerald,
> Is there is a way to record all the above mentioned data.

Yes, instrument your scripts to record the data.

> I was hoping that memory active command would have given me this
> info, but thats not the case.

What is this "memory active command" to which you refer? Tcl has no
"memory active command", so what are you referencing?

Gerald Lester

unread,
Jan 23, 2018, 10:45:37 PM1/23/18
to
I assume he is referring to the [memory] command:
http://www.tcl.tk/man/tcl8.6/TclCmd/memory.htm.

That command is used to debug memory usage in an extension.

Technically, the Tcl scripts are not leaking memory -- they just are not
cleaning up after themselves.

Rohan, is this a NaviServer application?

Rohan Yadav

unread,
Jan 24, 2018, 1:24:21 PM1/24/18
to
Hi Ritch,
I am referencing to memory commands supported by tcl
https://www.tcl.tk/man/tcl8.4/TclCmd/memory.htm

Rohan Yadav

unread,
Jan 24, 2018, 1:25:45 PM1/24/18
to
Hi Gerald,
Its not a NaviServer application. Its our inhouse application used for other projects.

Jim

unread,
Jan 24, 2018, 6:20:03 PM1/24/18
to
On 1/23/2018 7:45 PM, Gerald Lester wrote:
> Technically, the Tcl scripts are not leaking memory -- they just are not
> cleaning up after themselves.

This is mostly going to be a user inflicted wound, right? As opposed to
say an inherent memory cleanup issue that tcl suffers by design or
implementation. In otherwords, it's not tcl specific in any way...

And an effective strategy for memory clean-up:
Had the 'script' been encapsulated inside of a procedure, one
example, and a call made to that procedure -- then the memory would be
returned when the call returns and the 'scripts' would be effectively
cleaning up after themselves...(except for variables declared to be
global vs local, etc).

Jim

Gerald Lester

unread,
Jan 24, 2018, 6:53:43 PM1/24/18
to
Correct, unless the memory is being used by commands, e.g. doms created
by tDOM or TclOO since they are all commands.

Rich

unread,
Jan 24, 2018, 7:31:20 PM1/24/18
to
Jim <lmn...@gmail.com> wrote:
> On 1/23/2018 7:45 PM, Gerald Lester wrote:
>> Technically, the Tcl scripts are not leaking memory -- they just are not
>> cleaning up after themselves.
>
> This is mostly going to be a user inflicted wound, right?

Most likely.

> As opposed to say an inherent memory cleanup issue that tcl suffers
> by design or implementation. In otherwords, it's not tcl specific in
> any way...

Unlikely. Core Tcl is very good at cleaning up after itself. But it
can't know if the user is lappending data to a global list and never
removing it (but one possible example).

>
> And an effective strategy for memory clean-up:
> Had the 'script' been encapsulated inside of a procedure, one
> example, and a call made to that procedure -- then the memory would
> be returned when the call returns and the 'scripts' would be
> effectively cleaning up after themselves...(except for variables
> declared to be global vs local, etc).

Yep. One of the advantages of putting all one's code into procs.

But they also hinted at several C extensions they are loading. It is
much more likely that there is a leak in one of their C extensions than
there is any leak in core Tcl.

Rich

unread,
Jan 24, 2018, 7:33:02 PM1/24/18
to
Gerald Lester <Gerald...@kng-consulting.net> wrote:
> On 01/24/2018 05:19 PM, Jim wrote:
>> On 1/23/2018 7:45 PM, Gerald Lester wrote:
>>> Technically, the Tcl scripts are not leaking memory -- they just are
>>> not cleaning up after themselves.
>>
>> This is mostly going to be a user inflicted wound, right?  As opposed to
>> say an inherent memory cleanup issue that tcl suffers by design or
>> implementation.  In otherwords, it's not tcl specific in any way...
>>
>> And an effective strategy for memory clean-up:
>>   Had the 'script' been encapsulated inside of a procedure, one
>> example, and a call made to that procedure -- then the memory would be
>> returned when the call returns and the 'scripts' would be effectively
>> cleaning up after themselves...(except for variables declared to be
>> global vs local, etc).
>
> Correct, unless the memory is being used by commands, e.g. doms created
> by tDOM or TclOO since they are all commands.

Except that tDom is also very good at cleaning up after itself as well,
so one has to actually try to get it to not release the memory it
allocates when it is done.

This is possible to do, but if one uses it in its typical manner it is
pretty much self cleaning like the rest of Tcl.

0 new messages