tcl and twapi use a huge amount of memory with a little script, why?

[MartinLemburg@Siemens-PLM]

Hi,

I have a little script using twapi 4.1.27 and tcl 8.6.4 getting all processes with the following information:

- PID
- idle flag
- system flag
- name
- path
- list of threads with TID and state
- list of modules with handle and name

I use the following script to collect those information:

foreach pid [twapi::get_process_ids] {
set processInfo [dict create \
pid $pid \
idle [twapi::is_idle_pid $pid] \
system [twapi::is_system_pid $pid] \
name [twapi::get_process_name $pid] \
path [twapi::get_process_path $pid] \
threads [lmap tid [twapi::get_process_thread_ids $pid] {
dict create \
tid $tid \
state [lindex [twapi::get_thread_info $tid -state] 0]
}] \
modules [list] \
]

if {[catch {
set modulesInfo [twapi::get_process_modules $pid -handle -name]
}] == 0} {
dict set processInfo modules [lmap \
moduleInfo \
[twapi::recordarray getdict $modulesInfo] {
dict map \
{key value} \
$moduleInfo {
set key [string range $key 1 end]
}
}
]
}

dict set processes $pid $processInfo
}

At the end the tclsh executable managed around 2.55 GB (max. 3.00 GB), while …

% dict size $processes
195
% tcl::unsupported::representation $processes
value is a dict with a refcount of 3, object pointer at 000000000480F580, internal representation 0000000004958430:0000000000000000, no string representation
% string length $processes
281574

Requesting the module paths and sizes boosted up the memory usage to 4 GB (max. 6 GB).

Is this a twapi or a tcl dict issue?

Best regards,

Martin

[MartinLemburg@Siemens-PLM]

I re-wrote the script to not use dictionaries:

foreach pid [twapi::get_process_ids] {
set processInfo [list \

pid $pid \
idle [twapi::is_idle_pid $pid] \
system [twapi::is_system_pid $pid] \
name [twapi::get_process_name $pid] \
path [twapi::get_process_path $pid] \
threads [lmap tid [twapi::get_process_thread_ids $pid] {

list \

tid $tid \
state [lindex [twapi::get_thread_info $tid -state] 0]
}] \

]

if {[catch {
set modulesInfo [twapi::get_process_modules $pid -handle -name]

}] == 1} {
lappend processInfo modules [list]
} else {
lappend processInfo modules [lmap \
moduleInfo [twapi::recordarray getlist $modulesInfo] {
lmap key {handle name} value $moduleInfo {
list $key $value
}
} \
]
}

lappend processes $pid $processInfo
}

At the end the maximum memory consumption was around 3.6 GB again, but at least only 150 MB was left over at the end.

The variable processes held a comparable amount of data:

% llength $processes
398
% string length $processes
238274

So it is a tcl dict issue to use so much memory at the end, but why while collecting the data?

Best regards,

Martin

[Gerald Lester]

>>...
Not commenting on the memory issue, but a general observation, you do
not need the processInfo dictionary at all, you could have set processes
directory:
foreach pid [twapi::get_process_ids] {
dict set processes $pid pid $pid
dict set processes $pid idle [twapi::is_idle_pid $pid]
....
}

Also, a style point, it might be better to use proceesesDict instead of
just processes.


--
+----------------------------------------------------------------------+
| Gerald W. Lester, President, KNG Consulting LLC |
| Email: Gerald...@kng-consulting.net |
+----------------------------------------------------------------------+

[MartinLemburg@Siemens-PLM]

Hi Gerald,

You are right, there are different ways to style the code.

I'm a bit concerned to call "dict set" several times, if I can create one dictionary and assign it to a key at once.

But additional I try to point out, that there is a sub data structure holding dedicated process related data, which is managed inside a dictionary associated to the process identifier.

And as last point "dict lappend" cannot work this way:
dict lappend processesDict $pid modules $moduleInfoDict

Thus I need to do:
dict set processesDict $pid modules \
[linsert [dict get $processesDict $pid modules] end $moduleInfoDict]

Or in long:
set pidInfoDict [dict get $processesDict $pid]
dict lappen pidInfoDict modules $moduleInfoDict
dict set processesDict $pid $pidInfoDict

Or:
set modules [dict get $processesDict $pid module]
lappen modules $moduleInfoDict
dict set processesDict $pid modules $modules

Thousand ways lead to Rome.

Happy weekend!

[Ashok]

Martin,

I'll take a look at this in the next couple of days to see if twapi is
leaking memory somewhere.

I'm also curious because my WiTS application does something very similar
but does not show this kind of memory growth.

/Ashok

[Ashok]

OK, having traced through the code, twapi does indeed have a memory leak
associated with the get_thread_info call which is causing the memory
bloat you are seeing. This will be fixed in the next 4.2 beta release.

On a different note, from a performance point of view I think you would
be better off using the performance counter api to retrieve the counters
you seem to be interested in. See
http://www.magicsplat.com/book/performance_counters.html. Not sure it
will give you everything but worth checking.

/Ashok