terminated by signal SIGKILL (Forced quit)

[Luc]

My app collects a bunch of data from a SQLite database then runs a query
and plots a chart.

Then it clears the chart with the $::chart delete "all" canvas command.

Then it runs another query and plots the chart again.

Rinse and repeat over and over. It's a loop.

After about 10 queries give or take, it crashes:

terminated by signal SIGKILL (Forced quit)

I've run it many times and it never gets past the 11th run.

Any idea why this is happening?

--
Luc
>>

[Rich]

No, not really, because a sig kill is an external signal being sent to
the process from somewhere else, and we know nothing about the
surrounding environment where you are running this process.

One possibility, is the app's memory usage growing dramatically, such
that the Linux OOM killer (OOM = Out Of Memory) is activating and
killing it? Although I don't know if the OOM killer sends a SIGKILL.

[Robert Heller]

At Fri, 23 Dec 2022 13:25:23 -0300 Luc <n...@no.no> wrote:

>
> My app collects a bunch of data from a SQLite database then runs a query
> and plots a chart.
>
> Then it clears the chart with the $::chart delete "all" canvas command.
>
> Then it runs another query and plots the chart again.
>
> Rinse and repeat over and over. It's a loop.
>
> After about 10 queries give or take, it crashes:
>
> terminated by signal SIGKILL (Forced quit)

I don't believe SIGKILL is generated by any error condition. It is a signal
sent with the kill command (or something simular). Is there some monitoring
process that is "killing" processes that excede some usage limit?

>
> I've run it many times and it never gets past the 11th run.
>
> Any idea why this is happening?
>

--

Robert Heller -- Cell: 413-658-7953 GV: 978-633-5364
Deepwoods Software -- Custom Software Services
http://www.deepsoft.com/ -- Linux Administration Services
hel...@deepsoft.com -- Webhosting Services

[Luc]

On Fri, 23 Dec 2022 16:52:36 -0000 (UTC), Rich wrote:

> One possibility, is the app's memory usage growing dramatically, such
> that the Linux OOM killer (OOM = Out Of Memory) is activating and
> killing it? Although I don't know if the OOM killer sends a SIGKILL.

You are probably right. I am watching the output of 'free -m' and I see
the amount of free memory does drop steadily until it is around 100MB,
then the Tcl application crashes.

Is there anything I can do to prevent that? Some kind of 'flush memory'
command or hack? I am trying to unset all variables I can without breaking
the app completely at the end of every loop iteration, but that is not
solving the problem.

On Fri, 23 Dec 2022 17:06:27 +0000, Robert Heller wrote:
> I don't believe SIGKILL is generated by any error condition. It is a
> signal sent with the kill command (or something simular). Is there some
> monitoring process that is "killing" processes that excede some usage
> limit?

Maybe. I never put it there myself, but maybe Debian has some such
mechanism built in.

--
Luc
>>

[Robert Heller]

At Fri, 23 Dec 2022 14:39:58 -0300 Luc <n...@no.no> wrote:

>
> On Fri, 23 Dec 2022 16:52:36 -0000 (UTC), Rich wrote:
>
> > One possibility, is the app's memory usage growing dramatically, such
> > that the Linux OOM killer (OOM = Out Of Memory) is activating and
> > killing it? Although I don't know if the OOM killer sends a SIGKILL.
>
>
> You are probably right. I am watching the output of 'free -m' and I see
> the amount of free memory does drop steadily until it is around 100MB,
> then the Tcl application crashes.

Yes, this sounds like a memorrry leak.

>
> Is there anything I can do to prevent that? Some kind of 'flush memory'
> command or hack? I am trying to unset all variables I can without breaking
> the app completely at the end of every loop iteration, but that is not
> solving the problem.

You say are doing SQL queries -- what package are you using to do that? I
believe most of the SQL extensions I've used create some kind of query object
which needs to be "freed", usually with some cleanup subcommand. I've
guessing you are missing some data cleanup step.

Also, are you acclumulating results in memory? Maybe you should not do that.

>
>
> On Fri, 23 Dec 2022 17:06:27 +0000, Robert Heller wrote:
> > I don't believe SIGKILL is generated by any error condition. It is a
> > signal sent with the kill command (or something simular). Is there some
> > monitoring process that is "killing" processes that excede some usage
> > limit?
>
>
> Maybe. I never put it there myself, but maybe Debian has some such
> mechanism built in.
>

--

[Rich]

Luc <n...@no.no> wrote:
> On Fri, 23 Dec 2022 16:52:36 -0000 (UTC), Rich wrote:
>
>> One possibility, is the app's memory usage growing dramatically, such
>> that the Linux OOM killer (OOM = Out Of Memory) is activating and
>> killing it? Although I don't know if the OOM killer sends a SIGKILL.
>
>
> You are probably right. I am watching the output of 'free -m' and I see
> the amount of free memory does drop steadily until it is around 100MB,
> then the Tcl application crashes.

Do you have any swap space allocated?

> Is there anything I can do to prevent that?

Determine where the leak is occurring and correct it.

> Some kind of 'flush memory' command or hack? I am trying to unset
> all variables I can without breaking the app completely at the end of
> every loop iteration, but that is not solving the problem.

We know:

App runs - crashes. Afraid without a lot more info all we can offer
are generalities, such as "find the leak and plug it".

> On Fri, 23 Dec 2022 17:06:27 +0000, Robert Heller wrote:
>> I don't believe SIGKILL is generated by any error condition. It is
>> a signal sent with the kill command (or something simular). Is
>> there some monitoring process that is "killing" processes that
>> excede some usage limit?
>
>
> Maybe. I never put it there myself, but maybe Debian has some such
> mechanism built in.

The Linux kernal has had an out of memory killer since somewhere back
about the 2.4 or 2.5 version days. If you are indeed running out of memory
then the source of the signal might just be the kernel out of memory
killer.

https://linux-mm.org/OOM_Killer

[Luc]

On Fri, 23 Dec 2022 18:38:33 +0000, Robert Heller wrote:

> You say are doing SQL queries -- what package are you using to do that?

set ::dbfile "$::BASEDIR/somefile.db"
set ::sqllibrary "$::BASEDIR/libtclsqlite3.so"
load $::sqllibrary
sqlite3 sql $::dbfile -nomutex 1

> Also, are you acclumulating results in memory? Maybe you should not do
> that.

I thought about that, but I've been doing some debugging:

foreach i [info vars] {
if {[array exists $i]} {
rw a /dev/shm/allvars.txt "$i [array size $i]\n"
} else {
rw a /dev/shm/allvars.txt "$i [strlen [set $i]]\n"
}
}

Nothing seems to be growing horribly. Pretty modest numbers actually.


I noticed two interesting things:

1) There is a text widget that gets updated at every iteration.

$textbox delete 1.0 end
$textbox insert end "some data"

Disabling that text widget makes the free available memory drop
more slowly. Not much, but certainly noticeable.

2) I've found that destroying the entire parent window and recreating
everything in it on every iteration makes a big difference. The memory
still drops, but a lot slower. Still drops though. Slow but sure.
That puts a limit on the number of iterations I can run. That's bad news.

--
Luc
>>

[Rich]

Luc <n...@no.no> wrote:
> On Fri, 23 Dec 2022 18:38:33 +0000, Robert Heller wrote:
>
>> You say are doing SQL queries -- what package are you using to do that?
>
> set ::dbfile "$::BASEDIR/somefile.db"
> set ::sqllibrary "$::BASEDIR/libtclsqlite3.so"
> load $::sqllibrary

Why are you doing this above instead of "package require sqlite3"?

> sqlite3 sql $::dbfile -nomutex 1

How are you doing your queries?

>> Also, are you acclumulating results in memory? Maybe you should not do
>> that.
>
> I thought about that, but I've been doing some debugging:
>
> foreach i [info vars] {
> if {[array exists $i]} {
> rw a /dev/shm/allvars.txt "$i [array size $i]\n"
> } else {
> rw a /dev/shm/allvars.txt "$i [strlen [set $i]]\n"
> }
> }

What is "rw a"???

How big does that file become? Because it is also eating up memory
(/dev/shm is the Linux ramdisk, which uses RAM to store its contents).

> Nothing seems to be growing horribly. Pretty modest numbers actually.
>
>
> I noticed two interesting things:
>
> 1) There is a text widget that gets updated at every iteration.
>
> $textbox delete 1.0 end
> $textbox insert end "some data"
>
> Disabling that text widget makes the free available memory drop
> more slowly. Not much, but certainly noticeable.

The 'delete 1.0 end' should make the text widget memory usage not grow
forever.

> 2) I've found that destroying the entire parent window and recreating
> everything in it on every iteration makes a big difference. The
> memory still drops, but a lot slower. Still drops though. Slow but
> sure. That puts a limit on the number of iterations I can run.
> That's bad news.

In your original post you were doing something with graphs. Does the
graph library include a "delete/free/destroy" call that you are
missing?

[Luc]

On Fri, 23 Dec 2022 19:21:26 -0000 (UTC), Rich wrote:

> > set ::dbfile "$::BASEDIR/somefile.db"
> > set ::sqllibrary "$::BASEDIR/libtclsqlite3.so"
> > load $::sqllibrary
>
> Why are you doing this above instead of "package require sqlite3"?

I like that approach because I can bundle the .so library with the
applications I intend to share when they're ready. Including a
package is less simple and lots of people may not have the package.
The one single thing I dislike the most about Linux is this culture
of requiring packages or dependencies that many people may not have.
So inconsiderate! I love that so much about Tcl works without such
external dependencies.

For the record, my code does try to load the package and falls back
to the .so library if the package is not found. I care about making my apps
run even in the face of incidents.

So what is wrong with using the .so library? Is it bad? Why? What are
the undesired side effects?

> > sqlite3 sql $::dbfile -nomutex 1
>
> How are you doing your queries?

All according to this page:

https://www.sqlite.org/tclsqlite.html

> > I thought about that, but I've been doing some debugging:
> >
> > foreach i [info vars] {
> > if {[array exists $i]} {
> > rw a /dev/shm/allvars.txt "$i [array size $i]\n"
> > } else {
> > rw a /dev/shm/allvars.txt "$i [strlen [set $i]]\n"
> > }
> > }
>
> What is "rw a"???

rw is a proc that reads or writes to files. "rw a" is write/append.
That piece of code above was introduced later, to investigate the
problem that was observed before the code was introduced.

> How big does that file become? Because it is also eating up memory
> (/dev/shm is the Linux ramdisk, which uses RAM to store its contents).

Not big at all. A dozen Kb. It just prints the sizes of variable contents, not
the contents.
That piece of code above was introduced later, to investigate the
problem that was observed before the code was introduced.

> The 'delete 1.0 end' should make the text widget memory usage not grow
> forever.

Well, yes, and it keeps the content visible too. Without it, the text
widget would scroll so far down it could possibly find oil.

> In your original post you were doing something with graphs. Does the
> graph library include a "delete/free/destroy" call that you are
> missing?

My graph library is Tk and canvas. No third-party library whatsoever.

Is it reasonable that I have to destroy and recreate the entire GUI
at every iteration so the application remains barely stable?
The chart is far from being visually intense or busy. It's quite simple.
Why is it eating so much memory? Is there some "best practice" I am
overlooking here to avoid that kind of pitfall?


--
Luc
>>

[Rich]

Luc <n...@no.no> wrote:
> On Fri, 23 Dec 2022 19:21:26 -0000 (UTC), Rich wrote:
>
>> > set ::dbfile "$::BASEDIR/somefile.db"
>> > set ::sqllibrary "$::BASEDIR/libtclsqlite3.so"
>> > load $::sqllibrary
>>
>> Why are you doing this above instead of "package require sqlite3"?
>
>
> I like that approach because I can bundle the .so library with the
> applications I intend to share when they're ready.

Which you can still do, while supporting "package require X". Just
include the package loader scripts, and do an approprate "lappend
auto_path" at startup.

> Including a package is less simple and lots of people may not have
> the package.

Which is noble -- but does not require you directly load the .so file.

> So what is wrong with using the .so library? Is it bad? Why? What are
> the undesired side effects?

Some of the package load scripts perform more than simply a "load
*.so". For that subset you'll not have everything you should have
available. Whether your specific use is impacted I cannot say.

>> > sqlite3 sql $::dbfile -nomutex 1
>>
>> How are you doing your queries?
>
> All according to this page:
>
> https://www.sqlite.org/tclsqlite.html

Sigh... Not a useful answer in the least.

>> > foreach i [info vars] {
>> > if {[array exists $i]} {
>> > rw a /dev/shm/allvars.txt "$i [array size $i]\n"
>> > } else {
>> > rw a /dev/shm/allvars.txt "$i [strlen [set $i]]\n"
>> > }
>> > }
>>
>> What is "rw a"???
>
> rw is a proc that reads or writes to files. "rw a" is write/append.
> That piece of code above was introduced later, to investigate the
> problem that was observed before the code was introduced.

Ok, so a custom version of Tcllib's fileutil's
'::fileutil::appendToFile' proc.

> That piece of code above was introduced later, to investigate the
> problem that was observed before the code was introduced.

How did we know this fact, when you did not explicitly tell us this in
the original post?

>> The 'delete 1.0 end' should make the text widget memory usage not grow
>> forever.
>
> Well, yes, and it keeps the content visible too. Without it, the text
> widget would scroll so far down it could possibly find oil.

And consume an ever growing amount of memory....

>> In your original post you were doing something with graphs. Does
>> the graph library include a "delete/free/destroy" call that you are
>> missing?
>
> My graph library is Tk and canvas. No third-party library whatsoever.

Do you draw a new graph for each iteration?

If yes, do you delete all the old graphs objects first? If not, your
memory growth could also be an accumulation of undeleted canvas
objects.

> Is it reasonable that I have to destroy and recreate the entire GUI
> at every iteration so the application remains barely stable?

No.

> The chart is far from being visually intense or busy. It's quite simple.
> Why is it eating so much memory?

Since you have revealed almost nothing to us, why are you asking us why
it is using so much memory.

> Is there some "best practice" I am overlooking here to avoid that
> kind of pitfall?

A very likely answer is yes, but we can't help if you don't provide us
information with which to help. Pointing at
https://www.sqlite.org/tclsqlite.html does not help us see specifically
how you are doing something, and your memory leak is most likely the
direct result of something you are specifically (but unintentionally)
doing.

See if you can trim down to a minimal version that still 'leaks' --
then stop being so secretive and reveal that minimal version here.

Note also that trying to "trim down" can often reveal the cause,
because as you trim, if you suddenly have no more leak, then the last
thing you trimmed was likely the cause. So trim down in small
increments.

[Rich]

Luc <n...@no.no> wrote:
> On Fri, 23 Dec 2022 19:21:26 -0000 (UTC), Rich wrote:
>
>> > set ::dbfile "$::BASEDIR/somefile.db"
>> > set ::sqllibrary "$::BASEDIR/libtclsqlite3.so"
>> > load $::sqllibrary
>>
>> Why are you doing this above instead of "package require sqlite3"?
>

> So what is wrong with using the .so library? Is it bad? Why? What are
> the undesired side effects?

Also, for modern Tcl (i.e., 8.6+) Sqlilte3 is included as part of the
base Tcl install, so at least for sqlite, a "package require sqlite3"
should always succeed, provided someone has Tcl itself properly
installed.

[Luc]

On Sat, 24 Dec 2022 12:56:42 -0000 (UTC), Rich wrote:

> Do you draw a new graph for each iteration?
> If yes, do you delete all the old graphs objects first? If not, your
> memory growth could also be an accumulation of undeleted canvas
> objects.

Yes and yes. Like I said previously,

"My app collects a bunch of data from a SQLite database then runs a query
and plots a chart.
Then it clears the chart with the $::chart delete "all" canvas command.
Then it runs another query and plots the chart again. Rinse and repeat
over and over. It's a loop."

> Also, for modern Tcl (i.e., 8.6+) Sqlilte3 is included as part of the
> base Tcl install, so at least for sqlite, a "package require sqlite3"
> should always succeed, provided someone has Tcl itself properly
> installed.

I reject this whole idea of a "modern Tcl." Not everybody can afford to
have the latest and greatest. If I were coding for Windows, I would be
trying to support Windows 95. I hate this idea of excluding perfectly
capable old machines or environments.

See, I have 8.6.6 on Debian 9. I'm not sure, but I think sqlite3 did not
work with Tcl on my machine until I installed the libsqlite3-tcl package.
A lot of people have no idea of such concepts. Including me. I've run into
a lot of Python or Javascript software that requires things I have never
heard of.

This week I stumbled upon something that seemed interesting, but the
instructions said I had to use "pip" to install it. What the heck is "pip"?
Why in the world do I ever have to know? I very simply gave up and forgot
about it. Honestly, I couldn't care less about those things. I don't want
to learn about add-ons of those languages. I just want to use the darned
application. I write in Tcl so you may very reasonably tell me that I have
to try and become more familiar with packages, that is reasonable, but
I will never expect users of my software to have any more interest in those
things than a flat zero. My software has to be turn-key. If it isn't, it is
my job to make it so. I am just very lucky that my favorite programming
language can always enable me to deliver that.

I lump that together with the idea of making sure my apps will run on older
versions of Linux and Tcl. I don't want to have the latest and greatest Tcl.
Because people who decide to use my apps may not have the so much latest
and greatest. So ideally, I should be using Tcl/Tk 8.5 to test my apps.
I should write code that is likely to run on both old and new versions.
I want the "Runs on Woody" certification.

I could even go with 8.4, except that I really like the argument expansion
nipple...

> See if you can trim down to a minimal version that still 'leaks' --
> then stop being so secretive and reveal that minimal version here.
> Note also that trying to "trim down" can often reveal the cause,
> because as you trim, if you suddenly have no more leak, then the last
> thing you trimmed was likely the cause. So trim down in small
> increments.

Yes, I will do that. Later on. I'm taking a break now for a couple
of days. Thank you for the help and Merry Christmas!

--
Luc
>>

[Rich]

Luc <n...@no.no> wrote:
> On Sat, 24 Dec 2022 12:56:42 -0000 (UTC), Rich wrote:
>
>> Do you draw a new graph for each iteration?
>> If yes, do you delete all the old graphs objects first? If not, your
>> memory growth could also be an accumulation of undeleted canvas
>> objects.
>
>
> Yes and yes. Like I said previously,
>
> "My app collects a bunch of data from a SQLite database then runs a query
> and plots a chart.
> Then it clears the chart with the $::chart delete "all" canvas command.
> Then it runs another query and plots the chart again. Rinse and repeat
> over and over. It's a loop."

Ok, unless the canvas contains a leak under the hood (i.e., a bug) that
should clean up all memory allocated by canvas objects.

>> Also, for modern Tcl (i.e., 8.6+) Sqlilte3 is included as part of
>> the base Tcl install, so at least for sqlite, a "package require
>> sqlite3" should always succeed, provided someone has Tcl itself
>> properly installed.
>
>
> I reject this whole idea of a "modern Tcl." Not everybody can afford to
> have the latest and greatest. If I were coding for Windows, I would be
> trying to support Windows 95. I hate this idea of excluding perfectly
> capable old machines or environments.

Except, Tcl 8.6.0 was released Dec. 20, 2012
(https://core.tcl-lang.org/tips/doc/trunk/tip/311.md) so 8.6 is now ten
years old as of a few days ago. Yes, 8.6 is "modern Tcl", but it is
also old enough that anyone not intentionally sticking to an old
version of Tcl or an old OS install can easily have been on 8.6 by now.

>> See if you can trim down to a minimal version that still 'leaks' --
>> then stop being so secretive and reveal that minimal version here.
>> Note also that trying to "trim down" can often reveal the cause,
>> because as you trim, if you suddenly have no more leak, then the
>> last thing you trimmed was likely the cause. So trim down in small
>> increments.
>
> Yes, I will do that. Later on. I'm taking a break now for a couple
> of days. Thank you for the help and Merry Christmas!

Merry Christmas to you to.

[Christian Gollwitzer]

Am 25.12.22 um 01:14 schrieb Rich:

> Luc <n...@no.no> wrote:
>> On Sat, 24 Dec 2022 12:56:42 -0000 (UTC), Rich wrote:
>>
>>> Do you draw a new graph for each iteration?
>>> If yes, do you delete all the old graphs objects first? If not, your
>>> memory growth could also be an accumulation of undeleted canvas
>>> objects.
>>
>>
>> Yes and yes. Like I said previously,
>>
>> "My app collects a bunch of data from a SQLite database then runs a query
>> and plots a chart.
>> Then it clears the chart with the $::chart delete "all" canvas command.
>> Then it runs another query and plots the chart again. Rinse and repeat
>> over and over. It's a loop."
>
> Ok, unless the canvas contains a leak under the hood (i.e., a bug) that
> should clean up all memory allocated by canvas objects.
>

There is a known issue, if you do create canvas tags, those are not
freed, So if your plot package applies a tag to each data point, e.g. ,
and doesn't reset the counter for these tags, then you will create a leak.

Christian

[et4]

On 12/24/2022 9:40 AM, Luc wrote:

> I lump that together with the idea of making sure my apps will run on older
> versions of Linux and Tcl. I don't want to have the latest and greatest Tcl.
> Because people who decide to use my apps may not have the so much latest
> and greatest. So ideally, I should be using Tcl/Tk 8.5 to test my apps.
> I should write code that is likely to run on both old and new versions.
> I want the "Runs on Woody" certification.
>
> I could even go with 8.4, except that I really like the argument expansion
> nipple...
>

I agree with you about how it can be a pain to support "other" systems. My solution is starpaks. I like portable apps that are a single file you simply run. You end up providing a single file with whatever version of tcl/tk you want to use, add your own code, and it's all packed into the one file. If you must use 8.5 that's no problem either. You start with an 8.5 tclkit.

Then it doesn't matter whether the user has tcl or not or what version they are running.

In fact, I don't normally use the installed tcl on my windows system, I just use one of Ashok's tclkits and create a starpak that bundles in all my debugging tools. That replaces wish for me.

As to the memory leak, Rich is right, trim it down one thing at a time until the leak goes away. One way to do that is to "simulate" things like your graphics output. If your code is modular, you can easily just *dynamically* replace some procs to do effectively just enough to keep it's callers from barfing.

Remember, the [proc] command is not a declaration, it is a runtime dynamic procedure creator. You can keep your code intact and simply write another (tiny) version of the proc that replaces it entirely, all at runtime. You can even run the original for a while, and redefine it later in the run to see if there's any differences.

For example, maybe the graphics output might be reduced to some puts output (just so you know it's been called). To find the leak, your program doesn't have to produce the full results.

good luck

[Andreas Leitgeb]

Luc <n...@no.no> wrote:
> "My app collects a bunch of data from a SQLite database then runs a query
> and plots a chart.
> Then it clears the chart with the $::chart delete "all" canvas command.
> Then it runs another query and plots the chart again. Rinse and repeat
> over and over. It's a loop."

I'd suggest the "divide et impera" approach:

You replace, part by part, pieces of code by dummies.
Then the part whose replacement by dummy will stop the process
from growing in memory, is likely the culprit:

So,
- replace the sqlite select by a hardcoded result.
- augment some identifiers (in the hardcoded result) with counter-
variables, just so as to give different data each time.
- replace all item creations on the canvas by something that does nothing.

Knowing your own code better will allow you to find more parts to
replace by dummies, until you find the one pack-ratting or leaking
memory.

And beyond that you could also have a look at tools like valgrind.