Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Bug#1037042: graphicsmagick: GetImageDepth has a thread arena and memory leak
0 views
Skip to first unread message
Bob Friesenhahn
Jun 2, 2023, 3:00:05 PM6/2/23
to
Christophe,
It would be good to test a modern GM version which supports "resource
limited memory". Probably Debian's 1.4 version supports that.
For a version which supports the resource limited memory allocator,
you can set 'MAGICK_DEBUG=resource' in the environment prior to
running the program. This will cause very verbose output with a tally
similar to:
13:27:40 0:0.007952 0.050u 173194 resource.c/unknown/236/Resource:
memory +65.2MiB/81.0MiB/39.0GiB
13:27:40 0:0.027664 0.190u 173194 resource.c/unknown/818/Resource:
memory -3.3KiB/81.0MiB/39.0GiB
13:27:40 0:0.027692 0.190u 173194 resource.c/unknown/818/Resource:
.
.
.
13:27:40 0:0.033629 0.220u 173194 resource.c/unknown/818/Resource:
memory -65.2MiB/0B/39.0GiB
The "0B" at the end means that at least as related to memory allocated
by the resource limited memory allocator (used only for non-public
allocations), there are no leaks.
The type of leak you seem to be finding is a thread-specific memory
allocation leak where a destructor did not fire to free the memory
when the thread quit. I don't believe that GetImageDepth() uses
thread-specific memory but the "thread arena"s that you are talking
about likely do.
For allocations which specifically use a memory allocator, I have had
some good luck using Glibc's mtrace.
Bob
--
Bob Friesenhahn
bfri...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt
It would be good to test a modern GM version which supports "resource
limited memory". Probably Debian's 1.4 version supports that.
For a version which supports the resource limited memory allocator,
you can set 'MAGICK_DEBUG=resource' in the environment prior to
running the program. This will cause very verbose output with a tally
similar to:
13:27:40 0:0.007952 0.050u 173194 resource.c/unknown/236/Resource:
memory +65.2MiB/81.0MiB/39.0GiB
13:27:40 0:0.027664 0.190u 173194 resource.c/unknown/818/Resource:
memory -3.3KiB/81.0MiB/39.0GiB
13:27:40 0:0.027692 0.190u 173194 resource.c/unknown/818/Resource:
.
.
.
13:27:40 0:0.033629 0.220u 173194 resource.c/unknown/818/Resource:
memory -65.2MiB/0B/39.0GiB
The "0B" at the end means that at least as related to memory allocated
by the resource limited memory allocator (used only for non-public
allocations), there are no leaks.
The type of leak you seem to be finding is a thread-specific memory
allocation leak where a destructor did not fire to free the memory
when the thread quit. I don't believe that GetImageDepth() uses
thread-specific memory but the "thread arena"s that you are talking
about likely do.
For allocations which specifically use a memory allocator, I have had
some good luck using Glibc's mtrace.
Bob
--
Bob Friesenhahn
bfri...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt
Bob Friesenhahn
Jun 3, 2023, 10:00:04 AM6/3/23
to
I did manage to get the test program compiled on my PC (a laptop).
It was challenging since the source code (extracted from email) seemed
to have hidden characters in it that the C compiler did not like.
I do see the reported RSS very gradually creeping up. It seems to go
up and then go back down a little bit, but the overall average trends
up.
Looking at /proc/PID/smaps, the size attributed to 'heap' seems
stable.
It was challenging since the source code (extracted from email) seemed
to have hidden characters in it that the C compiler did not like.
I do see the reported RSS very gradually creeping up. It seems to go
up and then go back down a little bit, but the overall average trends
up.
Looking at /proc/PID/smaps, the size attributed to 'heap' seems
stable.
Bob Friesenhahn
Jun 3, 2023, 2:40:05 PM6/3/23
to
I am definitely able to confirm that memory consumption builds due to
invoking GetImageDepth() via a POSIX thread. The rate that it builds
is image sensitive since some images cause GetImageDepth() to perform
more OpenMP loops.
In /proc/PID/smaps I see multiple memory-mapped regions (including
"heap") building at once.
Running under valgrind reports no leaks.
I changed the sole memory allocation used in GetImageDepth() to use
the resource-limited memory allocator and it did not report any leaks.
My own testing is under Ubuntu 20.04 using GCC 10.
invoking GetImageDepth() via a POSIX thread. The rate that it builds
is image sensitive since some images cause GetImageDepth() to perform
more OpenMP loops.
In /proc/PID/smaps I see multiple memory-mapped regions (including
"heap") building at once.
Running under valgrind reports no leaks.
I changed the sole memory allocation used in GetImageDepth() to use
the resource-limited memory allocator and it did not report any leaks.
My own testing is under Ubuntu 20.04 using GCC 10.
László Böszörményi
Jun 4, 2023, 6:30:05 AM6/4/23
to
Hi,
On Sat, Jun 3, 2023 at 8:30 PM Bob Friesenhahn
<bfri...@simple.dallas.tx.us> wrote:
> I am definitely able to confirm that memory consumption builds due to
> invoking GetImageDepth() via a POSIX thread. The rate that it builds
> is image sensitive since some images cause GetImageDepth() to perform
> more OpenMP loops.
Unfortunately I can not reproduce. My processor is an Intel K variant
CPU, six cores and twelve threads, 64 Gb of RAM.
GM is 1.3.40 with two security fixes backported, compiled with GCC
v12.2.0. Tried with three PNG images, all memory consumption is static
from the beginning. Do I need some special case of PNG files to
experience this issue?
> My own testing is under Ubuntu 20.04 using GCC 10.
Do you think it might be a problem with another system component, a
GCC optimization or this is fixed meanwhile? At least I do wonder why
this issue is CPU / machine dependent.
Regards,
Laszlo/GCS
On Sat, Jun 3, 2023 at 8:30 PM Bob Friesenhahn
<bfri...@simple.dallas.tx.us> wrote:
> I am definitely able to confirm that memory consumption builds due to
> invoking GetImageDepth() via a POSIX thread. The rate that it builds
> is image sensitive since some images cause GetImageDepth() to perform
> more OpenMP loops.
CPU, six cores and twelve threads, 64 Gb of RAM.
GM is 1.3.40 with two security fixes backported, compiled with GCC
v12.2.0. Tried with three PNG images, all memory consumption is static
from the beginning. Do I need some special case of PNG files to
experience this issue?
> My own testing is under Ubuntu 20.04 using GCC 10.
GCC optimization or this is fixed meanwhile? At least I do wonder why
this issue is CPU / machine dependent.
Regards,
Laszlo/GCS
Bob Friesenhahn
Jun 4, 2023, 9:11:17 AM6/4/23
to
On Sun, 4 Jun 2023, László Böszörményi wrote:
> Hi,
>
> On Sat, Jun 3, 2023 at 8:30 PM Bob Friesenhahn
> <bfri...@simple.dallas.tx.us> wrote:
>> I am definitely able to confirm that memory consumption builds due to
>> invoking GetImageDepth() via a POSIX thread. The rate that it builds
>> is image sensitive since some images cause GetImageDepth() to perform
>> more OpenMP loops.
> Unfortunately I can not reproduce. My processor is an Intel K variant
> CPU, six cores and twelve threads, 64 Gb of RAM.
> GM is 1.3.40 with two security fixes backported, compiled with GCC
> v12.2.0. Tried with three PNG images, all memory consumption is static
> from the beginning. Do I need some special case of PNG files to
> experience this issue?
Using PNG is not important. The nature of the input image is
> Hi,
>
> On Sat, Jun 3, 2023 at 8:30 PM Bob Friesenhahn
> <bfri...@simple.dallas.tx.us> wrote:
>> I am definitely able to confirm that memory consumption builds due to
>> invoking GetImageDepth() via a POSIX thread. The rate that it builds
>> is image sensitive since some images cause GetImageDepth() to perform
>> more OpenMP loops.
> Unfortunately I can not reproduce. My processor is an Intel K variant
> CPU, six cores and twelve threads, 64 Gb of RAM.
> GM is 1.3.40 with two security fixes backported, compiled with GCC
> v12.2.0. Tried with three PNG images, all memory consumption is static
> from the beginning. Do I need some special case of PNG files to
> experience this issue?
important. Prepare a test image like
gm convert infile.png -depth 4 testfile.pnm
and then use testfile.pnm as input.
The PNM reader is now also also threaded (in very recent versions).
PNG should work as well to reproduce the bug.
>> My own testing is under Ubuntu 20.04 using GCC 10.
> Do you think it might be a problem with another system component, a
> GCC optimization or this is fixed meanwhile? At least I do wonder why
> this issue is CPU / machine dependent.
is being added to (presumably) many thread stacks, I see the number of
'anon' mappings increasing, and the "heap" similarly grows and, but is
released before the program exits.
It is possible that doing some OpenMP thing prior to using POSIX
threads (so OpenMP teams are already created) might change the
outcome. I have not tried that yet.
I have attached a version of threadarena.c which loops 1024 times and
then quits.
Bob Friesenhahn
Jun 4, 2023, 10:30:04 AM6/4/23
to
>
>> My own testing is under Ubuntu 20.04 using GCC 10.
> Do you think it might be a problem with another system component, a
> GCC optimization or this is fixed meanwhile? At least I do wonder why
> this issue is CPU / machine dependent.
As a further data point, I compiled the test program under Illumos
>> My own testing is under Ubuntu 20.04 using GCC 10.
> Do you think it might be a problem with another system component, a
> GCC optimization or this is fixed meanwhile? At least I do wonder why
> this issue is CPU / machine dependent.
(Solaris derivative) and it was able to run tens of thousands of loops
without any memory leak or other anomaly.
In this case GCC 7.5.0 was used.
Bob Friesenhahn
Jun 5, 2023, 11:40:04 AM6/5/23
to
On Mon, 5 Jun 2023, Beauregard,Christophe (ECCC) wrote:
> FWIW, I think I may have figured out what's happening. It's probably
> something to note in the http://www.graphicsmagick.org/OpenMP.html
> documentation.
>
> In a nutshell, my understanding is that libgomp keeps track of its
> thread pool in thread local storage. When that thread local storage
> happens to be in the main process, all is well, everything works
> fine, etc. When that thread local storage is in a transient pthread
> (and, apparently, you're running on a certain class memory/CPU
> configuration) and your transient thread terminates, that thread
> local storage doesn't (always) get cleaned up.
This is very unfortunate and seems like a serious bug.
> I suspect this isn't a particularly desirable situation, as
> GraphicsMagick appears to have put at least some effort into making
> the API thread-safe.
Considerable effort has been put into making the API as thread-safe as
possible, but invoking OpenMP code from a POSIX thread (or other sort
of thread) has not been significantly tested.
It would be good to write a bug report against gomp (it seems to still
be a GCC product) so that there is hope to get this fixed, or be
informed of a work-around.
> FWIW, I think I may have figured out what's happening. It's probably
> something to note in the http://www.graphicsmagick.org/OpenMP.html
> documentation.
>
> In a nutshell, my understanding is that libgomp keeps track of its
> thread pool in thread local storage. When that thread local storage
> happens to be in the main process, all is well, everything works
> fine, etc. When that thread local storage is in a transient pthread
> (and, apparently, you're running on a certain class memory/CPU
> configuration) and your transient thread terminates, that thread
> local storage doesn't (always) get cleaned up.
This is very unfortunate and seems like a serious bug.
> I suspect this isn't a particularly desirable situation, as
> GraphicsMagick appears to have put at least some effort into making
> the API thread-safe.
Considerable effort has been put into making the API as thread-safe as
possible, but invoking OpenMP code from a POSIX thread (or other sort
of thread) has not been significantly tested.
It would be good to write a bug report against gomp (it seems to still
be a GCC product) so that there is hope to get this fixed, or be
informed of a work-around.
0 new messages