Memory usage of the imaging library when the imaging library's default is used

[Rich Bell]

This is not new information, but I thought it might help others seeing a steady increase in memory consumption.

TL;DR

I am seeing a memory leak when the imaging library default font is used.  Ensuring that a font is specified for each label (top_label_font_path, unit_label_font_path, bottom_label_font_path, axis_label_font_path, rose_label_font_path) seems to have fixed it.

Backgound:

Running with:

OS: Linux-4.19.49v6v7-aufs-armv7l-with-debian-9.6

Python: 3.5.3 (default, Sep 27 2018, 17:25:39) 

[GCC 6.3.0 20170516]

PIL/Pillow: 4.0.0

I was seeing the following:

The following is a graph of memory size over the same time period.  It graphs the same weewx instance as above - no fonts specified for the images (base), an instance with the skin disabled (disabled), and one with the fonts specified (fontsall).

Here is a plot of the RSS over the same time.

Here are a few more data points that may or may not be of interest.  "noimage" does not call plot.render() while render does, but does not save the image (image.save).  

Here is the memory size plot.

And the RSS.

- rich

[Rich Bell]

Not sure what happened to the plots... Hopefully this works.

-rich

On Sun, Dec 1, 2019 at 3:43 PM Rich Bell <bell...@gmail.com> wrote:

This is not new information, but I thought it might help others seeing a steady increase in memory consumption.

TL;DR

I am seeing a memory leak when the imaging library default font is used.  Ensuring that a font is specified for each label (top_label_font_path, unit_label_font_path, bottom_label_font_path, axis_label_font_path, rose_label_font_path) seems to have fixed it.

Backgound:

Running with:

OS: Linux-4.19.49v6v7-aufs-armv7l-with-debian-9.6

Python: 3.5.3 (default, Sep 27 2018, 17:25:39) 

[GCC 6.3.0 20170516]

PIL/Pillow: 4.0.0

I was seeing the following:

The following is a graph of memory size over the same time period.  It graphs the same weewx instance as above - no fonts specified for the images (base), an instance with the skin disabled (disabled), and one with the fonts specified (fontsall).

Here is a plot of the RSS over the same time.

Here are a few more data points that may or may not be of interest.  "noimage" does not call plot.render() while render does, but does not save the image (image.save).  

Here is the memory size plot.

And the RSS.

- rich

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/fe7997e1-15d9-4f83-b338-44075df42ef1%40googlegroups.com.

[Thomas Keffer]

Good information, although I don't understand it.

The 'weeplot' image generator actually caches fonts, exactly for this reason. Many years ago, we experienced memory leaks, due to PIL not properly recycling font handles. So, in revision 8615d0f3, added over 7 years ago, we introduced the cache. So, either, for some reason, it isn't working for default fonts, or the memory leak lies within some special implementation that the default uses.

Are you using PIL or Pillow?

-tk

--

[Rich Bell]

Yeah, I don't understand it either. I saw the caching of fonts. The data seems to point to something that the Plot render method uses in the library. I did see that the default font is actually embedded in the imaging library code (not loaded from a file). I also used a true type font when I specified the font. I was having trouble recreating using a modified wee_reports, but that was before I narrowed in on the default font. It's got my curiousity, so I am going to try to recreate again. That would allow me to throw different things at it faster. 

From pip3 list, I am using Pillow 4.0.

- rich

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

[Rich Bell]

TL;DR - upgrading Pillow to 6.2.1 seems to have stopped the leak.

I was able to replicate with a modified version of wee_reports. This allowed me to reproduce the problem quicker. I was able to narrow down the problem to calls to draw.textsize and draw.text.  Some additional research lead me to this issue, https://github.com/python-pillow/Pillow/issues/2629 - which clinched the decision to upgrade Pillow.

With Pillow 4.0.0, generating the plot 2500 times resulted in this memory usage.

With Pillow 6.2.1, generating the plot 2500 times resulted in this memory usage.

Note, in both plots ignore the date/times. I just used the loop count as the time, so they start at epoch 0 or 1/1/1970 12 AM UTC...

- rich

[Thomas Keffer]

Well, that's certainly a simple enough solution!

Thanks, Rich, for the great sleuthing you did!

-tk

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/88b8542d-6394-4591-9f68-e220feba35a3%40googlegroups.com.

[vince]

What's interesting is Raspbian current and python3-pil 5.4.1-2 and weewx 4 beta do not show any memory leaks.