Steady memory leak in weewx

[Chris Adams]

I am running weewx 4.9.1 on Fedora 37 aarch64 on a Raspberry Pi 4. I have a Davis VantageVue weather system, connected via USB, and am uploading via FTP with SSL to a remote server. I don't have any plugins or anything else special installed/configured.

The weewx python3 process grows at a pretty steady rate, a little over 150MB resident per minute (I have an update interval of 1 minute, so basically every run it grows). It's been showing a memory leak for a bit, previously with weewx 4.8 on Fedora 35 (I just waited until getting everything current before looking into it, hoping it'd resolve itself that way :) ). Right now, I'm running it under a systemd unit that restarts it every 6 hours (previously was capping the RSS memory to 192M, but that crashed the process with the OOM killer, so switched to scheduled restarts, which are "nicer").

There's nothing unusual in the logs, just the expected output like:

Nov 27 15:07:16 python3[7451]: weewx[7451] INFO weewx.manager: Added record 2022-11-27 15:07:00 CST (1669583220) to database 'weewx.sdb'
Nov 27 15:07:16 python3[7451]: weewx[7451] INFO weewx.manager: Added record 2022-11-27 15:07:00 CST (1669583220) to daily summary in 'weewx.sdb'
Nov 27 15:07:19 python3[7451]: weewx[7451] INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 2.34 seconds
Nov 27 15:07:20 python3[7451]: weewx[7451] INFO weewx.imagegenerator: Generated 17 images for report SeasonsReport in 1.31 seconds
Nov 27 15:07:20 python3[7451]: weewx[7451] INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx
Nov 27 15:07:25 python3[7451]: weewx[7451] INFO weewx.reportengine: ftpgenerator: Ftp'd 23 files in 4.71 seconds

[Tom Keffer]

Every once in a while, a user will report the sudden appearance of a memory leak. Then, just as mysteriously, the leak will go away.

I can only assume that the cause is an underlying C library that got "upgraded," then upgraded again.

Do you recall the update history of your Fedora instance? Any recent upgrades? Any involving Pillow? Database adapters?

Python 3 offers the tracemalloc tool, which I've found to be pretty good at finding the offending library. 

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/a84edd0b-0f3f-4ea3-a246-5c15044d0c26n%40googlegroups.com.

[vince]

On Sunday, November 27, 2022 at 1:10:19 PM UTC-8 cma...@gmail.com wrote:

The weewx python3 process grows at a pretty steady rate, a little over 150MB resident per minute 

Couple questions - it looks like you used the rpm version of weewx, is that correct ?

And did you install the python prerequisites with pip3 or via rpms ?

I have a spare pi4 that I can install the same os+weewx on in Simulator mode on if you want somebody to try to recreate your issue....

[Chris Adams]

Well, it's been persistent for me. I noticed it months ago, when still running weewx 4.8 on Fedora 35 (still aarch64, aka 64-bit, version). I'm not sure how long it had been going on, I noticed that the Pi had OOMed it for using up all the RAM, before I limited it with systemd. But since I've upgraded to Fedora 37 (two major releases), that's basically everything getting rebuilt.

My attempts at finding python memory leaks in the past haven't been fruitful, but IIRC that was python2 (which I think didn't have tracemalloc). I'll see if I can poke at it.

[Chris Adams]

Yes, I'm using the RPM version from the weewx EL8 repo. Really the only significant change from that is to mask the systemd unit auto-created from the SysV-init script and using my own systemd unit (which then allows me to better control it). Everything is installed from RPM. Here's the log excerpt from service startup through the first report if that helps:

Nov 27 13:46:03 python3[7451]: weewx[7451] INFO __main__: Initializing weewx version 4.9.1
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO __main__: Using Python 3.11.0 (main, Oct 24 2022, 00:00:00) [GCC 12.2.1 20220819 (Red Hat 12.2.1-2)]
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO __main__: Located at /usr/bin/python3
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO __main__: Platform Linux-6.0.9-300.fc37.aarch64-aarch64-with-glibc2.36
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO __main__: Locale is 'en_US.UTF-8'
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO __main__: Using configuration file /etc/weewx/weewx.conf
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO __main__: Debug is 0
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO weewx.engine: Loading station type Vantage (weewx.drivers.vantage)
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO weewx.engine: StdConvert target unit is 0x1
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO weewx.wxservices: StdWXCalculate will use data binding wx_binding
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO weewx.engine: Archive will use data binding wx_binding
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO weewx.engine: Record generation will be attempted in 'hardware'
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO weewx.engine: Using archive interval of 60 seconds (specified by hardware)
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO weewx.restx: StationRegistry: Registration not requested.
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO weewx.restx: Wunderground: Posting not enabled.
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO weewx.restx: PWSweather: Posting not enabled.
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO weewx.restx: CWOP: Posting not enabled.
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO weewx.restx: WOW: Posting not enabled.
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO weewx.restx: AWEKAS: Posting not enabled.
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO weewx.engine: 'pyephem' detected, extended almanac data is available
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO __main__: Starting up weewx version 4.9.1
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO weewx.engine: Clock error is 0.01 seconds (positive is fast)
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO weewx.engine: Using binding 'wx_binding' to database 'weewx.sdb'
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO weewx.manager: Starting backfill of daily summaries
Nov 27 13:46:03 python3[7451]: weewx[7451] INFO weewx.manager: Daily summaries up to date
Nov 27 13:46:04 python3[7451]: weewx[7451] INFO weewx.manager: Added record 2022-11-27 13:46:00 CST (1669578360) to database 'weewx.sdb'
Nov 27 13:46:04 python3[7451]: weewx[7451] INFO weewx.manager: Added record 2022-11-27 13:46:00 CST (1669578360) to daily summary in 'weewx.sdb'
Nov 27 13:46:04 python3[7451]: weewx[7451] INFO weewx.engine: Starting main packet loop.
Nov 27 13:47:16 python3[7451]: weewx[7451] INFO weewx.manager: Added record 2022-11-27 13:47:00 CST (1669578420) to database 'weewx.sdb'
Nov 27 13:47:16 python3[7451]: weewx[7451] INFO weewx.manager: Added record 2022-11-27 13:47:00 CST (1669578420) to daily summary in 'weewx.sdb'
Nov 27 13:47:19 python3[7451]: weewx[7451] INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 3.21 seconds
Nov 27 13:47:21 python3[7451]: weewx[7451] INFO weewx.imagegenerator: Generated 17 images for report SeasonsReport in 1.36 seconds
Nov 27 13:47:21 python3[7451]: weewx[7451] INFO weewx.reportengine: Copied 5 files to /var/www/html/weewx
Nov 27 13:47:26 python3[7451]: weewx[7451] INFO weewx.reportengine: ftpgenerator: Ftp'd 23 files in 4.86 seconds

[vince]

On Sunday, November 27, 2022 at 5:01:07 PM UTC-8 cma...@gmail.com wrote:

Yes, I'm using the RPM version from the weewx EL8 repo. Really the only significant change from that is to mask the systemd unit auto-created from the SysV-init script and using my own systemd unit (which then allows me to better control it). Everything is installed from RPM. Here's the log excerpt from service startup through the first report if that helps:

In admittedly just a few hours of testing, a setup.py version using pip3 installations of the python modules and weewx in Simulator mode is 'not' growing at all in a quick test under Fedora-37 in a vagrant x86_64 VM so it's not quite an apples and apples comparison. 

I'll see if I can get a pi4 online with the same os you're running.  Looks like it might be a bit of a pain in the neck to get a headless installation set up with ssh running and on the network....but I'll see if I can take a crack of it tomorrow and see if a Simulator setup on a pi4 is leaking too...

[Chris Adams]

I fiddled around with tracemalloc, but I didn't get anything. I know programming, but my python is rudimentary; I think I'd need someone to send me what to do or a patch.

[vince]

Your lowest blood pressure option is likely to just run RaspiOS on it, which doesn't leak almost at all.

I just got f37 on a spare pi4 so I'll see if I can duplicate your problem (using a Simulator driver).

Geez what an awful experience trying to bring up a fedora pi in comparison to RaspiOS.

But wifi works so that's good once I found the right nmcli incantation.  Ugh.

Stay tuned - let me spin up weewx on it and let it run for a bit.

[vince]

ok - I'm not seeing a memory leak on f37 on a pi4 in a couple hours of uptime.  I installed 4.9.1 in setup.py mode using the Simulator driver and am wifi only for networking.  I might add that I followed the instructions for using a python venv that Tom posted in the wiki so my python stuff is installed into the venv via pip.  Once I set setenforce=0 to disable selinux (ugh) it starts fine via systemd.

(wild thought - I wonder if the rpms you installed are where the leak is.  Note that I used 'pip' to compile+install all the modules)

If you want to watch your memory growth you can add my memory extension (https://github.com/vinceskahan/vds-weewx-v3-mem-extension) which logs to a separate db and sets up a quickie webpage to view as well.

For a quick cli check, I'm running this script to output some data...edit the path to your index.html to match of course...

     while true; do echo ''; free -m ; ps axu|grep weewx; grep minutes /home/weewx/public_html/index.html | grep -v grep; echo ' '; sleep 360; done

Hit ^C to stop it.  Example output looks like:

               total        used        free      shared  buff/cache   available
Mem:            3808         775         901          11        2131        2833
Swap:           3807           0        3807
fedora     12153  1.0  1.2 351688 48124 ?        Ss   12:12   1:15 /home/weewx/weewx_venv/bin/python3 /home/weewx/bin/weewxd /home/weewx/weewx.conf
      <td class="data">0 days, 2 hours, 25 minutes</td>
      <td class="data">0 days, 1 hour, 52 minutes</td>

So my VSS is 351688 and RSS is 48124 for my weewx process.  It's rock solid here in the Simulator FWIW once it reached that size after a few minutes.

[vince]

One last update - installed weewx and all the packages from rpms and I see 'no' memory growth at all for the Simulator over f37 on a pi4.  Same VSS and RSS that weewx used when running pip-installed software in a python venv.  Unfortunately I can't duplicate your issue...

[Chris Adams]

I worked with tracemalloc some more... finally decided that it wasn't that I wasn't using tracemalloc right, it's that the leak isn't coming from python allocations. So I gave valgrind a try, and it flagged a leak that has a stack through SSL_SESSION_new. I'm pushing my weewx results through FTP and have TLS enabled... I temporarily disabled the requirement for TLS on the FTP server (pure-ftp), and disabled the weewx config to enable TLS, and it doesn't seem to be leaking memory now.

So... I guess the problem is in python's use of OpenSSL (at least from ftplib)... I figure if there was a leak in OpenSSL itself, across Fedora 35->Fedora 37, it would be more noticed.