ERROR weewx.cheetahgenerator: **** MemoryError

515 views
Skip to first unread message

garrya...@gmail.com

unread,
Dec 31, 2020, 11:10:09 AM12/31/20
to weewx-user


Weewx was started at about 10:30 PM on 2020-12-30.  Started logging MemoryError at 06:53:39 AM 2020-12-31.  Here's one instance:

2020-12-31T07:30:19-08:00 LockyerHomeServer /weewxd: weatherflowudp: MainThread: Listening for UDP broadcasts to IP address <broadcast> on port 50222, with timeout 90 and share_socket False...
2020-12-31T07:31:19-08:00 LockyerHomeServer /weewxd: weatherflowudp: MainThread: Listening for UDP broadcasts to IP address <broadcast> on port 50222, with timeout 90 and share_socket False...
2020-12-31T07:32:29-08:00 LockyerHomeServer /weewxd: weatherflowudp: MainThread: Listening for UDP broadcasts to IP address <broadcast> on port 50222, with timeout 90 and share_socket False...
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: Generate failed with exception '<class 'MemoryError'>'
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: **** Ignoring template /home/weewx/skins/Belchertown/index.html.tmpl
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: **** Reason: 
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****  Traceback (most recent call last):
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****    File "/home/weewx/bin/weewx/cheetahgenerator.py", line 323, in generate
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****      unicode_string = compiled_template.respond()
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****    File "_home_weewx_skins_Belchertown_index_html_tmpl.py", line 1273, in respond
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 1685, in _handleCheetahInclude
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****      file=file)
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 775, in compile
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****      compiler.compile()
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Compiler.py", line 1800, in compile
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****      self._swallowClassCompiler(self._popActiveClassCompiler())
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Compiler.py", line 1825, in _swallowClassCompiler
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****      classCompiler.cleanupState()
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Compiler.py", line 1305, in cleanupState
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****      self._swallowMethodCompiler(methCompiler)
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Compiler.py", line 1404, in _swallowMethodCompiler
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****      methodCompiler.cleanupState()
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Compiler.py", line 1089, in cleanupState
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****      self.commitStrConst()
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Compiler.py", line 494, in commitStrConst
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****      body = escapedNewlineRE.sub('\\1\n', reprstr[i+1:-1])
2020-12-31T07:32:42-08:00 LockyerHomeServer weewx[2983] ERROR weewx.cheetahgenerator: ****  MemoryError
2020-12-31T07:33:19-08:00 LockyerHomeServer /weewxd: weatherflowudp: MainThread: Listening for UDP broadcasts to IP address <broadcast> on port 50222, with timeout 90 and share_socket False...
2020-12-31T07:34:19-08:00 LockyerHomeServer /weewxd: weatherflowudp: MainThread: Listening for UDP broadcasts to IP address <broadcast> on port 50222, with timeout 90 and share_socket False...

Here's the output of free at about 07:35 AM:

pi@LockyerHomeServer:/home/weewx $ free
              total        used        free      shared  buff/cache   available
Mem:        8012324     4059728     2206528      684728     1746068     3011060
Swap:        102396           0      102396

As you can see, this system has 8GB of memory and usage is at about 50%.  It's usually in the single digits, definitely < 20%!

I've attached an edited syslog from 10:30 PM to this AM.  I only removed repetitive WeatherFlowUDP messages.

I'm going to first restart weewx to see it that clears the problem.

I'm running my BelchertownWxFeeds extension with pretty much every endpoint selected.  If a weewx restart doesn't help, I'll cut back / eliminate my extension.

Last thing I did last night was add the cmon extension.  It appears to be working.  But I will eliminate it if problem persists.

Finally, I will reboot.

Any ideas or suggestions?

Regards,

Garry
LHS-20201231-MemoryError.txt

garrya...@gmail.com

unread,
Dec 31, 2020, 11:16:49 AM12/31/20
to weewx-user
I just restarted weewx and memory usage dropped to about 15%!

I failed to note it in original post but CPU utilization was probably at about 50% consistently before weewx restart and dropped to <30% after restart.

Regards,

Garry

vince

unread,
Dec 31, 2020, 1:55:49 PM12/31/20
to weewx-user
Cheetah is notoriously hard to debug, and Belchertown is a very complicated skin under the hood.   Are you sure you didn't edit the Belchertown skin ?

If not, I'm going to guess here that the driver emitted something that the Belchertown skin didn't expect, or that it did 'not' emit something Belchertown expected.   A weewx restart should have cleared that out unless you had a sensor failure or craziness on your WeatherFlow gear, which (unfortunately) happens a lot.

First thing to try if it persists is to disable Belchertown and see if that helps.  If so, that helps pinpoint where to dig next.

To check your WF gear you could always run my standalone python UDP listener (link) to try to debug what the WF gear is sending.   If you run my listener you need to make sure the weewx WF driver is set with shared_sockets=true 'or' stop weewx while running my listener.   My listener has some python prerequisites to add to your system, so read the README carefully and do what it says.

Tom Keffer

unread,
Dec 31, 2020, 2:17:49 PM12/31/20
to weewx-user
I don't have anything to add to this conversation, except that it's curious that after 10+ years of never seeing a Python MemoryError, this is the second in two days.

Perhaps an update to an underlying library is leaking memory like crazy?

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/807eae4f-404c-48bc-9e27-43de1ac73c07n%40googlegroups.com.

Garry A Lockyer

unread,
Dec 31, 2020, 2:39:39 PM12/31/20
to weewx...@googlegroups.com
Thanks for the speedy responses!

A WeeWx restart seems to have fixed the problem for the moment.

Re: editing the Belchertown skin, nope haven’t touched it, other than interfacing with it via the include files (as generated by my BelchertownWxFeeds extension).  When all the endpoints (for testing) are enabled index.html is about 1.8MB, so perhaps that’s causing the problem.  I can easily reduce / eliminate endpoints and prefer to do that before eliminating the Belchertown skin.

Regards,

Garry Lockyer
E: Ga...@Lockyer.ca


On Dec 31, 2020, at 11:17, Tom Keffer <tke...@gmail.com> wrote:



vince

unread,
Dec 31, 2020, 2:44:53 PM12/31/20
to weewx-user
On Thursday, December 31, 2020 at 11:39:39 AM UTC-8 garrya...@gmail.com wrote:
Re: editing the Belchertown skin, nope haven’t touched it, other than interfacing with it via the include files (as generated by my BelchertownWxFeeds extension).  When all the endpoints (for testing) are enabled index.html is about 1.8MB, so perhaps that’s causing the problem.  I can easily reduce / eliminate endpoints and prefer to do that before eliminating the Belchertown skin.


There it is.  You touched it :-)

Usual debugging rules apply.   Reset it to a baseline unmodified config.  Add in changes one-by-one.  If it goes sideways, revert to the last known good and reverify that it stays good.
 

Garry A Lockyer

unread,
Dec 31, 2020, 10:05:15 PM12/31/20
to weewx...@googlegroups.com
Got MemoryError after about 9 hours after restart.  Have removed cmon by commenting out any mention of cmon in weewx.conf and restarted.

Regards,

Garry Lockyer
Former DEC Product Support Engineer :^)
Kepner-Tregoe Trained :^))
E: Ga...@Lockyer.ca


On Dec 31, 2020, at 11:44, vince <vince...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

garrya...@gmail.com

unread,
Jan 7, 2021, 1:00:40 PM1/7/21
to weewx-user
After many hours of troubleshooting and testing, I think I have an idea what's happening.

Background: I want to use weewx with the Belchertown skin and my extension that reads numerous RSS/Atom feeds  on a Pi Zero, with either a Davis Pro V2 or Weatherflow weather station.  I will eventually have four weather stations in my area.  I thought I had things working reasonably well so deployed one station.  It stopped working after about 3 days.  I can't access that system remotely so I built a very similar system (the next one to deploy) and it falso ailed due to lack of memory.  Weewx memory usage on my development system also increases over time - it can grow to over 1 GB in a few hours!

My extension was an extension to Belchetown based on the METAR extension.  The extension created Belchertown index hook include files each archive cycle.  While researching this problem I re-read the weewx customization guide and noted that extensions should NOT be dependent on other extensions so I decided to re-write my extension as a service (which looks like a much cleaner solution) without any dependencies on Belchertown (other than include file names).  All I've done so far is create a service (WxFeedsMemoryTest.py) to test weewx/Belchertown memory consumption.

The Problem: weewx/Bechertown memory usage increases over time.  It starts out at about 45 MB and grows at about 3MB per archive period/cycle (using my test case).  A 512 MB Pi will exhaust memory within a few days.

It appears that the problem is associated with the creation of the Belchertown include files while weewx/Belchertown is running:

- if the include file is 'static' as in not (re)created while weewx/Belchertown is running, memory usage is static - it does not grow beyond about 50 MB.

- if the include file is 'dynamic' as in (re)created while weewx/Belchetown is running, memory usage increases.
  - if the include file is created once, and becomes 'static', memory usage increases and then stabilizes.
  - if the include file is recreated continuously (such as on each archive cycle), memory usage increases each cycle.

It does not appear to matter if the include file is created directly, or created as a temporary file and then copied or renamed.

The attached service (WxFeedsMemoryTest.py) can be used to demonstrate the problem.  Please see installation and use instructions within the WxFeedsMemoryTest.py.

I'm going to continue to work on moving my extension from "an extension to an extension" to a service in the hope that this memory problem can be resolved.

With apologies in advance if I'm doing something to cause the problem, please review, advise and let me know what I can do to avoid the problem.

Regards,

Garry
WxFeedsMemoryTest.py

vince

unread,
Jan 7, 2021, 2:07:19 PM1/7/21
to weewx-user
Are you closing the file every time after you write it ?

Garry A Lockyer

unread,
Jan 7, 2021, 3:07:12 PM1/7/21
to weewx...@googlegroups.com
Yup!

Regards,

Garry Lockyer
E: Ga...@Lockyer.ca


On Jan 7, 2021, at 11:07, vince <vince...@gmail.com> wrote:

Are you closing the file every time after you write it ?

bell...@gmail.com

unread,
Jan 9, 2021, 3:39:29 PM1/9/21
to weewx-user

Garry,
I took your WxFeedsMemory.py and made some modifications.
First, I changed the logging to simple print calls. this was just a convenience for me to easily see what was going on.
Second, I stole some code from Vince’s mem extension, https://github.com/vinceskahan/vds-weewx-v3-mem-extension. I did this because I am used to the data it returns. (Note, I highly recommend this when debugging memory.)
Third, I added code so I could call your extension repeatedly in a loop.

When I called new_archive_record 100 times there was a slight increase in memory, but nothing alarming. But it was slow, approximately 2 minutes a call.

So, I commented out the flush and fsync calls and called new_archive_record 10,000 times. The slight increase seemed to stop around 6,000 calls.

I’m wondering if the amount of time it takes to run is causing problems when run as a service. I plan to uncomment the code and run it as an extension. I’ll let you know what I see.
rich

Garry A Lockyer

unread,
Jan 9, 2021, 4:52:08 PM1/9/21
to weewx...@googlegroups.com
Many, many thanks!

Before I wrote the service to investigate the problem, I wrote a standalone program to eliminate all WeeWX components.  It did NOT exhibit the problem.

Problem only seems to show up under WeeWX/Belchertown *and* if the include file changes / is recreated.

Anyway, I really appreciate you looking into my problem.  I’m actively working on a skin-independent service to read many RSS/Atom feeds and will utilize your suggestions above.

Regards,

Garry Lockyer
E: Ga...@Lockyer.ca


On Jan 9, 2021, at 12:39, bell...@gmail.com <bell...@gmail.com> wrote:



bell...@gmail.com

unread,
Jan 9, 2021, 7:41:51 PM1/9/21
to weewx-user
Garry,
I ran your test extension for a bit in my Ubuntu VM.  Doesn't appear to be leaking.
 16:42:24 {'mem_size': 338.53515625, 'mem_rss': 32.953125, 'mem_share': 11.28125}
 16:47:38 {'mem_size': 423.80859375, 'mem_rss': 47.75, 'mem_share': 11.796875}
 16:52:53 {'mem_size': 424.4140625, 'mem_rss': 48.2265625, 'mem_share': 11.796875}
 16:57:24 {'mem_size': 426.46875, 'mem_rss': 50.19921875, 'mem_share': 11.796875}
 17:02:32 {'mem_size': 426.46875, 'mem_rss': 50.21875, 'mem_share': 11.796875}
 17:07:20 {'mem_size': 427.1796875, 'mem_rss': 50.9921875, 'mem_share': 11.796875}
 17:12:27 {'mem_size': 427.1796875, 'mem_rss': 50.9921875, 'mem_share': 11.796875}
 17:17:26 {'mem_size': 427.1796875, 'mem_rss': 50.9921875, 'mem_share': 11.796875}
 17:22:24 {'mem_size': 427.1796875, 'mem_rss': 50.9921875, 'mem_share': 11.796875}
 17:27:19 {'mem_size': 424.6640625, 'mem_rss': 48.71875, 'mem_share': 11.796875}
 17:32:27 {'mem_size': 424.6640625, 'mem_rss': 48.71875, 'mem_share': 11.796875}
 17:37:17 {'mem_size': 424.5859375, 'mem_rss': 48.7109375, 'mem_share': 11.796875}
 17:42:21 {'mem_size': 424.58203125, 'mem_rss': 48.70703125, 'mem_share': 11.796875}
 17:47:24 {'mem_size': 424.984375, 'mem_rss': 49.140625, 'mem_share': 11.796875}
 17:52:21 {'mem_size': 424.984375, 'mem_rss': 49.140625, 'mem_share': 11.796875}
 17:57:15 {'mem_size': 425.05078125, 'mem_rss': 49.26953125, 'mem_share': 11.796875}
 18:02:11 {'mem_size': 425.05078125, 'mem_rss': 49.26953125, 'mem_share': 11.796875}
 18:07:13 {'mem_size': 425.1171875, 'mem_rss': 49.3359375, 'mem_share': 11.796875}
 18:12:14 {'mem_size': 425.1171875, 'mem_rss': 49.3359375, 'mem_share': 11.796875}
 18:17:12 {'mem_size': 425.21875, 'mem_rss': 49.4375, 'mem_share': 11.796875}
 18:22:10 {'mem_size': 425.09765625, 'mem_rss': 49.31640625, 'mem_share': 11.796875}
 18:27:13 {'mem_size': 425.0390625, 'mem_rss': 49.2578125, 'mem_share': 11.796875}
 18:32:15 {'mem_size': 425.0390625, 'mem_rss': 49.2578125, 'mem_share': 11.796875}
 18:37:15 {'mem_size': 425.03125, 'mem_rss': 49.25, 'mem_share': 11.796875}
 18:42:24 {'mem_size': 425.01953125, 'mem_rss': 49.2578125, 'mem_share': 11.796875}
 18:47:14 {'mem_size': 425.0078125, 'mem_rss': 49.24609375, 'mem_share': 11.796875}
 18:52:13 {'mem_size': 425.0078125, 'mem_rss': 49.24609375, 'mem_share': 11.796875}
 18:57:15 {'mem_size': 425.61328125, 'mem_rss': 49.8515625, 'mem_share': 11.796875}
 19:02:19 {'mem_size': 425.0078125, 'mem_rss': 49.2734375, 'mem_share': 11.796875}

When I have some time, I'll install on my test pi along with the Belchertown skin and Vince's mem extension and see if I see anything.
Yes, I enjoy a good memory 'puzzle'.
rich

vince

unread,
Jan 9, 2021, 10:11:39 PM1/9/21
to weewx-user
If your memory usage is stable, is there a problem ?
It's pretty typical after a weewx startup for the usage to go up a bit then level off nicely.

Garry A Lockyer

unread,
Jan 9, 2021, 11:02:51 PM1/9/21
to weewx...@googlegroups.com
My experience is that when I recreate an include file for Belchertown on each archive cycle, as an extension to Belchertown or as a service, memory usage increases with each cycle and eventually errors start.  I’ve seen the error I reported in this string and unresponsive systems.  So, yes, I think there’s a problem.

The memory test service I provided starts out at about 45 MB and increases at about 3 MB per cycle.  I’ve seen WeeWx memory usage grow to over 1 GB!

I’m working on a new version of my extension, as a service, and will report back on its memory usage, hopefully within only a couple of days.

Thanks for all your help!


Regards,

Garry Lockyer
E: Ga...@Lockyer.ca


On Jan 9, 2021, at 19:11, vince <vince...@gmail.com> wrote:

If your memory usage is stable, is there a problem ?

garrya...@gmail.com

unread,
Jan 16, 2021, 11:42:09 AM1/16/21
to weewx-user
Brief Update: January 16, 2021:

I've pretty much finished moving my extension to a service.

I started WeeWX at about midnight and at about 8:20 AM, memory usage has grown to over 941MB!  Allocated blocks was at 4,245,722.

So growing memory usage problem exists.

Bill indicated above he ran my test code on Ubuntu so I set up a Ubuntu Raspberry Pi Desktop system:

RPi 4 - 6 GB
Ubuntu Raspberry Pi OS 20.10
Python3 - 3.8.6
WeeWX 4.3
Belchertown 1.2

Later today I intend to add the mem extension.  Will report back soonest!  I will also run things on Raspberrry Pi OS (but I have to build a new one!).

Regards,

Garry
PS: I *think* I'm going to continue all development on Ubuntu as it seems more current - it has a later linux kernel and more current Python - but I'm not sure it's otherwise better than Raspberry Pi OS.  Both OS's have wireless mouse lag problems, fixed somewhat by changing 'mousepoll' setting.  Installation of Ubuntu was easy and I managed to get it to boot from a USB SSD.  In the first 8 hours or so, Ubuntu froze a couple of times, so now I keep an ssh session open so that I can reboot.

garrya...@gmail.com

unread,
Jan 16, 2021, 12:25:33 PM1/16/21
to weewx-user
I've added the mem extension.  My test station, with a large number of feeds selected, is at: https://lockyer.ca/weather/OsoyoosLakeNorthEast


Regards,

Garry

bell...@gmail.com

unread,
Jan 16, 2021, 2:09:46 PM1/16/21
to weewx-user

Garry,
If you want me to run something, just make it available and give me some instructions.
rich

Garry A Lockyer

unread,
Jan 16, 2021, 4:29:35 PM1/16/21
to weewx...@googlegroups.com
Thanks, will do but later today.


Regards,

Garry Lockyer
E: Ga...@Lockyer.ca


On Jan 16, 2021, at 11:09, bell...@gmail.com <bell...@gmail.com> wrote:



garrya...@gmail.com

unread,
Jan 17, 2021, 3:43:59 AM1/17/21
to weewx-user
I haven't managed to package the "latest & greatest" up for others but I did do some testing.

It occurred to me that since my extension is now a service, not dependent on a skin, I could turn off a skin to see if that changed the problem.  So I disabled the Belchertown skin and restarted WeeWX.

And it looks like the problem has gone away, so at this time, it looks like the cause of the leak is associated with Belchertown, but I don't know enough (anything really!) to say whether it's about WeedWX calling Belchertown or Belchertown itself.

The station website at: https://lockyer.ca/weather/OsoyoosLakeNorthEast is no longer being updated ('cause Belchertown is disabled) but the mem graphs at: https://lockyer.ca/weather/OsoyoosLakeNorthEast/mem are.  The discontinuity of memory growth around 2:30 PM Saturday was caused by an poorly handled error for one of the feeds in my code - that's now fixed.  The data before about midnight are for testing before I disabled Belchertown.  I'll probably re-enable Belchertown after a few hours.

In the morning, I will update my MemoryTest to provide the smallest / simplest code to demonstrate the problem and make it available for others to test with.

Thanks again for all your help,

Garry

vince

unread,
Jan 17, 2021, 12:28:24 PM1/17/21
to weewx-user
On Saturday, January 9, 2021 at 8:02:51 PM UTC-8 garrya...@gmail.com wrote:
My experience is that when I recreate an include file for Belchertown on each archive cycle, as an extension to Belchertown or as a service, memory usage increases with each cycle and eventually errors start.  

I continue to think you need to concentrate on what your code is doing and your os configuration.
  • If you disable Belchertown and 'enable' your custom code that generates the include file, does memory grow ?
  • You also haven't told us what hardware you're on.  Is it a pi ?  What os ?   What version ?
  • Your log says you're running python3.  Have you tried the same thing on a python2 system ?
  • Can you duplicate the problem using just the Simulator driver and your addition so we can try to replicate it ?

 
Message has been deleted

garrya...@gmail.com

unread,
Jan 17, 2021, 3:04:21 PM1/17/21
to weewx-user
- If you disable Belchertown and 'enable' your custom code that generates the include file, does memory grow ?
  - Yes.  I just started at test run with WXFeedsMemoryTest.py (attached here) and Belchertown enabled.  
    The station website is at: https://lockyer.ca/weather/OsoyoosLakeNorthEast and the mem graphs are 

- You also haven't told us what hardware you're on.  Is it a pi ?  What os ?   What version ?
  - From an earlier post:
    RPi 4 - 8 GB
    Ubuntu Raspberry Pi OS 20.10
    Python3 - 3.8.6
    WeeWX 4.3
    Belchertown 1.2

    - Problem was first noticed on RPi Zero W with Raspberry Pi OS Lite.
    - Problem was reproduced on RPi 4 - 4 & 8 GB with Raspberry Pi OS Desktop.
    - My extension was reduced to minimum code needed to reproduce the problem - that code is attached.

- Your log says you're running python3.  Have you tried the same thing on a python2 system ?
  - No.  I have no interest in python2. :^))

- Can you duplicate the problem using just the Simulator driver and your addition so we can try to replicate it ?
  - The run I just started (about 11:30 AM PST) is using the Simulator (rather than WeatherFlow), and memory usage 
    is growing.  I'll let it fun for 1-2 hours and then disable Belchertown.

Regards,

Garry

Sorry but I was not able to attach a file to this message!  I was able to previously under Raspberry Pi OS so maybe it's related to Ubuntu?

Here's the code for WXFeedsMemoryTest.py:

######################################################################################################
#   WXFeedsMemoryTest.py
#
#    Copyright (c) 2021 Garry Lockyer - Ga...@lockyer.ca
#    See the file LICENSE.txt for your rights.
#
#   Based on alarm.py by:
#    Copyright (c) 2009-2019 Tom Keffer <tke...@gmail.com>
#    See the file LICENSE.txt for your rights.

"""
To use this service, add the following to the weewx configuration file:

[WXFeedsMemoryTest]
    # No options are required or processed at this time.

This service requests data from an RSS feed:


and creates an include file that the Belchertown Skin can include.

*******************************************************************************
To enable this service:
1) Copy this file to the WeWX bin/user directory - /home/weewx/bin/user if WeeWX was 
   installed with setup.py. See https://bit.ly/33YHsqX for where your user
   directory is located.

2) Modify the weewx configuration file by adding this service to the option
"report_services", located in section [Engine][[Services]].
[Engine]
  [[Services]]
    ...
    report_services = weewx.engine.StdPrint, weewx.engine.StdReport, user.WXFeedsMemoryTest.WXFeedsMemoryTest

# I want WXFeedsMemoryTest to run before StdReport so I actually use:

    report_services = weewx.engine.StdPrint, user.WXFeedsMemoryTest.WXFeedsMemoryTest, weewx.engine.StdReport

"""

import weewx
from weewx.engine import StdService
import logging
import os
import sys
import gc
import requests
import shutil

# Inherit from the base class StdService:

class WXFeedsMemoryTest(StdService):
    """
    A WeeWX service extension to demonstrate that WeeWX with the Belchertown skin 
    memory usage grows after each archive period.

    All it does is get one RSS feed and produce one Belchertown include file.

    Other than creating an include in skins/Belchertown there are no dependencies 
    on any skin.
    """
    
    def __init__(self, engine, config_dict):
        # Pass the initialization information on to my superclass:
        super(WXFeedsMemoryTest, self).__init__(engine, config_dict)

        self.log = logging.getLogger( __name__ )
        assert self.log != None

        self.log.info( "WXFeedsMemoryTest::__init__(). . ." )

        self.bind( weewx.NEW_ARCHIVE_RECORD, self.new_archive_record )

        return
        
    def new_archive_record(self, event):
        """Gets called on a new archive record event."""
        
        self.log.info( "WXFeedsMemoryTest::new_archive_record(). . ." )

        self.log.info( "Process ID: %d" % os.getpid() )

        #   This URL is for Alberta 511's Road Conditions RSS feed.


        # The include file is used by the Belchertown Skin.
        #
        # The problem:
        #
        # If the include file is static, as in it is not re-created while 
        # Belchertown is enabled, weewx memory usage does not grow.
        #
        # If the include file is dynamic, as in it is re-created while 
        # Belchertown is enabled, weewx memory usage increases 
        # after each file re-creation.
        #
        # If this service is enabled AND Belchertown is disabled, memory 
        # usage does not grow,

        includeFileName = "/home/weewx/skins/Belchertown/index_hook_after_charts.inc"
        tempFileName    = "/home/weewx/skins/Belchertown/WXFeedsTemp.inc"

        # The include file can be created DIRECTLY or INDIRECTLY:
        #
        # DIRECTLY: File is opened, written to and closed.
        #
        # INDIRECTLY: A temp file is opened, written to and closed and then 
        # renamed or copied to the Belchertown skin filename.  During 
        # testing, the thinking was the problem was associated with the 
        # actual writing of the file and that the problem might go away if 
        # the file was created and then renamed or copied. It did not.

        directFileCreation  = True  # If True, include file is created directly, if False, include is create as a 
                                    # 'temporary' file and then copied or renamed.
                                    # Does not seem to affect problem.
        copyFile            = True  # For indirect include file creation:
                                    # If True, temporary file is copied, if False, temporary file is renamed.
                                    # Does not seem to affect problem.
        recreateFile        = True  # If True, recreate the include file, if False, do not recreate the include file.
                                    #
                                    # Recreating the include file appears to cause weewx memory usage to grow.
                                    #
                                    # If True, weewx memory usage will increase on each archive cycle.
                                    #
                                    # If False and the include file does not exist when weewx/Belchertown starts, it 
                                    # will be created the first time this method is called but not on subsequent calls.
                                    # Memory usage will stabilize.

        assert gc.isenabled() == True

        self.log.info( "WXFeedsMemoryTest:" )
        self.log.info( "Memory use before gc.collect():" )
        x, y, z = gc.get_count()
        self.log.info( "gc.get_count(): %d, %d, %d" % ( x, y, z ) )
        self.log.info( "sys.getallocatedblocks(): %d" % sys.getallocatedblocks() )
        #sys._clear_type_cache()
        gc.collect()
        self.log.info( "Memory use after gc.collect():" )
        x, y, z = gc.get_count()
        self.log.info( "gc.get_count(): %d, %d, %d" % ( x, y, z ) )
        self.log.info( "sys.getallocatedblocks(): %d" % sys.getallocatedblocks() )
        self.log.info( "" )

        try:
            session = requests.Session()
            assert session != None

            response = session.get( feedURL )
            assert response != None
        except:
            self.log.info("Alberta511RoadConditions: requests.get() returned error: %d!" % response.status_code)
        
        if response.status_code == requests.codes.ok :
            roadConditions = response.json()

            self.log.info( "Alberta511RoadConditions: Received %d entries." % len( roadConditions ) )
        
        if os.path.exists( includeFileName ) == True and \
           recreateFile == False:
            return

        if os.path.exists( includeFileName ) == True:
            os.remove( includeFileName )

        if os.path.exists( tempFileName ) == True:
            os.remove( tempFileName )

        if directFileCreation == True:
            file = open( includeFileName, "wt", 1 )
        else:
            file = open( tempFileName, "wt", 1 )

        assert file != None

        for roadCondition in roadConditions:
            HTML =   "<table>\n"                                                                    \
                   +     "<thead>\n"                                                                \
                   +         "<tr>\n"                                                               \
                   +             "<th></th>\n"                                                      \
                   +             "<th></th>\n"                                                      \
                   +         "</tr>\n"                                                              \
                   +     "</thead>\n"                                                               \
                   +     "<tbody>\n"                                                                \
                   +         "<tr>\n"                                                               \
                   +             "<td><b>Roadway Name:&nbsp;</b></td>\n"                            \
                   +             "<td>%s</td>\n" % roadCondition.get( "RoadwayName" )               \
                   +         "</tr>\n"                                                              \
                   +         "<tr>\n"                                                               \
                   +             "<td><b>Location Description:&nbsp;</b></td>\n"                    \
                   +             "<td>%s</td>\n" % roadCondition.get( "LocationDescription" )       \
                   +         "</tr>\n"                                                              \
                   +         "<tr>\n"                                                               \
                   +             "<td><b>Primary Condition:&nbsp;</b></td>\n"                       \
                   +             "<td>%s</td>\n" % roadCondition.get( "Primary Condition" )         \
                   +         "</tr>\n"                                                              \
                   +         "<tr>\n"                                                               \
                   +             "<td><b>Id:&nbsp;</b></td>\n"                                      \
                   +             "<td>%s</td>\n" % roadCondition.get( "Id" )                        \
                   +         "</tr>\n"                                                              \
                   +         "<tr>\n"                                                               \
                   +             "<td><b>Area Name:&nbsp;</b></td>\n"                               \
                   +             "<td>%s</td>\n" % roadCondition.get( "AreaName" )                  \
                   +         "</tr>\n"                                                              \
                   +         "<tr>\n"                                                               \
                   +             "<td><b>Last Updated:&nbsp;</b></td>\n"                            \
                   +             "<td>%s</td>\n" % roadCondition.get( "LastUpdated" )               \
                   +         "</tr>\n"                                                              \
                   +   "</tbody>\n"                                                                 \
                   + "</table>\n"                                                                   \
                   + "<hr>\n"

            file.write( HTML )
            file.flush()
            os.fsync( file.fileno() )

        del roadConditions
 
        file.close()
        del file

        if directFileCreation == False:
            # The include file should have been deleted above but 
            # so that we don't get any exceptions because the destination 
            # already exists, we'll delete it here, before a copy or rename.

            if os.path.exists( includeFileName ) == True:
                os.remove( includeFileName )

            if copyFile == True:
                shutil.copy2( tempFileName, includeFileName )
            else:
                os.replace( tempFileName, includeFileName )

        response.close()
        del response

        session.close()
        del session
    
        self.log.info( "WXFeedsMemoryTest:" )
        self.log.info( "Garbage collection after writing file:" )
        self.log.info( "Memory use before gc.collect():" )
        x, y, z = gc.get_count()
        self.log.info( "gc.get_count(): %d, %d, %d" % ( x, y, z ) )
        self.log.info( "sys.getallocatedblocks(): %d" % sys.getallocatedblocks() )
        #sys._clear_type_cache()
        gc.collect()
        self.log.info( "Memory use after gc.collect():" )
        x, y, z = gc.get_count()
        self.log.info( "gc.get_count(): %d, %d, %d" % ( x, y, z ) )
        self.log.info( "sys.getallocatedblocks(): %d" % sys.getallocatedblocks() )
        self.log.info( "" )
        
        return
#########################################################################################

vince

unread,
Jan 17, 2021, 4:11:18 PM1/17/21
to weewx-user
On Sunday, January 17, 2021 at 12:04:21 PM UTC-8 garrya...@gmail.com wrote:
    RPi 4 - 8 GB
    Ubuntu Raspberry Pi OS 20.10
    Python3 - 3.8.6
    WeeWX 4.3
    Belchertown 1.2
    - Problem was first noticed on RPi Zero W with Raspberry Pi OS Lite.
    - Problem was reproduced on RPi 4 - 4 & 8 GB with Raspberry Pi OS Desktop.
    - My extension was reduced to minimum code needed to reproduce the problem - that code is attached.



Are you setup.py or dpkg for your installation ? Reason I'm asking is that Tom previously wondered about a memory leak in other things like PIL/pillow.   That's also why I was wondering about python2 vs. python3, as that would change the versions of underlying things.

Let us know how it goes after you leave things alone for a few hours...
 
That URL returns a 800K file which is kinda big.  Do road conditions change 'that' often ?   Why not have a standalone cron job that does all the heavy lifting and writes out the output HTML and see if that helps any ?  That would almost eliminate weewx from the equation wouldn't it ?

 

Garry A Lockyer

unread,
Jan 17, 2021, 6:12:21 PM1/17/21
to weewx...@googlegroups.com
Thanks for your assistance.

I used setup.py to install WeeWX.

Yes, the HTML file is huge but it really is just a test case.  I suspect a smaller feed that generates a smaller include file would cause a smaller leak that wouldn’t be so obvious.  My actual extension has filters to select items of particular interest which would result in a smaller include file.

Re: cron, I’m thinking of using it to restart WeeWX every day.

I hadn’t thought about a separate job to do the RSS feed pulls because (in retrospect because of your good suggestion!) I’ve pretty much had “tunnel vision” working within WeeWX and Belchertown.  At this time, I’d bet that the memory problem will still exist because about 2 hours ago I disabled Belchertown.

The only change I made was setting ‘enable’ to false.  Memory usage is flat!

Regards,

Garry Lockyer
E: Ga...@Lockyer.ca


On Jan 17, 2021, at 13:11, vince <vince...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

vince

unread,
Jan 17, 2021, 7:06:13 PM1/17/21
to weewx-user
garry - I installed your thing and did not see a memory increase 'without' belchertown running, and but it 'is' leaking' it seems 'with' belchertown 1.2 installed.  I'm running the simulator and ubuntu-2004 in a Vagrant VM running python3 weewx 4.3.0 via setup.py -  Here's some raw numbers with a little commentary for when I did what.

echo "select dateTime,mem_size,mem_rss,mem_share from archive;"|sqlite3 /home/weewx/archive/mem.sdb | sed -e s/\|/\\t/g

---- ran without belchertown here ----

1610919015 132.80859375 51.07421875 15.6171875
1610919315 132.9375 51.375 15.6171875
1610919615 132.96484375 51.40234375 15.6171875
1610919915 132.96484375 51.40234375 15.6171875
1610920215 132.95703125 51.39453125 15.6171875
1610920515 132.95703125 51.39453125 15.6171875
1610920815 133.15234375 51.58984375 15.6171875
1610921115 133.15234375 51.875 15.62109375
1610921415 133.15234375 51.875 15.62109375
1610921715 133.32421875 52.046875 15.62109375

---- restarted with belchertown here ----
1610922315 140.2890625 65.08203125 15.7421875
1610922615 143.5234375 71.39453125 15.7421875
1610922915 145.21875 75.625 15.7421875
1610923215 146.71875 79.61328125 15.7421875
1610923515 147.96875 83.43359375 15.7421875
1610923815 149.60546875 87.3671875 15.6875

--- restarted with belchertown 'on' and .inc generator 'off' --------
1610925015 128.30078125 49.48046875 13.20703125
1610925315 128.6796875 50.39453125 13.20703125
1610925615 128.6796875 50.39453125 13.20703125
1610925916 129.1796875 50.6796875 13.20703125
1610926215 129.1796875 50.31640625 12.84375
1610926515 129.4296875 50.90625 12.96875
1610926815 129.4296875 50.90625 12.96875
1610927115 129.4296875 50.90625 12.96875
1610927415 129.4296875 50.90625 12.96875
1610927715 129.4296875 50.90625 12.96875
1610928015 129.4296875 50.90625 12.96875


Garry A Lockyer

unread,
Jan 17, 2021, 7:23:45 PM1/17/21
to weewx...@googlegroups.com
So you are seeing what I am - the leak is evident when belchertown is enabled *and* the .inc file is regenerated, yes?


Regards,

Garry Lockyer
E: Ga...@Lockyer.ca


On Jan 17, 2021, at 16:06, vince <vince...@gmail.com> wrote:

garry - I installed your thing and did not see a memory increase 'without' belchertown running, and but it 'is' leaking' it seems 'with' belchertown 1.2 installed.  I'm running the simulator and ubuntu-2004 in a Vagrant VM running python3 weewx 4.3.0 via setup.py -  Here's some raw numbers with a little commentary for when I did what.
--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

vince

unread,
Jan 17, 2021, 8:40:15 PM1/17/21
to weewx-user
Restarted with your add-on plus belchertown both enabled - the RSS is nominally growing at 4 MB per 5-minute archive period.

1610928917 139.53125 63.97265625 15.453125
1610929217 143.02734375 70.41796875 15.4921875
1610929517 144.46484375 74.43359375 15.4921875
1610929817 145.96484375 78.48046875 15.55078125
1610930117 147.46484375 82.65625 15.58203125
1610930417 149.09375 86.69921875 15.58203125
1610930717 150.34375 90.55078125 15.58203125
1610931017 151.84375 94.203125 15.24609375

garrya...@gmail.com

unread,
Jan 18, 2021, 12:27:42 PM1/18/21
to weewx-user
Thanks!

I think I've done all that I can to isolate the problem.

I'm going to get back to finishing up development and add a cron entry to restart WeeWX every 24 hours in the wee hours of the morning.

I will be re-deploying a Davis Pro 2 system very soon (within the week) and then deploying another Davis Pro 2 system shortly after that.  And sometime after than, a couple of WeatherFlow Tempest systems.

I'll post everything on GitHub soonest.

Regards,

Garry
PS: Re: Ubuntu Desktop on RPi, I've been using it for development for the last few days and it has not crashed or hung, so I think I'm going to stay with it.  Only issue is wireless mouse lag (Raspberry Pi OS had same/similar problem) and I've ordered a Logitech MK540 wireless keyboard/mouse combo to see if newer hardwaere works better (than my dated Microsoft mouse).

garrya...@gmail.com

unread,
Jan 19, 2021, 3:55:38 PM1/19/21
to weewx-user
I wasn't happy leaving this is as is so I did some more testing.

I looked in /home/weewx/bin/user/belchertown.py for any mention of the 4 include files it uses to add content - nothing there.

So, I looked in /home/weewx/skins/Belchertown/index.html.templ and found 4 lines that reference the include files - they are of the form:

                #if os.path.exists("index_hook_after_station_info.inc")
                <!-- Start of index_hook_after_station_info row -->
                <div class="row index-hook-after-station-info border-bottom">
                    #include "index_hook_after_station_info.inc"
                </div>
                <!-- End of index_hook_after_station_info row -->
                #end if
That caused me to look up the Cheetah '#include'  directive:

"7.6 #include

Syntax:

#include [raw] FILENAME_EXPR #include [raw] source=STRING_EXPR

The #include directive is used to include text from outside the template definition. The text can come from an external file or from a $placeholder variable. When working with external files, Cheetah will monitor for changes to the included file and update as necessary.

This example demonstrates its use with external files:

#include "includeFileName.txt"
The content of "includeFileName.txt" will be parsed for Cheetah syntax.

And this example demonstrates use with $placeholder variables:

#include source=$myParseText
The value of $myParseText will be parsed for Cheetah syntax. This is not the same as simply placing the $placeholder tag ``$myParseText'' in the template definition. In the latter case, the value of $myParseText would not be parsed.

By default, included text will be parsed for Cheetah tags. The argument ``raw'' can be used to suppress the parsing.

#include raw "includeFileName.txt" #include raw source=$myParseText

Cheetah wraps each chunk of #include text inside a nested Template object. Each nested template has a copy of the main template's searchList. However, #set variables are visible across includes only if the defined using the #set global keyword.

All directives must be balanced in the include file. That is, if you start a #for or #if block inside the include, you must end it in the same include. (This is unlike PHP, which allows unbalanced constructs in include files.)"

I decided to insert the 'raw' argument into the 4 '#include' directives to see if anything changed.

After about 1 hour. memory usage did not grow above about 65 MB!

I just removed the 'raw' argument and restarted WeeWX.  After just a few minutes, memory usage is almost 300 MB and growing!

As before, the station website is at: https://lockyer.ca/weather/OsoyoosLakeNorthEast and the mem graphs are at https://lockyer.ca/weather/OsoyoosLakeNorthEast/mem.

So, I think I have conclusively shown  that the problem is within Cheetah and is related to it parsing an include file.

I'm going to re-insert the 'raw' arguments and carry on.

Should I report this issue to the Cheetah folks or is there someone closer to the Cheetah developers that can better do that?

Regards,

Garry


vince

unread,
Jan 19, 2021, 4:15:07 PM1/19/21
to weewx-user
I'm thinking it's actually working as designed if you assume Pat intended that cheetah directives should be parsed within a .inc(luded) file.  You just might be the edge case where you have a rather massive include that does 'not' want cheetah directive parsing.

If you open an issue with Belchertown in github perhaps Pat could take a look when he gets around to it and verify if it looks like a Cheetah issue.  Maybe he can just add a switch to the templates in Belchertown to let you control whether the 'raw' argument is provided or not (?)

If you open an issue with Cheetah, then you'd need to have a no-weewx minimal test case to show what you're seeing, so they can try to recreate the issue.

But for your case you seem to have figured out a workaround.   Just patch your skin to set the cheetah behavior that works for you.
(and yecch - cheetah - yecccch :-)

Garry A Lockyer

unread,
Jan 19, 2021, 4:49:43 PM1/19/21
to weewx...@googlegroups.com
Thanks for your reply.

I’m going to respectfully disagree though re: things working as intended. . .

Yes, the include files I’m playing with are huge but only for testing - aggravating - the problem.  The problem was first noticed with very much smaller include files on a 512 MB system that crashed/hung after about 3 days.  I can reproduce the problem in under 10 minutes using bigger include files.

I believe the Cheetah generator should not consume ever increasing amounts of memory, that it has a memory leak that should be fixed.  I may report the problem to them by entering a brief summary and pointing them to this string.  All the information they need is in here.

But I do agree with your “and yecch - cheetah - yecccch”!  I prefer restarting WeeWX daily over hand editing index.html.tmpl but that frequency may not work for all cases so I’ll probably live with editing.

Regards and thanks again,

Garry Lockyer
E: Ga...@Lockyer.ca


On Jan 19, 2021, at 13:15, vince <vince...@gmail.com> wrote:

I'm thinking it's actually working as designed if you assume Pat intended that cheetah directives should be parsed within a .inc(luded) file.  You just might be the edge case where you have a rather massive include that does 'not' want cheetah directive parsing.
--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

vince

unread,
Jan 19, 2021, 5:13:51 PM1/19/21
to weewx-user
On Tuesday, January 19, 2021 at 1:49:43 PM UTC-8 garrya...@gmail.com wrote:
 I prefer restarting WeeWX daily over hand editing index.html.tmpl but that frequency may not work for all cases so I’ll probably live with editing.


You should be able to let weewx run literally years without resetting it.

Personally, I'd just edit/hack/patch your template file(s) and leave yourself a big README file so you don't forget you did it, in case you update Belchertown in the future.

But of course reporting a perceived bug with a nice concise description and a test case they can use to recreate the problem is always good.....although they're not going to battle through this thread.  Please do them a favor and do a quality bug report with something repeatable so they can reproduce the problem there.

bell...@gmail.com

unread,
Jan 19, 2021, 6:55:53 PM1/19/21
to weewx-user
I agree that something does not seem quite right with Cheetah’s ‘monitoring’ of include files. I took a different approach.
First, I wrote a simple WeeWX service that would perform the ‘touch’ command against the large file and a simple template that included the large file. When the file modified attribute changed, I observed the memory growth, otherwise all was stable. Also, when the file was not ‘touched’ the Cheetah processing was noticeably faster.
Next I wrote a standalone program to run a loop that ran the touch command and invoke Cheetah.  I ran it 3 times. Each time around loop 90, the memory growth stopped and the processing was fast as though Cheetah was using cached data. 
I am back to running under WeeWX to see if it will level off like it does standalone. I am also wading through the Cheetah  code and documentation.  As of now, I thoroughly confused. 
rich

Tom Keffer

unread,
Jan 19, 2021, 7:16:46 PM1/19/21
to weewx-user
I'm not quite following this discussion.

With Cheetah, you can compile the template and save the results. I believe this is what monitors the changed status of #include files. But, that's not the way WeeWX uses templates. It compiles, evaluates, then throws the results away. 

If you use the "raw" directive in an #include, then the file will not be parsed. So, how can any $ directives work?

Or, does Belchertown work differently?

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/bfd80498-e70b-4a26-9f76-6a922439a1ben%40googlegroups.com.

vince

unread,
Jan 19, 2021, 8:05:28 PM1/19/21
to weewx-user
No, I think you're following fine.

In his unique case, there are no $ directives nor cheetah parsing of the file being #include(d).  He's pre-generating it in his service as a complete html table and letting Belchertown #include that html in the right place on the resulting index.html page.    There's no alteration of the contents of the table being #include(d).  It's all pre-populated.   So for his unique needs, using raw in there is a workaround for the memory leak issue he found, as a short (?) term solution.

What I was suggesting is that if others run into this one, perhaps a github issue asking Pat to see if he can add a little more configurability to the skin .tmpl might be helpful.   Belchertown is so very configurable already, it looked to me at a quick glance that adding a switch or two to control raw-vs-not for the places where he has #include hooks might be worth thinking about.


Garry A Lockyer

unread,
Jan 19, 2021, 8:08:46 PM1/19/21
to weewx...@googlegroups.com
With apologies if you know much of this. . .

The Belchertown skin allows you to add content to the main page it emits each archive cycle via 4 “hook” files.  If the hook files are in Belchertown’s skin folder, their content is included in ‘index.html’.

I first learned about this feature ‘cause it is used by a WeeWX/Belchertown extension to fetch aviation weather observations (METARs) and forecasts (TAFS).  That extension is a search list extension and eventually I used that mechanism to create my own extension to fetch METARs and TAFs, and extended it to fetch weather information (current conditions and forecasts) from Environment Canada and NOAA’s National Weather Service, Storm Prediction Center and National Hurricane Service, as well as road information from 511 Alberta, Rive BC and Washington State Department of Transportation.

I deployed a Davis Pro 2 weather station with WeeWX, Belcherton and my extension and it crashed after about 3 days.  After much debugging and rewriting (including converting my Belchertown specific search list extension to a skin independent service to create the include files, using Ubuntu instead of Raspberry Pi OS, and creating a much smaller/simpler program to demonstrate the problem), I narrowed the problem down to the Cheetah generator, specifically the parsing of included files by the Cheetah generator.

With “out of the box” WeeWX and Belchertown, If the include files are recreated (each archive cycle) and the Belchertown skin is enabled, WeeWX memory usage will grow every cycle.  If the ‘raw’ argument is added to the ‘include’ directives in Belcherotwn’s ‘index.html.tmpl’ memory usage stabilizes.  The bigger the include files, the faster memory usage grows.

My extension does not use $ directives in the include files it creates so losing that functionality is not a problem for me but obviously it could be an issue for other extensions or include file generators.

I don’t think the Cheetah generator should cause increasing memory usage because it’s asked to include content, whether it’s asked to parse it or not.  I can live with modifying ‘index.html.tmpl’ but hope the underlying problem gets fixed.

I’ve entered as ‘issue’ against Belchertown so that its author is aware, can possibly provide a workaround in his code and/or “escalate” the issue to the Cheetah folks.

Regards,

Garry Lockyer
E: Ga...@Lockyer.ca


On Jan 19, 2021, at 16:16, Tom Keffer <tke...@gmail.com> wrote:



Tom Keffer

unread,
Jan 19, 2021, 8:14:20 PM1/19/21
to weewx-user
OK, got it.

Still, I'm surprised that memory would grow with a #include, raw, or not. As I say, compiled templates are thrown away after every use.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/7F58A737-4F41-4C62-8C7F-A5CA7678C672%40gmail.com.

vince

unread,
Jan 19, 2021, 8:16:03 PM1/19/21
to weewx-user
Indeed.  Everybody else is equally surprised.   But it's very reproducible and repeatable.

bell...@gmail.com

unread,
Jan 19, 2021, 8:21:31 PM1/19/21
to weewx-user
Ok, my standalone testcase was wrong. I think I have a valid one here, https://github.com/bellrichm/experiments
It just calls Cheetah.Template.Template in a loop.  If the include file modified attribute does not change, all seems stable. If it changes, memory usage seems to grow.
rich

vince

unread,
Jan 19, 2021, 10:38:24 PM1/19/21
to weewx-user
Just a followup - I took belchertown out of the picture and used my minimal demo-skin and added 'one line' that #include(d) a file generated by Garry's service, and it 'is' leaking memory albeit less quickly than the belchertown example.....

I'll let it run overnight and we'll see what it looks like after 12 more hours running.   The VM is set to only 1GB RAM but I'm hoping if it runs out it'll just go into swap and stay up.    OS is debian 10 under Vagrant/VirtualBox with 4.3.0 installed using python3 and setup.py

Some mem.sdb output (date/time, units, interval, mem_size, mem_rss, mem_share)...

1611109216 16 5 141.0625 71.734375 12.2734375
1611109516 16 5 142.5625 75.65234375 12.3359375
1611109816 16 5 144.0625 79.65625 12.3359375
1611110116 16 5 145.5625 84.5234375 12.3359375
1611110416 16 5 147.24609375 88.0859375 12.3359375
1611110716 16 5 148.49609375 91.43359375 12.3359375
1611111016 16 5 149.99609375 95.421875 12.3359375
1611111316 16 5 151.49609375 100.3359375 12.3359375
1611111616 16 5 152.99609375 103.8359375 12.3359375
1611111916 16 5 154.49609375 107.51171875 12.3359375
1611112216 16 5 155.99609375 111.5 12.3359375
1611112516 16 5 157.24609375 116.0859375 12.3359375
1611112816 16 5 158.74609375 119.5859375 12.3359375
1611113116 16 5 160.24609375 123.24609375 12.3359375
1611113416 16 5 161.74609375 127.234375 12.3359375
1611113716 16 5 163.24609375 132.11328125 12.3359375



Screen Shot 2021-01-19 at 7.33.46 PM.png

Garry A Lockyer

unread,
Jan 19, 2021, 10:45:42 PM1/19/21
to weewx...@googlegroups.com
Many thanks to all for poking at this!


Regards,

Garry Lockyer
E: Ga...@Lockyer.ca


On Jan 19, 2021, at 19:38, vince <vince...@gmail.com> wrote:

Just a followup - I took belchertown out of the picture and used my minimal demo-skin and added 'one line' that #include(d) a file generated by Garry's service, and it 'is' leaking memory albeit less quickly than the belchertown example.....
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/4916786b-3bb2-49a6-93b1-87a60a822dbdn%40googlegroups.com.
<Screen Shot 2021-01-19 at 7.33.46 PM.png>

bell...@gmail.com

unread,
Jan 20, 2021, 9:13:35 AM1/20/21
to weewx-user
Garry,
I might have a workaround for you. Wrap your html file in a template. something like this.
index.html.tmp (this is in the belchertown skin)
#include index_hook_after_charts.inc

index_hook_after_charts.inc (this is the 'wrapper', naturally if you 'owned' index.html.tmpl you would not need it)
#include raw generated.html

generated.html (the file you generate - name as you want)
your html

I ran this on my loop 10,000 times and saw no appreciable memory increase and performed better too.
rich

bell...@gmail.com

unread,
Jan 20, 2021, 9:24:53 AM1/20/21
to weewx-user
Note, it is growing - just much slower.

Garry A Lockyer

unread,
Jan 20, 2021, 9:32:56 AM1/20/21
to weewx...@googlegroups.com
Very clever!  Thanks!!

I’ll give it a try in a few hours.


Regards,

Garry Lockyer
E: Ga...@Lockyer.ca


On Jan 20, 2021, at 06:24, bell...@gmail.com <bell...@gmail.com> wrote:

Note, it is growing - just much slower.

Garry A Lockyer

unread,
Jan 20, 2021, 10:07:23 AM1/20/21
to weewx...@googlegroups.com
I haven’t made any changes yet but in thinking about your workaround, it will be interesting to see if Cheetah actuality processes the included include file or if it skips it because the outer include file’s attributes are static.

If Cheetah doesn’t process the inner include file, it still should be a way to inject the ‘raw’ argument without having to edit ‘index.html.tmpl’.  That is, my code would generate both the outer and inner include files each cycle, with the outer include file containing the ‘raw’ argument as you suggest.

Regards,

Garry Lockyer
E: Ga...@Lockyer.ca


On Jan 20, 2021, at 06:24, bell...@gmail.com <bell...@gmail.com> wrote:

Note, it is growing - just much slower.

vince

unread,
Jan 20, 2021, 11:19:32 AM1/20/21
to weewx-user
Well FWIW, it's not Belchertown.    I took my minimalist demo-skin and aded the one include line in it and boy does it grow.....

Two tests with a restart that's obvious.
Screen Shot 2021-01-20 at 8.18.25 AM.png

garrya...@gmail.com

unread,
Jan 20, 2021, 1:16:26 PM1/20/21
to weewx-user
Thanks to Vince, Rich and all who took the time to investigate this problem.

Happy that it’s not WeeWX or Belchertown, two fine programs!

I hope to release my WXFeeds service extension within the next few days.

Regards,

Garry

garrya...@gmail.com

unread,
Jan 21, 2021, 12:47:26 PM1/21/21
to weewx-user
It looks like the workaround Rich suggested works.

A couple of notes:

1. the generated filename must be in quotes (#include raw "generated file name")
    - the Cheetah generator will throw an error if not.

2. There must not be 'raw' arguments in index.html.tmpl
    - I left them in after previous debugging
    - if they are in, the Cheetah generator will not parse the generated file and the Cheetah directive will 
      be display in the output HTML.

I will be putting a control in my extension to enable/disable this workaround.

Again, thanks to Rich and others for helping with this.

Regards,

Garry

Pat

unread,
Jan 22, 2021, 3:00:47 PM1/22/21
to weewx-user
I'm tracking this thread a bit slowly... Is the raw portion within the #include custom to your setup - or is this something that should be added to the Belchertown skin?

Garry A Lockyer

unread,
Jan 22, 2021, 4:01:37 PM1/22/21
to weewx...@googlegroups.com
The workaround is to have “static” index hook files that each just have 1 Cheetah directive in them.  That directive has the ‘raw’ argument.  For my extension, I will implement the work around, using “index-hook-after-station-info.inc” (and assuming I got the file name correct!) as the example, such that:

index-hook-after-station-info.inc contains:

#include raw “WXFeeds-index-hook-after-station-info.inc”

and is not re-created.

My extension will generate “WXFeeds-index-hook-after-station-info.inc” each archive cycle.

I don’t think any changes to Belchertown are required other than perhaps a warning that memory usage can increase if the hooks are used without the workaround (until the Cheetah generator is fixed).

Thanks for your time, attention and the Belchertown skin!

Regards,

Garry Lockyer
E: Ga...@Lockyer.ca


On Jan 22, 2021, at 12:00, Pat <pobri...@gmail.com> wrote:

I'm tracking this thread a bit slowly... Is the raw portion within the #include custom to your setup - or is this something that should be added to the Belchertown skin?

Timothy L

unread,
Jan 22, 2021, 8:14:48 PM1/22/21
to weewx...@googlegroups.com
Would it be possible to post a step by step instruction for your changes if not too much trouble?

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/772028EA-5967-47A8-A130-5864076C10DD%40gmail.com.

Garry A Lockyer

unread,
Jan 22, 2021, 8:33:51 PM1/22/21
to weewx...@googlegroups.com
I’ll assume your talking to me.

There are no proposed changes to any existing software - at least not to WeeWX or the Belchertown skin.  Any changes would likely be modifications to the Cheetah generator so that it does not consume ever increasing amounts of memory, but I know nothing technical about the Cheetah generator so can’t say if such modifications are possible.

That being said . . .

*IF* someone uses the “index hook” feature of the Belchertown skin *AND* the hook include files are regenerated frequently, such as every archive period, they should be aware that they may experience ever increasing memory usage.  The bigger the include files, the greater the increase in memory usage.

The problem can be avoided by creating “static” index hook files that reference (with a ‘#include raw “filename” directive) the file containing the HTML to be included which can be periodically regenerated.

I intend to implement the workaround in my WeeWX extension such that a user does not have to worry about about the technicalities but that code isn’t written yet.

Regards,

Garry Lockyer
E: Ga...@Lockyer.ca


On Jan 22, 2021, at 17:14, Timothy L <lecoqacr...@gmail.com> wrote:



Timothy L

unread,
Jan 22, 2021, 9:40:04 PM1/22/21
to weewx...@googlegroups.com
Ok thank you Garry

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/04E005A6-BF4B-40A8-AE3F-0B3E80292402%40gmail.com.

wes...@gmail.com

unread,
Jan 23, 2021, 10:47:59 AM1/23/21
to weewx-user
has anyone filed an issue on the github project for cheetah?

garrya...@gmail.com

unread,
Jan 28, 2021, 2:39:39 AM1/28/21
to weewx-user

I just entered an issue against Cheetah3.

garrya...@gmail.com

unread,
Jan 28, 2021, 12:22:25 PM1/28/21
to weewx-user
The Cheetah folks have asked for a pure Cheetah method to reproduce the problem, which I do not have.

Perhaps someone else can assist them.


Regards,

Garry

bell...@gmail.com

unread,
Jun 25, 2023, 11:46:19 AM6/25/23
to weewx-user
In my experimental skin I was running into this issue, so I decided to delve into Cheetah’s internals and learned the following. As part of the compilation process the module is added to sys.modules and the generated code is cached. If one is dynamically generating template source and it is different, then it will not be found in the cache and an additional entry will be added to sys.modules and the cache. This will result in ever increasing memory usage. Since the OP was dynamically generating html and not Cheetah source, it could be included as ‘raw’. This skips the compilation step and the addition of it to sys.modules and the cache.
- rich

Tom Keffer

unread,
Jun 25, 2023, 4:21:55 PM6/25/23
to weewx...@googlegroups.com
Good sleuthing! Pretty crappy algorithm that allows a cache to grow without bounds.

The skins that come with WeeWX (Seasons, Standard) use static source template files. They are always the same. So, if I understand you correctly, each template should take only one slot in sys.modules.

Does the Belchertown skin do things differently?

-tk


On Sun, Jun 25, 2023 at 8:46 AM bell...@gmail.com <bell...@gmail.com> wrote:
In my experimental skin I was running into this issue, so I decided to delve into Cheetah’s internals and learned the following. As part of the compilation process the module is added to sys.modules and the generated code is cached. If one is dynamically generating template source and it is different, then it will not be found in the cache and an additional entry will be added to sys.modules and the cache. This will result in ever increasing memory usage. Since the OP was dynamically generating html and not Cheetah source, it could be included as ‘raw’. This skips the compilation step and the addition of it to sys.modules and the cache.
- rich

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/c33494f0-1ff1-464e-b37c-df48f02dc1d3n%40googlegroups.com.

bell...@gmail.com

unread,
Jun 25, 2023, 5:12:39 PM6/25/23
to weewx-user
Your understanding is correct. As part of my debugging I ran the Seasons skin and some additional skins. After each template was processed I logged the number of entries in sys.modules. On the first reporting cycle it increased by 1 for each template. After that it has not changed.
Belchertown allows the station owner to add custom content via #includes. The OP was creating this html content every archive and therefore the cache lookup failed. Since it was html it could be included with the ‘raw’ directive, skipping the whole Cheetah compilation process.
- rich

Tom Keffer

unread,
Jun 25, 2023, 5:22:50 PM6/25/23
to weewx...@googlegroups.com
Nicely done, Rich. Fits the facts perfectly.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/f6a401d0-614d-4b7d-b44e-9f58f0870f8en%40googlegroups.com.

Karen K

unread,
Jun 26, 2023, 12:35:15 AM6/26/23
to weewx-user
Tom Keffer schrieb am Sonntag, 25. Juni 2023 um 23:22:50 UTC+2:
Nicely done, Rich. Fits the facts perfectly.


So we could include some monitoring of sys.modules in the core loop of WeeWX and complain to the logs if there is such an issue? 

Tom Keffer

unread,
Jun 26, 2023, 8:01:03 AM6/26/23
to weewx...@googlegroups.com
That won't fix anything. Possible solutions:

1. Submit an issue to the Cheetah authors.

2. Adopt a best practice to not change template code.

3. Or, if you do, disable the Cheetah caching. Put this in the Belchertown code somewhere:

import Cheetah.Template.Templace
Cheetah.Template.Template._CHEETAH_useCompilationCache = False

-tk


--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/0813965d-814f-4297-ac1b-b4c40a364396n%40googlegroups.com.

Karen K

unread,
Jun 26, 2023, 9:25:38 AM6/26/23
to weewx-user
Tom Keffer schrieb am Montag, 26. Juni 2023 um 14:01:03 UTC+2:
That won't fix anything. Possible solutions:

No, that would not fix anything, but would warn users, that are not aware of the problem.
 
import Cheetah.Template.Templace
Cheetah.Template.Template._CHEETAH_useCompilationCache = False

I guess that is a reasonable solution. I did not know about that possibility.

bell...@gmail.com

unread,
Jun 26, 2023, 9:37:18 AM6/26/23
to weewx-user
That will stop Cheetah from using its cache, but it will still be added to sys.modules; so I think the memory will still grow over time.
I think (2) is very reasonable. If a skin author is dynamically creating a template, they can just take the next step and create the final content. That is what I ended up doing. I thought I was being smart and was dynamically creating a template because it would allow code reuse. Once I realized what was happening, I changed the skin to create the final content. (Note, I had looked into ways to not update sys.modules, but in my mind the added complexity was not worth it.)
rich
Reply all
Reply to author
Forward
0 new messages