Tracking down a long standing memory leak
114 views
Skip to first unread message
gjr80
Nov 5, 2016, 9:55:22 PM11/5/16
to weewx-development
I have had a long running memory leak on my on-line weewx system that I have finally managed to nail with what I think is a somewhat bizarre cause. Its a bit of a long story I am afraid...
My on-line system runs some some 20+ skins (some generating a single report only) and a multitude of search list extensions support these skins. The first time I noticed the issue was when weewx would stop producing output, weewx was still running but nothing was occurring (records being saved, reports being generated). I could stop weewx and start it again no problems and things wuld resume. The first time I gave it the benefit of the doubt. A couple of weeks later it happened again. A little investigation revealed that I was running out of memory. I use the cmon extension to monitor my system and I could see that weewx memory usage was a classic sawtooth, starting from a some value that I cannot remember now (this was well over a year ago) and gradually stepping up to a little under the 512 MB my (then) RPi had. I didn't have time to deal with the root cause (nor did I know how to find the root cause) so I lived with the issue. Some fairly frequent tweaking of my system tended to mask the issue as weewx was being restarted fairly regularly.
Move on a while and I replaced my RPi with an Odroid U3, the issue remained but at the Odroid has 2GB ram the time between lockups was greater. I now recorded some data; ram usage was increasing around 70-80 MB per day and I would generally get around 4 weeks between lockups. The ram usage would increase straight away after a weewx start and it would jump by around 3MB every hour or so. I upgraded from Debian Wheezy to Jessie and at first the issue disappered (I think) but it came back soon after. I had seen some posts about other users having a memory leak with PIL or pillow so I made sure it was up to date on my system. As I use Debian quite often the packages are some what behind the latest release so I forced Debian to use the latest package. Still no change. Given the number of skins and SLE I run I suspected that one of my skins or associated SLE was the source of the problem.
Step forward again to a couple of months back and I found some time to pare my system right back and gradually add 1 skin at a time. I eventually found the issue was caused by my Standard skin. My Standard skin is a modified version of the stock weewx Standard skin; I have moved some of the elements on the pages, have dropped the standard weewx plots in favour of Highcharts based plots, have added forecast output from the forecast extension as well as adding an 'alltime' page. So there were some substantial changes. I eventually tracked the issue down to a file that I was generating every hour using the forecast extension. Running the forecast extension with a 10 day WU forecast chews up a bit of time generating the resulting HTML. I use this forecast info on both my Saratoga PHP based site and my weewx site; they are somewhat different so for the Saratoga site I generate the forecast PHP page in its entirety and for the weewx page I generate a file forecast.incl that contains the forecast HTML. I generate these pages each hour considering the time to process and since the forecast info is slow changing. The forecast.incl file is used as an include in the weewx index.html.tmpl template that is generated every archive period. A convoluted arrangement that could no doubt be simplified. In any case, I found that if I stop generating the forecast.incl file the system memory was stable. So I restored everything else and left forecast.incl disabled and the system has been running fine since September, albeit without an updated forecast on my weewx page.
Yesterday I decided to sit down and nut out the issue. The template I use to generate forecast.incl is near identical to the stock forecast extension template forecast_table.inc.tmpl (I have added a couple of comments and dropped the verbatim CSS code from the stock template). The supporting SLE I use with this template is identical to the stock forecast extension SLE. So I replaced my forecast.incl.tmpl code with that from the forecast extension report, now the files were identical. The memory leak was evident immediately and was now around 3MB per archive period (rather than per hour) since I was generating this file every archive period. This was crazy I thought, I am using identical template and SLE code. I then noticed that the forecast extension generates a file forecast_table.inc but I was generating forecast.incl. So on the off chance I changed my template name from forecast.incl.tmpl to forecast_table.inc.tmpl. Memory leak disappeared. I changed to forecast.inc.tmpl, still no memory leak. I had to go but left it overnight, still no memory leak. This morning I changed to forecast.incl.tmpl again and the memory leak returned immediately. back to forecast.inc.tmpl and memory leak disappeared.
This seems bizarre, we have templates like index.html.tmpl that generate fine so the 4 letter extension seems to not be the issue per se. forecast.incl.tmpl generates fine but with a memory leak. I have been unable to get onto the cheetah site today to follow this up and see if cheetah has some sensitivity to .incl. I don't see how the forecast extension can be at fault, its just a template name that is being changed. I don't know if I have missed something obvious or not, but in any case long standing memory leak has disappeared and hopefully there will be no more scheduled restarts.
Gary
My on-line system runs some some 20+ skins (some generating a single report only) and a multitude of search list extensions support these skins. The first time I noticed the issue was when weewx would stop producing output, weewx was still running but nothing was occurring (records being saved, reports being generated). I could stop weewx and start it again no problems and things wuld resume. The first time I gave it the benefit of the doubt. A couple of weeks later it happened again. A little investigation revealed that I was running out of memory. I use the cmon extension to monitor my system and I could see that weewx memory usage was a classic sawtooth, starting from a some value that I cannot remember now (this was well over a year ago) and gradually stepping up to a little under the 512 MB my (then) RPi had. I didn't have time to deal with the root cause (nor did I know how to find the root cause) so I lived with the issue. Some fairly frequent tweaking of my system tended to mask the issue as weewx was being restarted fairly regularly.
Move on a while and I replaced my RPi with an Odroid U3, the issue remained but at the Odroid has 2GB ram the time between lockups was greater. I now recorded some data; ram usage was increasing around 70-80 MB per day and I would generally get around 4 weeks between lockups. The ram usage would increase straight away after a weewx start and it would jump by around 3MB every hour or so. I upgraded from Debian Wheezy to Jessie and at first the issue disappered (I think) but it came back soon after. I had seen some posts about other users having a memory leak with PIL or pillow so I made sure it was up to date on my system. As I use Debian quite often the packages are some what behind the latest release so I forced Debian to use the latest package. Still no change. Given the number of skins and SLE I run I suspected that one of my skins or associated SLE was the source of the problem.
Step forward again to a couple of months back and I found some time to pare my system right back and gradually add 1 skin at a time. I eventually found the issue was caused by my Standard skin. My Standard skin is a modified version of the stock weewx Standard skin; I have moved some of the elements on the pages, have dropped the standard weewx plots in favour of Highcharts based plots, have added forecast output from the forecast extension as well as adding an 'alltime' page. So there were some substantial changes. I eventually tracked the issue down to a file that I was generating every hour using the forecast extension. Running the forecast extension with a 10 day WU forecast chews up a bit of time generating the resulting HTML. I use this forecast info on both my Saratoga PHP based site and my weewx site; they are somewhat different so for the Saratoga site I generate the forecast PHP page in its entirety and for the weewx page I generate a file forecast.incl that contains the forecast HTML. I generate these pages each hour considering the time to process and since the forecast info is slow changing. The forecast.incl file is used as an include in the weewx index.html.tmpl template that is generated every archive period. A convoluted arrangement that could no doubt be simplified. In any case, I found that if I stop generating the forecast.incl file the system memory was stable. So I restored everything else and left forecast.incl disabled and the system has been running fine since September, albeit without an updated forecast on my weewx page.
Yesterday I decided to sit down and nut out the issue. The template I use to generate forecast.incl is near identical to the stock forecast extension template forecast_table.inc.tmpl (I have added a couple of comments and dropped the verbatim CSS code from the stock template). The supporting SLE I use with this template is identical to the stock forecast extension SLE. So I replaced my forecast.incl.tmpl code with that from the forecast extension report, now the files were identical. The memory leak was evident immediately and was now around 3MB per archive period (rather than per hour) since I was generating this file every archive period. This was crazy I thought, I am using identical template and SLE code. I then noticed that the forecast extension generates a file forecast_table.inc but I was generating forecast.incl. So on the off chance I changed my template name from forecast.incl.tmpl to forecast_table.inc.tmpl. Memory leak disappeared. I changed to forecast.inc.tmpl, still no memory leak. I had to go but left it overnight, still no memory leak. This morning I changed to forecast.incl.tmpl again and the memory leak returned immediately. back to forecast.inc.tmpl and memory leak disappeared.
This seems bizarre, we have templates like index.html.tmpl that generate fine so the 4 letter extension seems to not be the issue per se. forecast.incl.tmpl generates fine but with a memory leak. I have been unable to get onto the cheetah site today to follow this up and see if cheetah has some sensitivity to .incl. I don't see how the forecast extension can be at fault, its just a template name that is being changed. I don't know if I have missed something obvious or not, but in any case long standing memory leak has disappeared and hopefully there will be no more scheduled restarts.
Gary
Thomas Keffer
Nov 6, 2016, 8:04:07 AM11/6/16
to weewx-development
I certainly don't have any explanation either. I don't see why the template name would make any difference.
Have you been able to replicate the problem on another machine?
-tk
gjr80
Nov 6, 2016, 8:11:18 AM11/6/16
to weewx-development
Certainly had the same symptoms on a RPi to start with, the leak was the same rate and included the '.incl' template so I have no reason to believe it was anything else. Now that I know what causes the problem I might see if I can reproduce it on my spare RPi.
Thomas Keffer
Nov 6, 2016, 8:15:07 AM11/6/16
to weewx-development
It would also be worth paring down the forecast SLE until you find what part of it is causing the leak.
-tk
gjr80
Nov 6, 2016, 8:22:55 AM11/6/16
to weewx-development
Funny how the SLE has not shown any issues elsewhere. Will be good to try to get to the bottom of it - too many odd things going on.
Reply all
Reply to author
Forward
0 new messages