memory usage

[Rich Bell]

I'm running on: raspbian stretch, python 2.7.13, and WeeWX 3.9.1.

I was noticing a steady increase in memory usage. I narrowed it down to one service I had installed. This service binds to the loop event and makes over 10 calls to getSql. I updated the service to open and close the connection on each loop event. This seems to have stopped the memory increase.

Next, I wrote a small program that does the same steps. Create a connection, get a cursor, execute a select, fetchone, close the cursor. But this isn't exibiting the memory increase.

Next I wrote a wrapper class for the Connection class and subclassed the Cursor class, like WeeWX does. Running with these, I am again seeing the memory increase.

I don't see how the 'wrapper' classes should matter... Granted at 10+ calls per loop, this is aproximately 240 calls a minute - so it would add up fast; but I would think others would be seeing something too. I did see this thread about the forecast service, https://groups.google.com/forum/#!topic/weewx-user/H4GVpoI5l70/discussion It looks to make a lot of calls to getSql, but binds to archive, so it would happen a lot slower. Not sure if there was any resolution.

I've attached my test program. It can call the sqlite classes directly or use the 'wrapper' classes I stole from WeeWx. I may be all wet, but I'm seeing something strange in my environment... I'll keep digging.

-rich

 

[Leon Shaner]

Rich,

You're not the only one seeing memory leaks, but so far yours is among the more complete reports, helping to narrow it down.  Thanks for this!

I know of several of the extended development team who have been trying to reproduce and instrument to zero in on it.  I am sure they will be appreciative of your efforts here.

Regards,

\Leon

--

Leon Shaner :: Dearborn, Michigan (iPad)

On Nov 4, 2019, at 8:53 PM, Rich Bell <bell...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/3753360b-d4c7-44b3-82ca-a8140514b310%40googlegroups.com.

<amemtest.py>

[Thomas Keffer]

I have experienced memory leak issues as well, but only under 32-bit Debian systems.

About 6 months ago, I tried to isolate the problem, and, like you, came to the conclusion that it was in the sqlite3 "wrappers." In fact, I came up with a memory test program that looks very similar to your amemtest.py, except that it uses the weedb API.

Then, mysteriously, the problem went away with a Debian upgrade. Well, almost went away. WeeWX still leaks memory, but only a couple megabytes a week --- something I can live with. The plot below is memory usage over the last month. Complete series here (lower-right plot).

-tk

--

[Rich Bell]

TK,

Thanks for confirming I am not totally crazy. In my case it is aggressive enough to bring the pi down in about a week. When I get a chance I’ll fire up my ubuntu machine and see what I see. I’m curious enough to keep looking, but doubt I will find anything more.

And again, thanks for all your work on WeeWX

-rich

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

[Rich Bell]

I updated the program to make it easier for me to gather data. Probably too much information below.…

Looks like I am still seeing the problem on my Ubuntu 16.04. This run simulates approximately 1 week's worth of calls.

Wrapping the Connection class, subclassing the Cursor class (WeeWX simulation).

Before looping 2419200 times: mem_size: 31.593750 mem_rss: 7.300781 mem_share: 4.250000

After loop:: mem_size: 256.062500 mem_rss: 232.277344 mem_share: 4.878906

Garbage collected 0 objects

After garbage collection:: mem_size: 256.062500 mem_rss: 232.277344 mem_share: 4.878906

The next 6 runs are different implementations of the connect, get cursor, execute statement, fetchone, close cursor loop.

This simulates what WeeWX does. A wrapper around the Connection and a subclass for the Cursor. This is what I originally saw.

Before looping 100000 times: mem_size: 10.472656 mem_rss: 5.980469 mem_share: 3.933594

After loop:: mem_size: 15.535156 mem_rss: 11.933594 mem_share: 4.742188

Garbage collected 0 objects

After garbage collection:: mem_size: 15.535156 mem_rss: 11.933594 mem_share: 4.742188

Using the sqlite3 module directly.

Before looping 100000 times: mem_size: 10.472656 mem_rss: 5.996094 mem_share: 3.949219

After loop:: mem_size: 10.597656 mem_rss: 6.875000 mem_share: 4.707031

Garbage collected 0 objects

After garbage collection:: mem_size: 10.597656 mem_rss: 6.875000 mem_share: 4.707031

Wrapping the Connection class, using sqlite3 Cursor class. Still leaking. So doesn’t appear to be the Cursor subclass.

Before looping 100000 times: mem_size: 10.472656 mem_rss: 6.000000 mem_share: 3.953125

After loop:: mem_size: 15.535156 mem_rss: 11.937500 mem_share: 4.746094

Garbage collected 0 objects

After garbage collection:: mem_size: 15.535156 mem_rss: 11.937500 mem_share: 4.746094

Change the Connection class to subclass Sqlite Connection class and override the cursor method to use a factory to get an instance of the cursor. This looks promising, but we still need a wrapper of the connection to fit into the WeeWX architecture to support multiple database implementations (Sqlite, MySQL, etc)

Before looping 100000 times: mem_size: 10.472656 mem_rss: 5.972656 mem_share: 3.925781

After loop:: mem_size: 10.597656 mem_rss: 6.906250 mem_share: 4.738281

Garbage collected 0 objects

After garbage collection:: mem_size: 10.597656 mem_rss: 6.906250 mem_share: 4.738281

Create a subclass of the Sqlite Connection class that overrides the cursor method. Have the Connection wrapper instantiate this instead of the SQlite Connection class. This would fit into WeeWx quite easily.

Before looping 100000 times: mem_size: 10.472656 mem_rss: 5.996094 mem_share: 3.949219

After loop:: mem_size: 10.597656 mem_rss: 6.937500 mem_share: 4.769531

Garbage collected 0 objects

After garbage collection:: mem_size: 10.597656 mem_rss: 6.937500 mem_share: 4.769531

Subclass the SQLite Connection class and also act as a wrapper. This would also fit into WeeWx using multiple inheritance in the Connection class.

Before looping 100000 times: mem_size: 10.472656 mem_rss: 5.984375 mem_share: 3.937500

After loop:: mem_size: 10.597656 mem_rss: 6.914062 mem_share: 4.746094

Garbage collected 0 objects

After garbage collection:: mem_size: 10.597656 mem_rss: 6.914062 mem_share: 4.746094

The shim class to provide a cursor via a factory seems the best to me. The next three are just upping the loop size to see if anything shows up. Looks like a loop of 100,000 is reasonable to show possible issues.

Before looping 100000 times: mem_size: 10.472656 mem_rss: 5.503906 mem_share: 3.601562

After loop:: mem_size: 10.597656 mem_rss: 6.878906 mem_share: 4.710938

Garbage collected 0 objects

After garbage collection:: mem_size: 10.597656 mem_rss: 6.878906 mem_share: 4.710938

Before looping 200000 times: mem_size: 10.472656 mem_rss: 6.003906 mem_share: 3.957031

After loop:: mem_size: 10.597656 mem_rss: 6.949219 mem_share: 4.781250

Garbage collected 0 objects

After garbage collection:: mem_size: 10.597656 mem_rss: 6.949219 mem_share: 4.781250

Before looping 2500000 times: mem_size: 10.472656 mem_rss: 5.496094 mem_share: 3.593750

After loop:: mem_size: 10.597656 mem_rss: 6.871094 mem_share: 4.703125

Garbage collected 0 objects

After garbage collection:: mem_size: 10.597656 mem_rss: 6.871094 mem_share: 4.703125

Next up is another program that uses the weedb APIs. The first is 3.9.1 as is. The second is with an updated sqlite.py with the shim class. Each is using a loop of 100,000.

./amemtest2.py

mem_size: 11.859375 mem_rss: 7.367188 mem_share: 4.089844

mem_size: 16.804688 mem_rss: 13.269531 mem_share: 4.832031

Garbage collected 0 objects

mem_size: 16.804688 mem_rss: 13.269531 mem_share: 4.832031

./amemtest2.py

mem_size: 11.609375 mem_rss: 7.410156 mem_share: 4.136719

mem_size: 11.609375 mem_rss: 7.410156 mem_share: 4.136719

Garbage collected 0 objects

mem_size: 11.609375 mem_rss: 7.410156 mem_share: 4.136719

The update to sqlite.py is about 10 LOC. I want to run it in my ‘production’ environment for a few days. If all seems to be behaving, I will submit a pull request.

- Rich

[Tarmo]

Thank you for this!