Memory leak in StdReport?

[Lucas Heijst]

Tom,

Currently running weewx 4.0.0b17 and analysing the memory leak in my pi31 system.

On the system runs two weewx instances (mben and tfrc) and a 5-minute webcam cronjob.

The memory leak of my webcam cronjob task was caused by creating variables like:

WEBCAM_ID=3

CAMERA=picamera$WEBCAM_ID

DATETIME=$(date +"%Y-%m-%d %H:%M:%S")

EPOCH=$(date +"%s")

...

The used memory is freed by:

unset WEBCAM_ID

unset CAMERA

unset DATETIME

unset EPOCH

...

The memory leak of the weewx instances mben and tfrc are caused by the weewx reporting tasks, see data below:

-----

used  time

 mem

----- -----

73348 10:30

95504 11:07

48576 11:38 no webcam, no mben, no tfrc (no increase of memory)

56136 11:41 mben without modbus read and modbus service

57384 11:43

57876 11:44

58360 11:45

64836 11:47 used memory increased with 6476 due to mben31 report

64572 11:48

64560 11:49

64548 11:50

66240 11:51 used memory increased with 1692 due to mben31 report

55808 11:53 mben without mben31 report

56092 11:54

55828 11:55

55808 11:56

56000 11:57

56000 11:58

56516 11:59

56772 12:00

56164 12:02 mben now with modbus service (fully functional; without reporting)

56436 12:03

55920 12:04

76056 12:09 mben without mben31 report; tfrc started without tfrc31 report

76388 12:10

77084 12:11

77248 12:12

77432 12:13

77312 12:14

76908 12:15

76976 12:16

77268 12:21

77248 12:24 hardly any increase of used memory the last 15 minutes

-----

When all reports are disabled, the memory leak on my system is practically zero.

Attached the skin settings of mben31.

Any clue how reduce this memory leak?

Luc

[Thomas Keffer]

I'm not following. I thought you said you could get rid of the leak by doing the unsets. If so, is there still a problem?

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/d91724fe-da46-4b4e-9913-59f55aa91657%40googlegroups.com.

[Thomas Keffer]

By the way, in my experience, these leaks are caused by underlying "C" libraries or, occasionally, by failure of PIL to release graphics resource (such as font handles).

[Lucas Heijst]

Tom,

I have experienced two types of memory leak.

1. In cronjobs when defining variables without unset them afterwards.

2. With weewx reports (mben with 84+ generated graphs each run).

Luc

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-de...@googlegroups.com.

[Thomas Keffer]

I'm sorry, Luc, not trying to be difficult, but what is your question? I can't possibly debug your memory leak, especially with the complicated set of drivers and skins you are running.

WeeWX itself does not have any memory leaks that I'm aware of. It is 100% Python, and Python uses garbage collection. If there are circular references, it can take a while for the interpreter to find unused and recycle memory, but WeeWX does not use any circular references.

Here's an example of a station that has been running continuously for well over a year. Present memory use is VIRT=75060 RES=37736 SHR=7144. So, very long runtimes are possible without leaks. 

Net-net, if there are leaks, I am pretty confident they are in the underlying C libraries.

-tk

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/b5c12aeb-c8e1-489e-b457-04d2f8783ca0%40googlegroups.com.

[Lucas Heijst]

Tom,

Due to memory grow my system crashed after 9-10 days. So I wrote here that I would implement a cron job to restart my system after 5 days.

Vice Skahan convinced me to trace down the cause of the memory leaks and that I did.

No more, no less.

Luc

On Sunday, 29 March 2020 20:54:56 UTC-3, Tom Keffer wrote:

I'm sorry, Luc, not trying to be difficult, but what is your question? I can't possibly debug your memory leak, especially with the complicated set of drivers and skins you are running.

WeeWX itself does not have any memory leaks that I'm aware of. It is 100% Python, and Python uses garbage collection. If there are circular references, it can take a while for the interpreter to find unused and recycle memory, but WeeWX does not use any circular references.

Here's an example of a station that has been running continuously for well over a year. Present memory use is VIRT=75060 RES=37736 SHR=7144. So, very long runtimes are possible without leaks. 

Net-net, if there are leaks, I am pretty confident they are in the underlying C libraries.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/b5c12aeb-c8e1-489e-b457-04d2f8783ca0%40googlegroups.com.

[Lucas Heijst]

Tom,

I I forgot to say these same set of programs and webcam cronjob were running since a year now with weewx 3.9.1, raspian stretch and python2.

Now they are running weewx 4.00b16-17, raspbian stretch and python2 or python3 with the memory problems mentioned.

Luc

[Thomas Keffer]

1. You are getting memory leaks in 4.0.0 running both Python 2 and Python 3?

2. The biggest architectural differences between v3.9 and v4.0 is in how derived types are handled (V4 uses the new extensible type system) and in how strings are handled (v4 uses unicode), The former is pure Python. The latter is not, but I've got to assume that the Python Software Foundation has thoroughly debugged that code.

3. Cheetah uses an underlying C library ("NameMapper"). It's possible they screwed up something in their Python 3 port.

That's all I can think of off the top of my head.

-tk

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/f0a096da-16c4-48cc-86e6-13e9f18fba96%40googlegroups.com.

[gjr80]

How about disabling all image generation in your skins since PIL has been a frequent leaker in the past? Granted it doesn’t explain why 3.9.x/python2 is fine but 4.0.0/python2 leaks but you have identified StdReport as having leaks. So surely you need to focus on that and given PILs history.... (plus it is an easy check to make).

If the leaks stop that would tend to clear extensible types and point the finger at PIL.

My limited experience with a couple of ‘rounds‘ of leaks over the years is that they come and go, I presume as some underlying library is updated. That could explain the difference in behaviour between 3.9.x/python2 and 4.0.0/python2.

Gary

[Lucas Heijst]

Gary,

The system with weewx 3.9.1 runned with raspbian stretch of March 2019. I did not upgrade raspbian stretch until recently when I started all over with fresh install of the latest version of raspbian stretch. The update and upgrade renewed 64 modules. 

The tfrec package won’t compile without errors in raspian buster, thats why I stick to raspbian stretch.

Ps. I run now both programs with a single graph image each.

Luc

[gjr80]

So maybe one of those updated modules is the problem. Running with one plot still calls PIL so if PIL is the leak it will still show. As Vince said you really need to work systematically to find the culprit. PIL seemed to be low hanging fruit to me.

Gary

[rich T]

Luc

Don't know if this will be the solution to your issue: https://groups.google.com/forum/#!searchin/weewx-user/memory$20leak$20pil%7Csort:date/weewx-user/qeVSNzy7Zck/k-tAssLwAQAJ

Rich

[Glenn McKechnie]

On 30/03/2020, Lucas Heijst <ljm.h...@gmail.com> wrote:
[...]

> 2. With weewx reports (mben with 84+ generated graphs each run).
>
> Luc

I have a fairly complex set of graphs ( multiple plots, large size,
anti-alias turned on) and I have a persistent and steady climb until
the OOM killer takes over. If I simplify the graphing then it holds up
very well, but there's no fun in that!

Rather than a Cronjob to force a set restart, I've turned off StdReport
report_services = weewx.engine.StdPrint # , weewx.engine.StdReport
and then once engine.py has finished its post_loop(self, _event):
function, it writes a log message to loginfo (rsyslog).
syslog picks up that message and then fires off an instance of wee_reports.

The advantage of this is that...
1. The Reports are called at the correct point in the weewx cycle.
2. Once wee_reports finishes it closes completely (assuming another
problem isn't holding it open) and completely frees the memory it's
used.
3. Weewx stays up and I don't waste anymore time looking for a memory
leak that I can do nothing about, or is way above my skill set. I've
chased PIL versions and run memory profilers but never had a suitable
Eureka moment.

The disadvantage is that...
1. weeWX uptime is reported as zero
2. engine.py needs a one line patch (a suitable loginfo message)
3. rsyslog is easy but a set procedure needs following. One that's
simple in hindsight.

now, in a week, weewx might use 2 Meg. It basically flatlines with this action.

If you want I'll track down the scripts, post them and outline the full logic.

--


Cheers
Glenn

rorpi - read only raspberry pi & various weewx addons
https://github.com/glennmckechnie

[Lucas Heijst]

Gary,

You are right, one graph was enough to trigger the memory grow! The strange thing tonight was an huge increase in memory for about one hour (200kB) followed by the ‘normal’ grow without any clue what happened in de syslog.

Luc

[Lucas Heijst]

Rich,

Two things. My system is using PIL 4.0.0. which seem to cause problems.

And..

Recently I changed in all my weewx skin files the path of the font files. Now they point to /home/weewx/bin/user/myfiles/fonts.

They used to point to another directory where I copied my ttf fonts.

Maybe PIL doesn’t like this change in the font path.

Luc

[Lucas Heijst]

Glen,

I like the method you use by firing the report in another way but I am not keen in modifying engine.py (each time with a new version of weewx).

I could put the report calls in a cron job task, but then they are executed even when weewx is not running.

Let me think about it. The timing when the report is started is not a problem I expect. I have runned many times the report by hand while the weewx instances were running somewhere in their cycli.

Luc

On Sunday, 29 March 2020 23:32:44 UTC-3, Glenn McKechnie wrote:

[Glenn McKechnie]

Luc,

(Bugger, and this one to the list)
NB:
The script lines wordwrap with my mailer, some may need.. straightening?

On 30/03/2020, Lucas Heijst <ljm.h...@gmail.com> wrote:

> Glen,
>
> I like the method you use by firing the report in another way but I am not
> keen in modifying engine.py (each time with a new version of weewx).
> I could put the report calls in a cron job task, but then they are executed
>
> even when weewx is not running.
> Let me think about it. The timing when the report is started is not a
> problem I expect. I have runned many times the report by hand while the
> weewx instances were running somewhere in their cycli.

The following is a description of bypassing the internal weewx report
generation, and replacing it with a seperate instance.
It looks complicated but in actual fact, it's pretty simple and runs
flawlessly here.

To recap...
The objective is to remove any memory problems that occur during the
report cycle (whatever the cause) by ensuring that the report cycle is
seperated from the main weewx instance. This allows it to release all
its memory upon completion. It starts afresh each time it's called.
The reason for doing it this way is to ensure that the reports are
generated at the same time weewx would have done them, the theory
being that there is less chance of interference/breakage of weewx.


It consists of ...
1. patching engine.py, and tweaking weewx.conf
2. adding a call to rsyslog
3. creating a calling script for rsyslog to action.
4. creating a wee_report calling script, that gets out of the way of rsyslog.

The following expands on those 4 points.

1.
$ diff -Naur /home/weewx/bin/weewx/engine.py.org
/home/weewx/bin/weewx/engine.py
--- /home/weewx/bin/weewx/engine.py.org 2020-03-30 17:49:40.760732824 +1100
+++ /home/weewx/bin/weewx/engine.py 2020-01-26 03:47:05.588871567 +1100
@@ -602,6 +602,11 @@
raise ValueError("Unknown station record generation
value %s" % self.record_generation)
self.old_accumulator = None

+ # Glenn McKechnie - flag to run wee_reports
+ # also turn off StdReport
+ # report_services = weewx.engine.StdPrint # , weewx.engine.StdReport
+ log.info("engine: weewx loop has finished")
+
# Set the time of the next break loop:
self.end_archive_delay_ts = self.end_archive_period_ts +
self.archive_delay

Those 5 lines that start with a "+" are the only change that's made to
engine.py. It doesn't break anything. If it's not there the following
won't work and no independent report run will be performed.
It's only one-line that actually matters, the comments are there to
aid my memory when I need to, very occasionally, find and revisit it.
If it is there and you don't use it, your logs suffer the indignity of
a pointless message.
If it is there and weewx is not running, then wee_reports will not be
called by the scripts, as distinct from a CRON entry which would only
stay silent if it could detect a stopped weewx.

The other change is to weewx.conf where the report_services line has
the weewx.engine.StdReport entry commented out.

report_services = weewx.engine.StdPrint # , weewx.engine.StdReport

2.
I've redirected my syslog output to a weewx specific log file so I use
the rsyslog.d/weewx.conf example file. The next step is to add the
following additional lines to the top of that rsyslog.d conf file.
$ /etc/rsyslog.d/weewx.conf
#
# This will break rsyslog if the called command takes too long to complete.
# Therefore, the script that this points to has to be simplistic. In fact
# this points to a one-line call to the real script before it
backgrounds/completes.
# This script is mode 0644
# source https://www.slideshare.net/rainergerhards1/writing-rsyslog-p
#
module(load="omprog")
if $rawmsg contains 'weewx loop has finished' then
action(type="omprog"
binary="/home/weewx/bin/user/runreports.sh")
#

3.
The script that is being called consists of six lines, but only one
truly matters and that's the call to run_weereports.sh
$ /home/weewx/bin/user/runreports.sh
#!/bin/sh
# script called from rsyslog.conf or equivalent ... rsyslog.d/weewx.conf
`bash /home/weewx/bin/user/run_weereports.sh &`
exit 0
# This script needs to be executable
# run chmod 0755 /home/weewx/bin/user/runreports.sh

It calls /home/weewx/bin/user/run_weereports.sh and then drops into
the background and thereby returns control to the rsyslog daemon. It
may not look like much but this script is an important intermediate
step. If this was the actual script that invoked wee_reports it would
take too long to complete and you'd find rsyslog would break. (ie:-
you'd start getting log messages to your terminal and nothing to any
of the log files!)

4.
Now, The following is the actual call to wee_reports, run_weereports.sh
This is mine, yours can be whatever you like. I write a PID file and
with that I can decide to run wee_reports every archive , or every
alternate archive interval. It doesn't matter, just so long as it runs
wee_reports once and that you know what's going on.
This has syslog entries, you can comment them out once you've finished
with them.

$ /home/weewx/bin/user/run_weereports.sh
#!/bin/bash
# script called from /home/weewx/bin/user/runreports.sh
pid_file='/var/run/runreports.pid'
tymestamp=$(date +%d%m%Y-%H:%M)

if [ -f $pid_file ]
then
`rm -f $pid_file`
logger --id --tag weewx_weatherpi "Skipped this weewx report by
removing $pid_file"
else
$(/home/weewx/bin/wee_reports --config=/home/weewx/weatherpi.conf &
echo $! > $pid_file)
logger --id --tag weewx_weatherpi "Ran wee_reports manually,
created $pid_file"
# comment the following to skip alternate wee_report runs
# `rm -f $pid_file`
fi
# This script can be made executable if you wish
# run chmod 0755 /home/weewx/bin/user/run_weereports.sh

exit 0

Putting it all together...

Change anything in the scripts that fits your setup. My weatherpi.conf
is probably the equivalent of your weewx.conf. My directories may not
match yours. Tweak to suit.

I'd suggest starting with the last script, run it and make sure it's
doing what you want. (running wee_reports)

Next, create the second (calling ) script and make sure that it does
what you expect it to. (running the above script, that then runs
wee_reports)

Then I'd add the rsyslog modifications and restart rsyslog. Nothing
spectacular should happen, except that it restarts and stays running.
Ah... If you don't have seperate weewx logging, then that addition I
mention (the 12 lines including module(load="omprog") ) goes into
your /etc/rsyslog.conf file after the #### RULES #### header (It's
been a while since I did it that way, I'm pretty sure that was the
spot.)

The next step is to test it by running the command...
logger "weewx loop has finished"
and that simulates what the (not yet) modified engine.py will utter.
When that message hits syslog wee_reports should fire up and do its
thing. (ie:- the above calling script runs and that executes the
script that actually fires up wee_reports)

Up to this point, the dominoes have been placed but only a manual step
can set them in motion.
So, if all the above works as intended then modify engine.py, then
weewx.conf and you've set the trigger. Restart weewx and it should
fire. Review your reports. Watch your logs, there should be a series
of wee_report entries occur at the appropriate point (ie: same as
before) in the weewx cycle.

Run it for a while and check your memory usage and it should be flat
lining. That's if StdReports was the cause of your memory leak (which
is probably in the version of PIL or other C lib as Tom notes.)

If you still get a message about the report cycle being blocked by a
previous run (I was fairly certain that message still occured but as I
think about it , doubt enters. Hmmm.) then there is a serious problem
within the actual report run. I've had something similar with mysql
hitting the roof due to a ... Okay, I should go back and revisit that
issue. :-(

That's it. Hopefully I've missed nothing, and that it's of use.

> On Sunday, 29 March 2020 23:32:44 UTC-3, Glenn McKechnie wrote:
>>

[Lucas Heijst]

Glenn,

I have chosen for another approach.

The MyReport service (see below) is currently embedded in tfrc.py and modbusenergy.py.

Service MyReport opens a subproces which spawns the report command.

Output on syslog:

=====

Mar 30 10:25:01 pi31b CRON[1395]: (root) CMD (/home/weewx/util/init.d/webcam.debian 1> /var/log/webcam.txt 2> /var/log/webcam.err)

Mar 30 10:25:04 pi31b mben[595] INFO user.modbusenergy: huidige P1 1209 P2 147 P3 1209 PTot 2565 ETotEbsBer 68133.9

Mar 30 10:25:05 pi31b mben[595] INFO weewx.manager: Added record 2020-03-30 10:25:00 -03 (1585574700) to database 'weewx_mben31'

Mar 30 10:25:05 pi31b mben[595] INFO weewx.manager: Added record 2020-03-30 10:25:00 -03 (1585574700) to daily summary in 'weewx_mben31'

Mar 30 10:25:05 pi31b mben[595] INFO user.modbusenergy: Start MyReport for /home/weewx/weewx_mben.conf

Mar 30 10:25:06 pi31b wee_reports[1422] INFO weewx.engine: StdConvert target unit is 0x10

Mar 30 10:25:06 pi31b wee_reports[1422] INFO weewx.wxservices: The following values will be calculated: pressure=prefer_hardware, barometer=prefer_hardware, altimeter=prefer_hardware, windchill=prefer_hardware, heatindex=prefer_hardware, dewpoint=prefer_hardware, inDewpoint=prefer_hardware, rainRate=prefer_hardware

Mar 30 10:25:06 pi31b wee_reports[1422] INFO weewx.wxservices: The following algorithms will be used for calculations: altimeter=aaASOS, maxSolarRad=RS

Mar 30 10:25:06 pi31b wee_reports[1422] INFO user.modbusenergy: service version is 0.12

Mar 30 10:25:06 pi31b wee_reports[1422] INFO user.modbusenergy: mben_service: debug_mben = 1

Mar 30 10:25:06 pi31b wee_reports[1422] INFO user.modbusenergy: mben_service: binding=mbendevices31_binding

Mar 30 10:25:06 pi31b wee_reports[1422] INFO weewx.engine: Archive will use data binding wx_binding

Mar 30 10:25:06 pi31b wee_reports[1422] INFO weewx.engine: Record generation will be attempted in 'hardware'

Mar 30 10:25:06 pi31b wee_reports[1422] INFO weewx.engine: Using archive interval of 60 seconds (specified by hardware)

Mar 30 10:25:06 pi31b wee_reports[1422] INFO user.modbusenergy: MyReport service version is 0.12

Mar 30 10:25:07 pi31b wee_reports[1422] INFO weewx.cheetahgenerator: Generated 1 files for report mben31 in 0.63 seconds

Mar 30 10:25:18 pi31b tfrc[542] INFO weewx.manager: Added record 2020-03-30 10:25:18 -03 (1585574718) to database 'cmon31'

Mar 30 10:25:18 pi31b tfrc[542] INFO weewx.manager: Added record 2020-03-30 10:25:18 -03 (1585574718) to daily summary in 'cmon31'

Mar 30 10:25:19 pi31b tfrc[542] INFO weewx.manager: Added record 2020-03-30 10:25:00 -03 (1585574700) to database 'weewx_tfrc31'

Mar 30 10:25:19 pi31b tfrc[542] INFO weewx.manager: Added record 2020-03-30 10:25:00 -03 (1585574700) to daily summary in 'weewx_tfrc31'

Mar 30 10:25:19 pi31b tfrc[542] INFO user.tfrc: Start MyReport for /home/weewx/weewx_tfrc.

Mar 30 10:25:20 pi31b wee_reports[1452] INFO weewx.engine: StdConvert target unit is 0x10

Mar 30 10:25:20 pi31b wee_reports[1452] INFO weewx.wxservices: The following values will be calculated: pressure=prefer_hardware, barometer=prefer_hardware, altimeter=prefer_hardware, windchill=prefer_hardware, heatindex=prefer_hardware, dewpoint=prefer_hardware, inDewpoint=prefer_hardware, rainRate=prefer_hardware

Mar 30 10:25:20 pi31b wee_reports[1452] INFO weewx.wxservices: The following algorithms will be used for calculations: altimeter=aaASOS, maxSolarRad=RS

Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: service version is 0.18

Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: sysinfo: Linux pi31b 4.19.66-v7+ #1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l

Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: CPU variant: 0x0

Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cconfpuinfo: CPU architecture: 7

Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: CPU part: 0xd03

Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: CPU implementer: 0x41

Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: BogoMIPS: 76.80

Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: Revision: a02082

Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: Serial: 00000000496ca922

Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: model name: ARMv7 Processor rev 4 (v7l)

Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: Hardware: BCM2835

Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: CPU revision: 4

Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: processor: 3

Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: Features: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32

Mar 30 10:25:20 pi31b wee_reports[1452] INFO weewx.engine: Archive will use data binding wx_binding

Mar 30 10:25:20 pi31b wee_reports[1452] INFO weewx.engine: Record generation will be attempted in 'hardware'

Mar 30 10:25:20 pi31b wee_reports[1452] INFO weewx.engine: Using archive interval of 300 seconds (specified by hardware)

Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.tfrc: MyReport service version is 0.4

Mar 30 10:25:21 pi31b wee_reports[1452] INFO weewx.cheetahgenerator: Generated 1 files for report tfrc31 in 0.54 seconds

Mar 30 10:25:29 pi31b wee_reports[1452] INFO weewx.imagegenerator: Generated 5 images for report tfrc31 in 7.17 seconds

Mar 30 10:25:29 pi31b wee_reports[1422] INFO weewx.imagegenerator: Generated 84 images for report mben31 in 21.51 seconds

Mar 30 10:25:29 pi31b mben[595] INFO user.modbusenergy: MyReport for /home/weewx/weewx_mben.conf finished

Mar 30 10:25:39 pi31b mben[595] INFO weewx.manager: Added record 2020-03-30 10:25:39 -03 (1585574739) to database 'mbendevices31'

Mar 30 10:25:39 pi31b mben[595] INFO weewx.manager: Added record 2020-03-30 10:25:39 -03 (1585574739) to daily summary in 'mbendevices31'

Mar 30 10:25:55 pi31b wee_reports[1452] INFO weewx.reportengine: ftpgenerator: Ftp'd 2 files in 26.36 seconds

Mar 30 10:25:55 pi31b tfrc[542] INFO user.tfrc: MyReport for /home/weewx/weewx_tfrc.conf finished

Mar 30 10:26:04 pi31b mben[595] INFO user.modbusenergy: huidige P1 178 P2 133 P3 155 PTot 465 ETotEbsBer 68134.0

Mar 30 10:26:04 pi31b mben[595] INFO weewx.manager: Added record 2020-03-30 10:26:00 -03 (1585574760) to database 'weewx_mben31'

Mar 30 10:26:05 pi31b mben[595] INFO weewx.manager: Added record 2020-03-30 10:26:00 -03 (1585574760) to daily summary in 'weewx_mben31'

=====

Luc

Modified files:

In weewx_tfrc.conf:

...

        report_services = weewx.engine.StdPrint, user.tfrc.MyReport

In weewx_mben.conf:

...

        report_services = weewx.engine.StdPrint, user.modbusenergy.MyReport

In tfrc.py and modbusenergy.py:

...

import subprocess

from subprocess import Popen,PIPE

...

DRIVER_NAME = 'TFRC'

DRIVER_VERSION = '0.4'

MY_REPORT_CONF = '/home/weewx/weewx_tfrc.conf' # (for tfrc.py)

or

MY_REPORT_CONF = '/home/weewx/weewx_mben.conf' # (for modbusenergy.py)

...

############################## Service ##############################  

class MyReport(StdService):

    """ MyReport replaces StdReport"""

    def __init__(self, engine, config_dict):

        super(MyReport, self).__init__(engine, config_dict)

        loginf("MyReport service version is %s" % DRIVER_VERSION)

        self.bind(weewx.NEW_ARCHIVE_RECORD, self.handle_new_archive)

    def handle_new_archive(self, event):

        loginf("Start MyReport for %s" % MY_REPORT_CONF)

        #start and process things, then wait

        reportArg = "--config=" + MY_REPORT_CONF

        p = subprocess.Popen(["/home/weewx/bin/wee_reports", reportArg], stdout=PIPE)

        p.communicate() #now wait plus that you can send commands to process

        loginf("MyReport for %s finished" % MY_REPORT_CONF)

############################## Conf Editor ##############################

On Monday, 30 March 2020 08:30:09 UTC-3, Glenn McKechnie wrote:

Luc,

(Bugger, and this one to the list)
NB:
The script lines wordwrap with my mailer, some may need.. straightening?