WEEWX-WD and Weewx 4.2 to 4.4

[tim lambert]

Gang,

Currently running weewx 4.2 (will be upgrading in the next few days to 4.4) on a Raspberry Pi 4 (32-bit OS) with a Rainwise MK-III and IP-100 providing station data -- everything has been working properly for several weeks and is populating a remote MySQL DB.

Once I upgrade weewx, I'd like to incorporate using Steel Gauges and Saratoga Templates, hence my query:  Can I utilize the WEEWX-WD as outlined BitBucket Weewx-WD?

The Steel Gauges and Saratoga Templates will be hosted on a remote, public accessible server -- so I'll be sending the clientraw and testtag files via FTP.

Thanks in Advance!

- Tim

[gjr80]

Hi,

I wouldn't be using WeeWX-WD from bitbucket, it has not been touched for some years now. WeeWX-WD on github is more up to date, the current version is v2.1.2 from November 2020. It should have no problems running on WeeWX v4.x under python 2 or python3. The SteelSeries Weather Gauges have directly supported WeeWX of v2.5.0 (of the SteelSeries Weather Gauges) or for about six years, and there is now a WeeWX SteelSeries Gauges extension that installs the necessary skin to support the SteelSeries Weather Gauges (this is the preferred means of providing the SteelSeries Weather Gauges, WeeWX-WD did support the SteelSeries Weather Gauges via the WD customclientraw.txt but since the advent of direct WeeWX support and the WeeWX extension the SteelSeries Weather Gauges support has been deprecated in WeeWX-WD). You should find the WeeWX extension will work under WeeWX v4.x and python2 or python3.

I would suggest that you upgrade to WeeWX v4.4.0 and get WeeWX running to your satisfaction before installing either WeeWX-WD v2.1.2 or the WeeWX SteelSeries Weather Gauges extension (in whatever order suits). Once you have the SteelSeries Weather Gauges extension running OK you can, if you want, look at installing the Realtime gauge-data extension which will allow the gauges to update on every loop packet rather than on each report cycle (how often they will update will depend on how often your station emits loop packets, for your Rainwise this should be up to every couple of seconds).

Gary

[Tim Lambert]

Thanks Gary for the insight and guidance.

 

 

Sent from Mail for Windows 10

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/gvReyZeqOJQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/6942d0aa-94bf-46e9-bb56-8232a425f95fn%40googlegroups.com.

 

[tim lambert]

Hi Gary,

I'm taking things in baby steps.

Earlier this week I successfully upgraded from weewx 4.2 to weewx 4.4 (which was surprisingly easy with how weewx addresses upgrades).   Everything is stable, so I'm moving onward to the next steps.

I've taken the route to implement WeeWX SteelSeries Weather Gauges extension (not real-time yet) and WeeWX-WD v2.1.2 -- everything installed and I can see the SteelGauge info on my remotely hosted website, as well as see the clientraw.text and other key files for Saratoga Templates on the remotely hosted server.  

However, even though I have the archive_interval in weewx.conf set to 120 (seconds) the FTP of the files generated by weewx is occurring every 300 seconds (5 minutes) -- which I confirmed from reviewing the log files.   How can I get the FTP to execute every 120 seconds?

FYI, weewx.conf for Standard Archive is

[StdArchive]

    archive_interval = 120
    record_generation = hardware
    loop_hilo = True
    data_binding = wx_binding

Should I change the record_generation to software?   Noting the Rainwise IP-100 is inserting records into the MySQL database tables every minute -- which is what I want.

Thanks in Advance for your guidance!

-- Tim

[Tom Keffer]

You didn't say what kind of weather station you are using, but for stations that support hardware logging and if you specify hardware record generation,  then weewx will use the interval specified in your hardware.

If this is the case, then to change use the wee_device utility with the --set-interval option. For example, to change to two minutes:

wee_device --set-interval=2

-tk

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/e8c94cec-3d75-4ac1-93d3-19cfcae8a1b6n%40googlegroups.com.

[Tim]

Tom,

The station is a RainWise MK-III with an IP-100.

Regards,

Tim

Sent from my Hi-Tech Etch-A-Sketch

On Feb 4, 2021, at 14:20, Tom Keffer <tke...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEC0JotC0Rrp0NirvcUx8%3DvL6kzY9W7%3D1SbdU75_ke1i%3DQ%40mail.gmail.com.

[Tom Keffer]

That station (using the ip100 driver) does not support hardware record generation, so WeeWX will fall back on software record generation. If you are still getting 5 minute updates despite setting archive_interval to 120, then one of two things is happening:

1. weewxd is using a different configuration file than you think it is; or
2. You didn't restart weewxd after changing the value.

If neither of those are true, then we will need to see the log. Set debug=1, restart weewx, post the log from startup through the first reporting cycle.

tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/ECA9484A-7708-449E-B0C1-DDB22A0F9DEA%40comcast.net.

[Tim Lambert]

Tom,

 

Attached is the logfile for 2 cycles.

 

Per Line 45 @ 08:56:54 – The 120sec Interval for wwewx.engine is loaded

Per Line 137 –  Line 141 @ 08:58:33 – The Remote MySQL DB is being updated

Per Line 143 @ 08:58:47 – The rpt.engine starts

Per Line 218 @ 09:00:40 – Error  weewx.engine unable to shutdown StdReport thread

Then it appears weew opted to perform a restart by itself

Per Line 333 – Line 337 @ 09:02:32 – The Remote MySQL DB is being updated

Per Line 353 @ 09:02:53 – Start FTP

Per Line 421 @ 09:03:11 – End FTP

Per Line 423 @ 09:03:11 – Run SS

Per Line 471 @ 09:04:22 – Start Reports

Per Line 487 – Line 491 – The Remote MySQL DB is being updated

Per Line 492 @ 09:04:48 – Report Thread is aborted, existing Report Thread is running

Per Line 552 @ 09:06:40– Error  weewx.engine unable to shutdown StdReport thread

Then it appears weew opted to perform a restart by itself

Per Line 653 @ 09:08:05 – Start FTP

Per Line 716 @ 09:08:23 – End FTP

Per Line 723 @ 09:08:32 – Run SS

 

Hope the above analysis helps pinpoint the problem.  

 

I’m concerned about the Error related to the shutdown of the StdReport Thread and the subsequent automatic restart.  My other concern is the aborting the Report Thread, because the existing Report Thread is still running.   Both appear to be impacting the 120 second interval to generate reports and subsequently FTP, as well as the write of the DB every 120 seconds.

 

 

Thanks in Advance for your help.  

 

 

Regards,

 

Tim

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEAK_512pG%3DDPbghbVCnJSJQjk2Q9uEHW12g4WEZkP4CTw%40mail.gmail.com.

 

[Tom Keffer]

The problem is that the machine running weewx doesn't have enough "oomph" to run all the reports you are asking of it in two minutes. For example, here's the results of one reporting cycle:

Feb  5 09:00:23 murph weewx[15605] INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 95.95 seconds
Feb  5 09:00:31 murph weewx[15605] INFO weewx.imagegenerator: Generated 15 images for report SeasonsReport in 7.99 seconds
Feb  5 09:00:32 murph weewx[15605] INFO user.stackedwindrose: Generated 1 images for wdStackedWindRose in 1.11 seconds
Feb  5 09:02:05 murph weewx[15605] INFO weewx.cheetahgenerator: Generated 1 files for report wdTesttags in 93.15 seconds
Feb  5 09:02:53 murph weewx[15605] INFO weewx.cheetahgenerator: Generated 2 files for report wdClientraw in 47.34 seconds
Feb  5 09:03:19 murph weewx[15605] INFO weewx.cheetahgenerator: Generated 2 files for report SteelSeries in 8.65 seconds
Feb  5 09:03:25 murph weewx[15605] INFO weewx.imagegenerator: Generated 11 images for report SteelSeries in 5.64 seconds

That adds up to almost 300 seconds --- way more than the 120 seconds you are giving it. You will either have to go to a much longer reporting cycle (5 minutes may not even do it), or pare down the number of reports, or simplify the reports, or get a more capable server.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/601d8a3f.1c69fb81.c2a39.613cSMTPIN_ADDED_MISSING%40gmr-mx.google.com.

[Tim]

Tom,

Is the issue CPU or RAM? 

I’m running a Pi 4 with 4gb of RAM.

Regards,

Tim

Sent from my Hi-Tech Etch-A-Sketch

On Feb 5, 2021, at 11:11, Tom Keffer <tke...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEB9Fcu3tLZpGVFqhTqoQjXYzLgMBP-R5vjRqBx-jyXVQw%40mail.gmail.com.

[p q]

A PI 4 is pretty powerful. What else is running?

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/135EFAB1-F91F-46F9-9C63-4EAFC60CA7DA%40comcast.net.

[Tim Lambert]

Tom,

 

I have consoleWD from Weather Display and Pure FTPd (which is normally idle) running.

 

I see a few options:

 

1. Move consoleWD to another Pi 4 w/4GB RAM and 32GB SD Card
2. Not use the SeasonsReport and defer any graph generation on the Public (Remote) Server – simply generate the clientraw, testtags and ss files
3. Implement a Pi 4 w/8 GB RAM and 128GB  SD Card

 

I’m going to work thru the options to see what works.

 

Thanks for the insight!

 

Regards,

 

Tim

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAA1SM23_zwkXQSSU_QtQ58vUPcWbRDk%3Di4emxmS5DgHNOx7ohg%40mail.gmail.com.

 

[p q]

Fyi. Im running a modified Standard skin on a Pi3. I recently upgraded from a 2B with no issues, so I think your hardware is sufficient but something is hogging resources. 

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/601da934.1c69fb81.eb367.c8e1SMTPIN_ADDED_MISSING%40gmr-mx.google.com.

[vince]

You have something major happening on that pi of yours. Here's some data from a clean RaspiOS and newly installed weewx running the Simulator driver on a 4GB pi4...

Basically no RAM usage....

root@pi4p1:/home/weewx# free

              total        used        free      shared  buff/cache   available

Mem:        3919788       72836     3516564        8620      330388     3705176

Swap:        102396           0      102396

And no CPU usage....

root@pi4p1:/home/weewx# uptime

 11:50:49 up 40 min,  1 user,  load average: 0.00, 0.00, 0.01

Logs for the first couple cycles - the initial time for a skin is always longer than the normal time after that....

Feb  5 11:30:46 pi4p1 weewx[1839] INFO __main__: Starting up weewx version 4.4.0

Feb  5 11:30:46 pi4p1 weewx[1839] INFO weewx.engine: Clock error is -0.09 seconds (positive is fast)

Feb  5 11:30:46 pi4p1 weewx[1839] INFO weewx.engine: Using binding 'wx_binding' to database 'weewx.sdb'

Feb  5 11:30:46 pi4p1 weewx[1839] INFO weewx.manager: Starting backfill of daily summaries

Feb  5 11:30:46 pi4p1 weewx[1839] INFO weewx.manager: Empty database

Feb  5 11:30:46 pi4p1 weewx[1839] INFO weewx.engine: Starting main packet loop.

Feb  5 11:35:16 pi4p1 weewx[1839] INFO weewx.manager: Added record 2021-02-05 11:35:00 AKST (1612557300) to database 'weewx.sdb'

Feb  5 11:35:16 pi4p1 weewx[1839] INFO weewx.manager: Added record 2021-02-05 11:35:00 AKST (1612557300) to daily summary in 'weewx.sdb'

Feb  5 11:35:18 pi4p1 weewx[1839] INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 1.79 seconds

Feb  5 11:35:25 pi4p1 weewx[1839] INFO weewx.imagegenerator: Generated 60 images for report SeasonsReport in 6.35 seconds

Feb  5 11:35:25 pi4p1 weewx[1839] INFO weewx.reportengine: Copied 5 files to /home/weewx/public_html

Feb  5 11:40:16 pi4p1 weewx[1839] INFO weewx.manager: Added record 2021-02-05 11:40:00 AKST (1612557600) to database 'weewx.sdb'

Feb  5 11:40:16 pi4p1 weewx[1839] INFO weewx.manager: Added record 2021-02-05 11:40:00 AKST (1612557600) to daily summary in 'weewx.sdb'

Feb  5 11:40:17 pi4p1 weewx[1839] INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 0.80 seconds

Feb  5 11:40:18 pi4p1 weewx[1839] INFO weewx.imagegenerator: Generated 15 images for report SeasonsReport in 0.51 seconds

[Tim Lambert]

Vince,

 

Some info from my Pi4:

 

pi@murph:~/consolewdfiles $ free

              total        used        free      shared  buff/cache   available

Mem:        3919788      250800     2881384      229552      787604     3305456

Swap:        102396           0      102396

 

pi@murph:~/consolewdfiles $ uptime

12:58:26 up 15:15,  3 users,  load average: 0.11, 0.19, 0.24

 

pi@murph:~/consolewdfiles $ users

pi pi pi

 

The 3 users are consolewd (from Weather Display which runs at start-up under systemctl) , weewx (which also runs at start-up) and an ssh session

 

Any ideas how to identify the ‘resource hog’?

 

Regards,

 

Tim

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/7a3c3f02-37e0-4ba1-9a45-d45797eee986n%40googlegroups.com.

 

[Tom Keffer]

Your machine should be more than capable of running weewx.

However, your weak spot is probably MySQL, particularly if you are running it on an SD card. MySQL is much slower than sqlite.

See the wiki article SQLite vs MySQL.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/7a3c3f02-37e0-4ba1-9a45-d45797eee986n%40googlegroups.com.

[p q]

I'm no Linux expect but I'd start with top. 

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEAKnx3HS_8g-OBz%2BoAkNu9ky-Z6e1xp9DD8HNFGwmxeZg%40mail.gmail.com.

[Tim Lambert]

Tom,

 

MySQL is on a remote/hosted server – it is not running on the Pi.   I’ve provided the connection parameters in the weewx.conf

1.       weewxd is using a different configuration file than you think it is; or

2.       You didn't restart weewxd after changing the value.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEAKnx3HS_8g-OBz%2BoAkNu9ky-Z6e1xp9DD8HNFGwmxeZg%40mail.gmail.com.

 

[Tim Lambert]

Thanks Team!

 

I have another Pi4 w/4GB of RAM sitting idle – I’m going to install weewx on it, and uninstall on the other box.   

 

This will isolate weewx to run by itself (with exception of Pure FTPd and Apache Server)

 

 

 

Regards,

 

Tim

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/601db9f5.1c69fb81.d7df7.95d2SMTPIN_ADDED_MISSING%40gmr-mx.google.com.

 

[vince]

You could run that little amount of stuff on a pi zero. 

You're not out of RAM.  You're not out of CPU.  I'm guessing you're waiting for I/O unless your SD card is throwing a zillion errors and is close to failing.   Check that by running 'dmesg' and looking at your syslogs.  For how to debug https://bencane.com/2012/08/06/troubleshooting-high-io-wait-in-linux/ is one resource to look at.

[Tom Keffer]

If it's on a remote server, it can still be a problem. 

A tag like $day.outTemp.max involves a simple database lookup. A typical skin can have hundreds of such tags, resulting in hundreds of queries. SQLite excels at these kinds of frequent small queries, MySQL does not.

Try running with sqlite and see if it makes a difference. It's just a few options in your configuration file to give it a go.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/601db9f5.1c69fb81.d7df7.95d2SMTPIN_ADDED_MISSING%40gmr-mx.google.com.

[gjr80]

I’m not sure MySQL is the issue per se, rather I suspect the remote connection. The main WeeWX-WD template testtags.php.tmpl has some 1000 odd $day, $month, $year tags; of the other four clientraw templates there is some 500 $day, $month, $year tags  And we haven’t even gotten to the search list extensions of which there are numerous and some quite database intensive. I run WeeWX-WD on a RPi 3B+ along with about 10 other skins and and my total report generation time is just over 45 seconds. That is using MySQL locally and running a second instance of WeeWX-WD to support a GW1000.

I recall in the syslog extract extract there was a lost connection to the MySQL server, seems to me to point to some issues accessing the remote database server.

Gary

[Tom Keffer]

Good points. Thanks, Gary.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/3b181fa0-029d-4c57-8649-4a49b3413a7fn%40googlegroups.com.

[gjr80]

Sorry, autocorrect problem. I should say that I run a second instance of WeeWX not WeeWX-WD.

Gary

[Tim Lambert]

Gary,

 

I think you hit on the root cause.

 

I’m standing up another Pi with an external HD attached to support the DB – the idea is to keep the DB traffic off the SD card.

 

Regards,

 

Tim

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/3b181fa0-029d-4c57-8649-4a49b3413a7fn%40googlegroups.com.

 

[vince]

Just for kicks, run "top", hit '1' to see all the cpu cores individually, and see if you see anything with a percentage other than 0.0 next to "ni" in the output.   That'll help us see if you have things waiting for i/o.

You might need to let it run for a little to watch things start/stop and change states on the various cpus, but concentrate on the 'ni' item in each line there.   To get out of top, hit 'q' for quit.

My pi3 reports the following:

top - 18:04:58 up 9 days, 30 min,  3 users,  load average: 0.12, 0.15, 0.12

Tasks: 137 total,   2 running, 135 sleeping,   0 stopped,   0 zombie

%Cpu0  :  4.0 us,  0.3 sy,  0.0 ni, 95.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu1  :  1.0 us,  0.0 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu2  :  1.7 us,  0.0 sy,  0.0 ni, 98.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu3  :  0.3 us,  1.0 sy,  0.0 ni, 98.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

MiB Mem :    926.1 total,     30.5 free,    349.4 used,    546.2 buff/cache

MiB Swap:    100.0 total,     98.5 free,      1.5 used.    443.5 avail Mem

[tim lambert]

Thanks everyone for your input...  You have all pointed me to what is beginning to look like the root cause -- using a remote MySQL server.

So far, I've stood up another Pi 4 to be dedicated for weewx (for now).  A basic installation using the Seasons skin -- reports/graphs are generating in roughly 1.2 seconds.  FTP of the public_html is taking 7 seconds.  

Tomorrow, I'll stand up Maria DB (Raspberry version of MySQL) on the same Pi, with the DB being on a 1TB attached drive.   Once Maria DB is installed and configured, I'll add in phpMyAdmin (to help with quick DB Table analysis).   Next will be to install WD Extensions and SS Gauges, which will utilize the DB on the attached drive. 

Hoping for a marked performance improvement -- I'll share my learnings.

-- Tim

[tim lambert]

Team,

A quick update...

I've completed standing up WeeWx 4.4 on a dedicated (for now) Pi 4, with the IP-100 extension using the Seasons Skins and FTP the Seasons reports/graphs to a remote public server.  The performance stats are:

Seasons Reports == 1.21 seconds

Season Images == 0.54 seconds

FTP == 6.89 seconds

Total Processing Time == 8.64 seconds

Next I added in writing to to a MariaDB on a 1TB hard drive connected directly to the Pi4.   The aforementioned performance stats did not change.

Next, I implemented the WD Extension, using the local MariaDB to write the databases suggested by WD Extensions.  The performance stats:

Seasons Reports == 1.21 seconds

Seasons Images == 0.60 seconds

WD Files (testtags, windrose & clientraw files) == 1.13 seconds

FTP == 8.26 seconds

Total Processing Time == 11.20 seconds

Finally, I added in the Steel Gauges, which the average performance stats:

Seasons Reports == 2.42 seconds

Seasons Images == 0.63 seconds

WD Files (testtags, windrose & clientraw files) == 4.27 seconds

Steel Gauges == 0.74 seconds

FTP == 16.02 seconds

Total Processing Time == 24.08 seconds

Note:  the only other delta between the original configuration and the new configuration, besides moving the MySQL/MariaDB from a remote/hosted server to a local server with an HD -- the original configuration also had consoleWD from Weather Display.

Conclusion --  the data exchange between weeWX and the remote/hosted server was extremely intensive and thus slowing down the reporting process (which Gary and Tom both thought was the root cause).

Thanks for helping me sleuth this one -- hopefully others will benefit from the errors in my architecture.

-- Tim