weewx-multi performance combined with multiple skins on RPi3/4

[f4n...@gmail.com]

How many weewx-instances and skins can be run simultaneously with acceptable performance on a Raspberry Pi 3B and 4?

I wanted to put as many weewx instances as possible on one RPi3 B as I have an usv/ups board (StromPi 2) installed on it. Multiple RPi3/4 with a battery backup each would get a bit expensive.

In its final form it should run at least 4 instances, maybe with the possibility to add one more sensor in the future:

2x SDR instances for 2 different outdoor sensor arrays a / b

1x interceptor driver with FOSHKplugin for sensor array a

1x USB driver for sensor array b which is recieved with the USB-console

The both SDR instances will also record some sensors of the secondary sensor array each, so:

sensor array a primary + sensor array b secondary

sensor array b primary + sensor array a secondary

(to have further redundancy if one of the instances stops or gets maintenance)

I wanted to keep the standard seasons skin, aswell as install the neowx-material skin for each instance. BUT:

After installation of neowx-material it already needs around 38 seconds to generate the report files for one single instance. The ftp uploads already are in 1 to 2 minutes too late for this instance, on both the seasons and neowx-material skin. Tried it with the sdr-instance, the interceptor-instance (with just the seasons skin until now) is less affected by this.

Is my neowx-material config somehow broken or is this a normal beaviour for a 1 minute reporting interval with the neowx-material skin? 

I even wanted to add belchlertown and weather34 in the future but I guess that will be just too much, as it seems they will consume even more resources, with the live reporting possiblities?

Is the performance primarily cpu or memory (sd card) dependable?

[f4n...@gmail.com]

With this skin enabled, cpu usage stays for around 50 out of 60 seconds at 100%. It also skips some reports as the old one isnt finished (found in log).

If i disable it (removing from weewx conf) cpu mostly stays at 70% and peaks to 100 for around 10 to 20 seconds. RAM usage is around 350 MB. 

Is there a way to only let it run let's say every 10 minutes, while letting the seasons skin run every minute? Skin dependent reporting interval possible?

[f4n...@gmail.com]

Or could I send the live data from one raspberry to another (or any other machine with more performance reserves) to another weewx installation with own database, which then does the reports for the heavier skins? Something like a realtime.txt or IP port from one weewx machine to another? I know I can do it partially with FOSHKplugin for ecowitt-client data, but how could i forward weewx data from SDR and USB?

[f4n...@gmail.com]

I just found out rtl_433 is the cause of the 70% nonstop cpu usage. I optimized the rtl_433 command:

cmd = rtl_433 -f 868.3M -M time:utc -M protocol -M level -F json -Y classic -Y autolevel -Y squelch -s 250k -R 78 -R 32

(added squelch, autolevel and reduced bandwith to 250k).

CPU usage down to 15%, let's see how this goes. Maybe I will try out whether neowx-material will still occupy the cpu for so long.

[f4n...@gmail.com]

Still takes about 38 seconds, for the neowx-material report... And around 45 seconds full 100% cpu time per minute

[matthew wall]

On Friday, March 4, 2022 at 6:08:22 AM UTC-5 f4n...@gmail.com wrote:

I just found out rtl_433 is the cause of the 70% nonstop cpu usage. I optimized the rtl_433 command:

cmd = rtl_433 -f 868.3M -M time:utc -M protocol -M level -F json -Y classic -Y autolevel -Y squelch -s 250k -R 78 -R 32

(added squelch, autolevel and reduced bandwith to 250k).

CPU usage down to 15%, let's see how this goes. Maybe I will try out whether neowx-material will still occupy the cpu for so long.

thank you for this!  i will add it to the weewx-sdr readme.  do you know how much each of those options affect the performance?  i would guess that the '-R 32' is the biggest one, since that tells rtl_433 to look for only one specific type of data.

as for weewx, the report generation is the biggest load.  i often run multiple (4-5) weewx instances on a low-power machine for data collection, and generate only a single, simple skin for each instance on the data collector.  but i then upload the data to another, more capable machine for aggregation.  i typically use influx for the aggregation, with a grafana front-end.

if you want weewx to generate reports on the aggregation end of things, then you'll have to figure out a way to get the data from the data collector to the aggregator.  i can imagine a few ways of doing this:

a) configure the collector to save to a mysql database, then run the wee_report utility on the aggregator to generate the reports, using the mysql database as the source

b) write a weewx driver that receives data, and a weewx uploader that sends it

c) use the wunderground uploader to send data, and the weewx-interceptor driver to receive it

i am sure there are other options!

m

[f4n...@gmail.com]

First i just tried with -s 250k and the both -R and it went down from 70 to 25%. -Y autolevel -Y squelch then brought it down another 10%

But I'm wondering if this could have impact on the reception quality for sensors which are further away? So if i ever try to recieve a signal from a station i set up let's say 90 to 100m away? 

I was already impressed with SDR as it finds the WH2310 much faster, i mean immediately, after a 2 to 3 minute sensor downtime (don't know if it's the distance or just the sensor low on power..just around 30m away though ) as the console often needs more than 1 hour to find it again.

Is Influx and grafana easy to learn? I'm really not much of a developer, as html, css and lets say the weewx config files already are enough of a hassle for me, if I make some error somewhere :D 

So I'll probably never be able to write such a driver, but it would be a great solution imo, nearly unlimited weewx data replication possibilities. If anyone with some spare time and skills to do it would be open to code it. I'd be happy to help with beta testing.

Option c is a bit too much wunderground dependant and the performance is often lacking when trying to access wunderground data.

Option a: sounds interesting too, so could i save to an additional mysql database on an external device/storage and run wee_reports periodically on that device?

[matthew wall]

On Friday, March 4, 2022 at 9:51:16 AM UTC-5 f4n...@gmail.com wrote:

Is Influx and grafana easy to learn? I'm really not much of a developer, as html, css and lets say the weewx config files already are enough of a hassle for me, if I make some error somewhere :D 

they are pretty easy to install for individual use, but it gets rather complicated if you intend to run them in any public-facing situation.  both grafana and influx have cloud-based services, so you could use the cloud version without having to install anything yourself.

use the weewx-influx extension to send data to influx, then use grafana to browse and display the data. i have attached a screenshot of a basic weather dashboard from a weewx+influx+grafana installation.

 

Option a: sounds interesting too, so could i save to an additional mysql database on an external device/storage and run wee_reports periodically on that device?

i have updated the 'strategies for publishing data from weeWX' to include this as scenario 7.

https://github.com/weewx/weewx/wiki/dashboards

 

[f4n...@gmail.com]

Thanks maybe I will try this out sometime.

I had an Idea: If I understand the strategies wiki right: can I save to the local sqlite database and to an external mysql database simultaneously? If so, I could send it to a mysql database on my webhosting and then access it with another weewx installation on another computer and let the heavier skins run there?  So:

raspberry Pi A [sqlite local reports] --> webserver [mysql] <--> raspberry Pi B (or other computer) [skin reports]

Could I access the mysql database simultaneously with weewx on system B? Would it run reports automatically without an attached weather station?

[matthew wall]

On Saturday, March 5, 2022 at 10:33:26 AM UTC-5 f4n...@gmail.com wrote:

I had an Idea: If I understand the strategies wiki right: can I save to the local sqlite database and to an external mysql database simultaneously?

weewx is designed to save to only one database

i have no idea whether this would work, but you might try putting two instances of StdArchive into the service list.  that *might* let you save to two different databases.  create your own archiving service by deriving from the StdArchive class, and put it in the weewx user directory.  should not be necessary to define any methods.  then refer to it in your weewx config file:

archive_services = weewx.engine.StdArchive, user.MyArchive

If so, I could send it to a mysql database on my webhosting and then access it with another weewx installation on another computer and let the heavier skins run there?  So:

raspberry Pi A [sqlite local reports] --> webserver [mysql] <--> raspberry Pi B (or other computer) [skin reports]

Could I access the mysql database simultaneously with weewx on system B? Would it run reports automatically without an attached weather station?

this is the 'standard' way to do it.  make weewx save to a mysql database, either on the computer doing the collection, or on the computer doing the aggregated reports.  then make the reporting instance of weewx pull data from the mysql database.

 

[Tom Keffer]

Nope. Won't work. Unfortunately, the state machine is tangled up in the archiving service. For example, the decision on when to pause the LOOP packets and generate archive records is done by StdArchive. Having two instances would probably result in chaos.

They really should be separated, but that's a scary project that could lead to a lot of subtle bugs.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/e2c101ac-464c-412d-8daa-e32752e98effn%40googlegroups.com.

[f4n...@gmail.com]

Could weewx be configured to generate an e.g. realtime.txt / clientraw.txt or something like that, and another weewx system to access and read it live (into its own loop packets/archive records)?

[vince]

There is another thread currently talking about using weewx-wd to 'generate' the file, but Gary is saying there's no way currently he knows of to 'ingest' it.  You might want to take a look in the other thread about weewx-wd.