Belchertown skin - large rsync uploads?

[gszla...@gmail.com]

Now that I got Belchertown running on my test bed machine it was now time to install on the "big iron". I did a fresh Belchertown skin install with MH's mqtt weewx extension and a local MQTT broker on my rpi zero 2W and am "mirroring" the local LAN Belchertown skin to my web host using rsync with passwordless SSH.

On day 1 rsync was uploading 580000 bytes (.58 MB) at a time and all was good - everything performing as it should.

On day 2 rsync was now uploading almost 1.3 MB and at that rate I will be exceeding my monthly bandwith limit of 100GB. I am "realtime" - updating every 20 seconds using Ecowitt equipment.

It looks like rsync is copying everything in var/www/html/weewx/belchertown everytime it runs. Is that normal?

[vince]

rsync syncs whatever you have it configured to sync and does so every archive interval.

A 20-second archive interval is very quick especially since your ecowitt gear only changes readings once per minute probably for most of the readings other than wind.   You might want to look into the websockets setup for Belchertown if you are looking for realtime updates.

FWIW - mine rsyncs a bit under 3 MB every 5-minute period but I have a lot of skins and images to upload.

You could probably get there via a cron entry that runs less often if you wanted to dial back how often you rsync, assuming you turned it off in weewx of course.

[gszla...@gmail.com]

Thanks vince. You are right - all my sensors update every 60 seconds except the anemometer which is every 16.5 secs so for "realtime",  I chose 20 secs. However I expected rsync to upload only the file differences rather than re-copying the entire local skin. I think the whole template as-is without adding any extra graphs - is about 1.2 MB to 1.3 MB. Rsync is uploading the whole she-bang at a time. Definitely "heavy" on resources when you add a few instances of weewx and a bunch of weather templates plus GW1000 custom upload.(realtime)...Not complaining though..rsync is still the way to go. Many thanks for all the suggestions and recommendations..much appreciated!

[Paul R Anderson]

If I understand correctly you're concerned about exceeding your monthly bandwidth of your web host. On some web hosts only outbound data transfer in excess of your plan's data transfer allowance is subject  to overage charges. Your rsync uploads from the local host to the web server would not count toward your limit. I know this to be true with AWS Lightsail, and Linode. Both AWS and Linode have plans for $5.00 a month that might be enough to host your site.

Thanks,

Paul

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/1e0ea0d2-cb84-4e25-82f9-610fae5c4d98n%40googlegroups.com.

[cric...@pobox.com]

The other thing...  Do you have shell access on your hosting provider?  You could verify that between rsync runs, the files aren't

getting modified somehow.  Failing that, stop weewx long enough to pull all the files from your web host and check them against

your sources locally.  Another option would be to capture a recursive long listing (and maybe all the file cksums) of your source directory

before each rsync run.  Diff the captures and see if it's those same files that are getting pushed.  rsync in intended to only re-send when

there are differences, and it sounds like you have more differences than expected, so that needs to be understood before worrying about

the next thing.

Chris