Exclude list or Overwrite behaviour for RSync Report

[Chris Alemany]

Good morning all,

I’m having some trouble with the weewx RSync report overwriting files that have changed on the remote side independent of weewx. Are there any provisions in the RSync report to exclude a list of files and/or folders (including in skins) or otherwise change the way the process operates so that files that exist on both ends but with a newer timestamp on the remote side don’t get overwritten by the older local file?
Thanks very much

Chris
https://alberniweather.ca

[Chris Alemany]

And just to be complete, this is on weewx 5.0.2, pip install on Debian.

[vince]

Check out the options in https://ss64.com/bash/rsync_options.html and see if you can make vanilla rsync from the shell do (or not do) what you want.  If that doesn't solve it, I'd doubt weewx can help much there as a wrapper over rsync.   The -u option might be what you're looking for possibly.

But the rsync uploader (rsyncupload.py) doesn't seem to have the ability to provide additional flags to the underlying rsync command.  I'm sure Tom would appreciate a PR to add more functionality if you can figure out an incantation works and some more general purpose thing to add to weewx (possibly something like a rsync_options = "whatever" thing?).

That said - if you're trying to use weewx as the source of the authoritative file to upload something, why would you externally mess with the file(s) on the far side after that ?  I'm sure there's some use case you have but I can't think of a reason why it would be necessary....

[Chris Alemany]

Thanks for the hints Vince. I ended up looking through the old Wiki and came upon this page about backing up databases with the rsync report and how you can spit rsync into multiple skins.

https://github.com/weewx/weewx/wiki/Using-the-RSYNC-skin-as-a-backup-solution

My use case is that I'm using divumWX/weather34 which has its own builtin rsync so there seems to be some conflict between the two as divumWX is a subdirectory of the main htmlroot.

Like you said, it does not appear that there is any way to add options to the builtin weewx rsync.py file or any extra options in the skin or weewx.conf. So the alternative is to narrow what weewx is uploading by pointing it to subdirectories rather than updating the entire public_html directory and in that way excluding the directory I don't want weewx to deal with.

I might add a feature request to the GitHub to add options like you suggested.

Thanks!

Chris

[Glenn McKechnie]

On Tue, 2 Apr 2024 at 15:00, Chris Alemany <chri...@gmail.com> wrote:

Thanks for the hints Vince. I ended up looking through the old Wiki and came upon this page about backing up databases with the rsync report and how you can spit rsync into multiple skins.

https://github.com/weewx/weewx/wiki/Using-the-RSYNC-skin-as-a-backup-solution

That was a good solution for a while, until - like you - I wanted more.

I eventually pulled the rsyncupload.py code from the main tree and stuffed it into a skin.

The following was written a while ago. I still use it as is, but it's rough and untidy.  Really does need a clean up but I haven't and still don't have the time do that, so you get it warts and all...

https://github.com/glennmckechnie/weewx-rsynctransfer

It should install  *Gulp*, it should be self explanatory *Gulp*

There are plenty of notes and examples in it that show it can handle the additional rsync command line options.

my email address is in the code, there's the  issues tab on github and I always read this group.

Cheers

 Glenn

Various WeeWx addons at
https://github.com/glennmckechnie

[vince]

Seems complicated.  

I was suggesting something like adding a general purpose 'rsync_options' variable in that section of weewx.conf, and adding the value  to the assembled rsync command in the run(self) routine in rsyncupload.py ?     Then the user could add any combination of the rsync options to the command if needed.   Perhaps something like the following (untested):

pi@pi4:/tmp$ diff rsyncupload.py.orig rsyncupload.py -u
--- rsyncupload.py.orig 2024-04-02 08:18:48.311469290 -0700
+++ rsyncupload.py 2024-04-02 08:19:57.970078706 -0700
@@ -46,6 +46,7 @@
         self.delete = delete
         self.port = port
         self.ssh_options = ssh_options
+        self.rsync_options = rsync_options   # <== new
         self.compress = compress
         self.log_success = log_success
         self.log_failure = log_failure
@@ -96,6 +97,7 @@
         if self.timeout is not None:
             cmd.extend(["--timeout=%s" % self.timeout])
         cmd.extend(["-e"])
+        cmd.extend([rsync_options])    # <=== new
         cmd.extend([rsyncsshstring])
         cmd.extend([rsynclocalspec])
         cmd.extend([rsyncremotespec])

And add a rsync_options = "whatever you want here" to the [[RSYNC]] section in weewx.conf ala:

   [[RSYNC]]
        delete = 0
        skin = Rsync
        enable = true
        server = x.x.x.x 
        user = myremoteuser
        path = /my/remote/path/here
        log_success = true
        log_failure = true

        rsync_options = "-x -y -z whatever"       # <=== new

Disclaimer - totally untested and no idea if there are security risks of doing it this way, but it would seem to be about as flexible as you can get toward supporting whatever rsync options the underlying os supports with a minimum of change to the existing uploader....

[Chris Alemany]

Hmm, both strategies have advantages. It'd be great if weewx had the ability to add in rsync options so that if you just wanted to do a single thing, exclude a directory, or maybe conform to a specific non-standard setting on the remote host, then you can easily do that in weewx.

The extension is nice to have to create a full suite of rsync options so people can use it to backup stuff, move stuff, etc.

I've created a GitHub issue/request

https://github.com/weewx/weewx/issues/951

Thanks to you both.
Chris

[John Kline]

The problem with an rsync_options (similar to the existing ssh_options) is that the code already has rsync options specified individually (e.g., delete and compress).  It would be messy to have two ways to specify delete and compress.

It might be best to add an explicit exclude option.  BTW, exclude can be specified multiple times, so the value should be an array.

On Apr 2, 2024, at 11:56 AM, Chris Alemany <chri...@gmail.com> wrote:

Hmm, both strategies have advantages. It'd be great if weewx had the ability to add in rsync options so that if you just wanted to do a single thing, exclude a directory, or maybe conform to a specific non-standard setting on the remote host, then you can easily do that in weewx.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/0d2bbda5-dc2c-4075-bcae-44b6b4c01063n%40googlegroups.com.

[John Kline]

I should add that the way I solved this problem is to put the files outside of the reporting path (/home/weewx/public_html for me).  In that way, they aren’t swept up in the rsync report.

On Apr 2, 2024, at 6:08 PM, John Kline <jo...@johnkline.com> wrote:



[vince]

Agree that one way to do it would be ideal, unless there are backward compatibility reasons to leave the current switches 'also' valid.  I guess it would be possible to tear apart a hypothetical rsync_options="whatever here" string to not break folks, but I didn't want to write the code this time, just suggest where it might be doable to at least think about as a feature addition.

[John Kline]

The reason to have backward compatibility would be to not break people on a point release.

 

On Apr 2, 2024, at 6:35 PM, vince <vince...@gmail.com> wrote:

Agree that one way to do it would be ideal, unless there are backward compatibility reasons to leave the current switches 'also' valid.  I guess it would be possible to tear apart a hypothetical rsync_options="whatever here" string to not break folks, but I didn't want to write the code this time, just suggest where it might be doable to at least think about as a feature addition.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/9e81f999-3e60-4b31-88cb-3d1ddbd74cc0n%40googlegroups.com.

[Tom Keffer]

I can do it the same way the MySQL driver handles extra parameters: you list them with the others. The code strips off the ones it recognizes, then passes the rest on to the rsync command.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/AE72C34C-BB70-4EDE-83CD-03BF9877BA24%40johnkline.com.