Re: rsync errors

[Mark F]

I'm having this exact same trouble.  Did you figure out what was going on?  I've tried using hostnames/ip addresses, changing parameters, etc.  I'm able to paste verbatim the output from the debug statement:

rsyncupload: rsync invocation: rsync --archive -e ssh /data/weewx/weewx/public_html/ weewx@<myserver>:public_html

And have a successful rsync session.   However, my log reports an rsync return of exit status 12, same as you.

On Wednesday, January 2, 2013 5:46:48 PM UTC-6, Josh London wrote:

Hello

I just upgraded to the latest version and encountered a few errors with respect to enabling rsync.

1) I had to create the Rsync and skin.conf file within the skins directory. I just copied from Ftp and changed the reportengine to weewx.reportengineRsyncGenerator. Not sure if this is the expected behavior, but through me for a loop until I realized what weewx was expecting.

2) The generator is creating the following exception in the log:

rsyncupload: rsync invocation: rsync --archive --stats --delete -e ssh /home/weewx/public_html/ myu...@myserver.net:/home1/myuser/public_html/wx

reportengine: Caught unrecoverable exception in generator weewx.reportengine.RsyncGenerator

****  Command '['rsync', '--archive', '--stats', '--delete', '-e', 'ssh', '/home/weewx/public_html/', 'myu...@myserver.net:/home1/myuser/public_html/wx']' returned non-zero exit status 12

****  Traceback (most recent call last):

****    File "/home/weewx/bin/weewx/reportengine.py", line 131, in run

****      obj.start()

****    File "/home/weewx/bin/weewx/reportengine.py", line 154, in start

****      self.run()

****    File "/home/weewx/bin/weewx/reportengine.py", line 220, in run

****      rsyncData.run()

****    File "/home/weewx/bin/weeutil/rsyncupload.py", line 77, in run

****      output = subprocess.check_output(cmd)

****    File "/usr/lib/python2.7/subprocess.py", line 544, in check_output

****      raise CalledProcessError(retcode, cmd, output=output)

****  CalledProcessError: Command '['rsync', '--archive', '--stats', '--delete', '-e', 'ssh', '/home/weewx/public_html/', 'myu...@myserver.net:/home1/myuser/public_html/wx']' returned non-zero exit status 12

****  Generator terminated...

I am able to run the rsync command directly and successfully sync with the server, so I don't think it is an authentication issue.

Note: I've edited the user and server names from the log.

cheers and thanks for all the effort

Josh

[Bill]

Curious that you can paste in the same command and have it work, obviously
something is different. I assume you are using key authentication (so you
don't need to type in a password). So the question is do you have access to
the keys in both commands?

This might be caused by not running as the same userid, ie the weewx command
and the one you type in are different users? When you created the keys make
sure they are in the right places with permissions that will work (they are
usually in ~/.ssh). You might also turn on verbose in the ssh command and
see what it has to say.

Bill

[Mark F]

On my system, weewx runs as root.  Root can ssh to my destination host using key authentication (password free), as user weewx on the destination host (i.e. "# ssh weewx@myserver" results in non interactive success).  So yes, I believe I have access to the keys in both commands (command line and via weewx).

Initiating an ssh session with verbosity was useful in that it showed me there was a GSSAPI issue.  I felt certain that resolving this would fix my problem, but unfortunately, no luck.

I'm guessing I'll have to get into the python a bit and have it log a little more verbosity than just the exit status of the rsync command.

Thanks, and I'm open to any more suggestions you have.

[Bill]

You can fix the GSSAPI issue by commenting out the lines:

# GSSAPIAuthentication yes
# GSSAPIDelegateCredentials no

in the file /etc/ssh/ssh_config, or just change the first to 'no'.

This shouldn' make any difference though, other than speeding up the login (it
will now skip the GSSAPIAuthentication, which likely times out anyhow).

You are right, you might have to see what it does from the Python side. How
about a simple test by starting Python (as root) and doing:

import os
os.system( "rsync -e 'ssh -v' ... the rest of your command")

Might be interesting to see what that says?

Bill

[Mark F]

I'd already dealt with the GSSAPI issues just as you noted.  Thanks for the heads up regardless.

Good idea on running the rsync from within Python.  What do you know - no problem.

Transferred: sent 4736, received 2456 bytes, in 0.7 seconds
Bytes per second: sent 6613.3, received 3429.5
debug1: Exit status 0

This line in weeutil/rsyncupload.py:

syslog.syslog(syslog.LOG_DEBUG, "rsyncupload: rsync invocation: %s" % " ".join(cmd))

Results in the cmd in my syslog which works.  This is the precise command that is invoked?  New to Python, so I'm trying to figure that part out.  

I feel it must be related somehow to the user performing the ssh, as in your first response.. just can't isolate that.

[Mark F]

OK, solved!

I reproduced the code interactively in python, and found that the error was that rsync couldn't locate ssh:

rsync: Failed to exec ssh: No such file or directory (2)

When I include the options '-e', 'ssh' separately in the subprocess call, this occurs.  However, when I combine them inside one quote together ('-e ssh'), it doesn't.   Odd, but whatever.  Tweaked the code, and all is happy now.

Now I'm going to add support for nonstandard ssh port in the config and rsync code.  I'll attach a patch here if interested.

[Thomas Keffer]

Definitely interested!

-tk

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Tom Keffer

kef...@threefools.org
+1 541-386-8891 (h)
+1 541-490-9507 (c)

Skype: tkeffer

[Mark F]

Two patch files attached - 

rsyncupload.patch: bin/weeutil/rsyncupload.py

reportengine.patch: bin/weewx/reportengine.py

The config file will take a new argument under the RSYNC report section.  For example:

port = 11111

If this doesn't exist, it will use the default ssh port.

I am not doing any sanity checks to make sure the port is within valid TCP ranges, etc.  Rsync will fail if you try something odd, which I imagine is good enough. Let me know if anyone disagrees.

[Bill]

Excellent news I guess there is something about your search path as user and
as root that are different?

The other thing you need to watch with rsync, is dealing with special
characters can be tricky, especially spaces.

Bill

[Thomas Keffer]

Mark,

Could you send me the updated version of rsyncupload.py, rather than the patch? My version is not matching yours at all.

-tk

[Mark F]

Attached.