Rsyc Error

[Tom -KQ5S]

I am getting the following error from time to time.  Rsync will work fine for a while and then I get the error.  I might get the error for a couple of cycles then it will work again, then the error again.  I have included the reply from my internet host as well.  Ideas?

Oct 26 07:50:26 raspberrypi weewx[30090] DEBUG weewx.reportengine: Running report 'RSYNC'

Oct 26 07:50:26 raspberrypi weewx[30090] DEBUG weewx.reportengine: Found configuration file /home/weewx/skins/Rsync/skin.conf for report 'RSYNC'

Oct 26 07:50:26 raspberrypi weewx[30090] DEBUG weewx.reportengine: Cannot read localization file /home/weewx/skins/Rsync/lang/en.conf for report 'RSYNC': Config file not found: "/home/weewx/skins/Rsync/lang/en.conf".

Oct 26 07:50:26 raspberrypi weewx[30090] DEBUG weewx.reportengine: **** Using defaults instead.

Oct 26 07:50:26 raspberrypi weewx[30090] DEBUG weeutil.rsyncupload: rsyncupload: cmd: [['rsync', '--archive', '--stats', '-e', 'ssh -p 2222', '/home/weewx/public_html/', ‘xx...@kq5s.com]]

Oct 26 07:50:27 raspberrypi weewx[30090] ERROR weeutil.rsyncupload: rsync reported errors. Original command: ['rsync', '--archive', '--stats', '-e', 'ssh -p 2222', '/home/weewx/public_html/', ‘xxx...@kq5s.com:xxxx’]

Oct 26 07:50:27 raspberrypi weewx[30090] ERROR weeutil.rsyncupload: **** kex_exchange_identification: Connection closed by remote host

Oct 26 07:50:27 raspberrypi weewx[30090] ERROR weeutil.rsyncupload: **** Connection closed by xxx.xx.xxx.xx port 2222

Oct 26 07:50:27 raspberrypi weewx[30090] ERROR weeutil.rsyncupload: **** rsync: connection unexpectedly closed (0 bytes received so far) [sender]

Oct 26 07:50:27 raspberrypi weewx[30090] ERROR weeutil.rsyncupload: **** rsync error: unexplained error (code 255) at io.c(228) [sender=3.2.3]

==========================================================

From Internet host.  I'm not sure where the user weewx is coming from.  It is not the user set in my config file for rsync.

Regarding SSH, a review of the server's logs shows two repeated occurrences for your user ID "weewx". I've copied an example of each error below:

Oct 11 15:37:17 ecbiz246 sshd[55604]: Invalid user weewx from xxx.xx.xxx.xx port 60374
Oct 11 15:37:17 ecbiz246 sshd[55604]: Received disconnect from xxx.xx.xxx.xx port 60374:11: Bye Bye [preauth]
Oct 11 15:37:17 ecbiz246 sshd[55604]: Disconnected from xxx.xx.xxx.xx port 60374 [preauth]
 

Oct 11 06:46:23 ecbiz246 sshd[3762]: drop connection #28 from [xxx.xx.xxx.xx]:42380 on [xxx.xx.xxx.xx]:2222 past MaxStartups
...
Oct 11 09:30:20 ecbiz246 sshd[3762]: drop connection #27 from [xxxx.xx.xxx.xx]:47222 on [xxx.xx.xxx.xx]:2222 past MaxStartups

It appears your Pi is intermittently trying to connect with an incorrect SSH user string, and subsequent connections are maxing the limit for concurrent SSH connections (MaxStartups), so the excess connections are simply dropped by the server. I suggest reviewing the SSH connection string for the PI, as well as ensuring it is closing connections correctly. Additionally, you will want to ensure you are not opening too many concurrent connections for transfer. If the issue persists you may want to increase the amount of time between uploads, as we are unable to increase the MaxStartups setting on shared server.

[kq5...@gmail.com]

Anyone? 

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/038c234b-113e-4ace-9a62-24c896e5bac2n%40googlegroups.com.

[cric...@pobox.com]

Is this a new setup, or has it worked in the past?

What userID is running weewx?

What version of weewx are you running?

The contents of my rsync/skin.conf is:
SKIN_NAME = Rsync
SKIN_VERSION = 4.10.2

[Generators]
    generator_list = weewx.reportengine.RsyncGenerator

Does rsync of your /home/weewx/public_html to your web hosting service work from the command line?

If, as which user?

[Tom -KQ5S]

It has worked in the past.  I am unsure when it started the working/not working behavior since I don’t look at the logs that often and the error was not showing up in the logwatch file.

I am running as user pi.

version = 4.10.2

Pi OS Bullseye

SKIN_NAME = Rsync

SKIN_VERSION = 4.10.2

[Generators]

    generator_list = weewx.reportengine.RsyncGenerator

The fact that it is intermittent makes me think the authorizations are OK.  Below is the last couple of iterations where you can see it failed and then worked

Oct 27 13:55:21 raspberrypi weewx[15431] ERROR weeutil.rsyncupload: rsync reported errors.:

Oct 27 14:00:26 raspberrypi weewx[15431] INFO weeutil.rsyncupload: rsync'd 149 files (919,702 bytes) in 1.94 seconds

Oct 27 14:05:20 raspberrypi weewx[15431] INFO weeutil.rsyncupload: rsync'd 29 files (448,591 bytes) in 1.34 seconds

Oct 27 14:10:22 raspberrypi weewx[15431] INFO weeutil.rsyncupload: rsync'd 32 files (453,999 bytes) in 1.28 seconds

Oct 27 14:15:19 raspberrypi weewx[15431] ERROR weeutil.rsyncupload: rsync reported errors.

Oct 27 14:20:22 raspberrypi weewx[15431] INFO weeutil.rsyncupload: rsync'd 32 files (454,465 bytes) in 1.29 seconds

Oct 27 14:25:19 raspberrypi weewx[15431] ERROR weeutil.rsyncupload: rsync reported errors. Original command: ['rsync', '--archive', '--stats', '-e', 'ssh -p 2222', '/home/weewx/public_html/', 'kq5...@kq5s.com:public_html']

Oct 27 14:30:21 raspberrypi weewx[15431] ERROR weeutil.rsyncupload: rsync reported errors

Oct 27 14:35:20 raspberrypi weewx[15431] INFO weeutil.rsyncupload: rsync'd 32 files (457,673 bytes) in 1.39 seconds

Oct 27 14:40:22 raspberrypi weewx[15431] INFO weeutil.rsyncupload: rsync'd 32 files (457,727 bytes) in 1.28 seconds

[cric...@pobox.com]

Just an observation, but that's a LOT of files and data getting uploaded, and the total time is

close to 5 minutes, which is the default archive interval.  That MaxStartups limit may be set to 1,

so is it possible weewx is starting another upload cycle while the last one is in progress?  You could

try changing this to 600 and see if it helps (in your weewx.conf):

[StdArchive]
    archive_interval  = 300

Chris

[kq5...@gmail.com]

I’m using the standard 5 minute interval.  The files being uploaded are the files generated by the standard and seasons report and the Saratoga extension.  As you can see the upload takes less than 2 seconds.  When I was using ftp to upload I was getting errors due to the time it took.  Rsync was suggested and it has been working well.  I don’t know much about how weewx and rsync work but my internet host seems to think it is a software problem.

It appears your Pi is intermittently trying to connect with an incorrect SSH user string, and subsequent connections are maxing the limit for concurrent SSH connections (MaxStartups), so the excess connections are simply dropped by the server. I suggest reviewing the SSH connection string for the PI, as well as ensuring it is closing connections correctly. Additionally, you will want to ensure you are not opening too many concurrent connections for transfer. If the issue persists you may want to increase the amount of time between uploads, as we are unable to increase the MaxStartups setting on shared server.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/44ac6f5d-f47d-4456-be6f-ddcf1fc1391an%40googlegroups.com.

[cric...@pobox.com]

Yeah, I need to read slower.  :^)

You are running weewx as user pi? Default is to run the process as root.

So, I'm assuming since you running as pi, you've rync'd your those ssh keys to the remote host?

BTW, do you have debug=1 (or 2) in your weewx.conf?  That might help shed some light.

Please post your [[RSYNC]] config from the [StdReport].

The one other thing to try is running the rsync by hand from the command line several times and see

if you can reproduce the error and get more messaging.

I'm pretty sure weewx only starts one rsync at a time.  Looking through the code, I can only find one place

where those error messages are generated, and it always prints the command, and should print any of the cmd's output.

bin/weeutil/rsyncupload.py: log.error("rsync reported errors. Original command: %s", cmd)

I'm going to have to defer to the experts on this one.

Chris

[Tom -KQ5S]

I appreciate the help but I am getting in over my head.  The fact that it is intermittent makes me think the setup is OK or else it would fail all the time. This is one reason I posted on this group thinking it might be something in the weewx code or that my internet provider was blowing smoke. 

I did set debug=2 and got the below results.   I have not seen this particular error message:

Oct 29 11:25:20 raspberrypi weewx[9966] ERROR weeutil.rsyncupload: **** rsync error: error in rsync protocol data stream (code 12) at io.c(228) [sender=3.2.3]

====================================================

Oct 29 11:25:20 raspberrypi weewx[9966] DEBUG weewx.reportengine: Running report 'RSYNC'

Oct 29 11:25:20 raspberrypi weewx[9966] DEBUG weewx.reportengine: Found configuration file /home/weewx/skins/Rsync/skin.conf for report 'RSYNC'

Oct 29 11:25:20 raspberrypi weewx[9966] DEBUG weewx.reportengine: Cannot read localization file /home/weewx/skins/Rsync/lang/en.conf for report 'RSYNC': Config file not found: "/home/weewx/skins/Rsync/lang/en.conf".

Oct 29 11:25:20 raspberrypi weewx[9966] DEBUG weewx.reportengine: **** Using defaults instead.

Oct 29 11:25:20 raspberrypi weewx[9966] DEBUG weeutil.rsyncupload: rsyncupload: cmd: [['rsync', '--archive', '--stats', '-e', 'ssh -p 2222', '/home/weewx/public_html/', ‘XX...@kq5s.com:public_html']]

Oct 29 11:25:20 raspberrypi weewx[9966] ERROR weeutil.rsyncupload: rsync reported errors. Original command: ['rsync', '--archive', '--stats', '-e', 'ssh -p 2222', '/home/weewx/public_html/', ‘XX...@kq5s.com:public_html']

Oct 29 11:25:20 raspberrypi weewx[9966] ERROR weeutil.rsyncupload: **** kex_exchange_identification: Connection closed by remote host

Oct 29 11:25:20 raspberrypi weewx[9966] ERROR weeutil.rsyncupload: **** Connection closed by 199.xxx.xxx.xx port 2222

Oct 29 11:25:20 raspberrypi weewx[9966] ERROR weeutil.rsyncupload: **** rsync: connection unexpectedly closed (0 bytes received so far) [sender]

Oct 29 11:25:20 raspberrypi weewx[9966] ERROR weeutil.rsyncupload: **** rsync error: error in rsync protocol data stream (code 12) at io.c(228) [sender=3.2.3)

======================================================

Original from my internet provider.

Regarding SSH, a review of the server's logs shows two repeated occurrences for your user ID "weewx". I've copied an example of each error below:

Oct 11 15:37:17 ecbiz246 sshd[55604]: Invalid user weewx from xxx.xx.xxx.xx port 60374
Oct 11 15:37:17 ecbiz246 sshd[55604]: Received disconnect from xxx.xx.xxx.xx port 60374:11: Bye Bye [preauth]
Oct 11 15:37:17 ecbiz246 sshd[55604]: Disconnected from xxx.xx.xxx.xx port 60374 [preauth]
 

Oct 11 06:46:23 ecbiz246 sshd[3762]: drop connection #28 from [xxx.xx.xxx.xx]:42380 on [xxx.xx.xxx.xx]:2222 past MaxStartups
...
Oct 11 09:30:20 ecbiz246 sshd[3762]: drop connection #27 from [xxxx.xx.xxx.xx]:47222 on [xxx.xx.xxx.xx]:2222 past MaxStartups

It appears your Pi is intermittently trying to connect with an incorrect SSH user string, and subsequent connections are maxing the limit for concurrent SSH connections (MaxStartups), so the excess connections are simply dropped by the server. I suggest reviewing the SSH connection string for the PI, as well as ensuring it is closing connections correctly. Additionally, you will want to ensure you are not opening too many concurrent connections for transfer. If the issue persists you may want to increase the amount of time between uploads, as we are unable to increase the MaxStartups setting on shared server.

[vince]

You are posting very cryptic information making helping you very difficult.  And your font looks like you're yelling.

We need to see pieces of your actual configuration files:

• post your weewx.conf RSYNC section(s)
• post your ~/.ssh/config section for the host(s) you referenced in the RSYNC section in weewx.conf
• (make sure you show use the ssh config for the user weewx runs as, whatever you are using)

It almost sounds like you have multiple RSYNC sections in weewx and you are trying to upload twice, once to one user, once to a different user.  Regardless, we need to see your actual config file sections to help much.

[paul.ba...@gmail.com]

I just measured the time it takes for me to do the full rsync of /var/www/html/weewx . 

Rersults:    real 0m0.438s  /  user 0m0.050s  /  sys 0m0.107s 

Source was my main computer (5y old mother board) to a RPi400 (Rpi4B witj 4 GB mem, inside a keyboard)

So your connection must be very slow to make it around 5 minutes.

In this case, a possible "trick" could be to use the "compression" option of rsync.  Add   -z   or   --compression  in the options of rsync.

The size of all transmitted files is reduced by ~30 %. In facts, the image files are almost unchanged, but they are all rather small, while the Index.html and other text files are strongly compressed.

30 %  reductionis not much, but is still 3.5 min instead of 5 which would solve you problem.

Before changing the configurations, you could test just rsync with the following command on the machine running weewx (not the www server):

    time  rsync  -av  /var/www/html/weewx/index.html  <www user>@<your www-server>:

Then remoce  index.html  on the <www server>  and try the same rsync on the weewx machine, but with compression:

    time  rsync  -avz  /var/www/html/weewx/index.html  <www user>@<your www-server>:

Do you get some/sufficient  improvement ?  I would be interested to see your results and possibly make some (helpfull ?) comments.

Good luck !     Yours,   Paul

[cric...@pobox.com]

Tom,

    So, from your hosting provider, you are getting two types of errors.  One for userID, the other for MaxStartups.  The #27 and #28 are the number of

Startups, so something is getting hung up somewhere between you and your hosting service, and weewx seems to be the victim.  This sure seems like

a basic networking/ssh setup issue that's best debugged by hand outside of weewx.  One more thing you can try is simply ssh'ing to your hosting site,

but include a -v (or two or three v's) in the command line.  It generates a LOT of messages, but take your time to see what it's telling you if there's a failure.

Make sure all your permissions are correct in all the places you have an .ssh directory (directory: chmod 2700, files: chmod 0600).

Chris

[vince]

I'm not understanding what you guys think you see.   I don't see anything in his logs saying the total time to do things is exceeding the 5-minute default archive interval.  His possibly edited log snippet above showed rsync happening every 5 minutes which is normal.  It only takes a couple seconds which is normal.   The number of files is reasonable given the skins he's using. 

What is 'not' normal is the support ticket reply from his ISP saying he's intermittently trying to connect to an invalid user id on the remote side.

Again - lets see your RSYNC stanza(s) from weewx.conf and also your .ssh/config stanza for the account you're using locally to connect to the ISP server.   This is almost certainly a rsync/ssh configuration issue based on what I've seen in the thread above.

[Tom -KQ5S]

Thanks, Vince for the help as I have seen you help many others on here.  I wasn't shouting before I merely copied and pasted what my ISP sent me.  That is the font they used.  

The only files in my /root/.ssh directory are the rsa keys and a known_hosts file.  I don't remember creating a config file so that may be part of the problem.  I see where the instructions under [RSYNC] say to put the server in the .ssh/config file.  Is the config file as simple as:

===================================================

Host website

        Hostname kq5s.com

       User pi (or name in my [Rsync] part of weewx.conf)

        Port 2222

==================================================

[RSYNC]]
        # rsync'ing to a webserver is treated as just another report
        skin = Rsync
        enable = true
        server = kq5s.com
        user = XXXXX ( not sure if I should post this or not.  It is a user on my internet site)
        path = public_html
        port = 2222
        

        delete = 0

Tom

[Tom -KQ5S]

I don't know if it makes a difference but I just did a search and I have the same .ssh folder under /home/pi/.ssh as well as /boot/.ssh.  I must have put it there while trying to trouble shoot the problem.

[vince]

you said you run as user 'pi' so you need to provide your /home/pi/.ssh/config file so we see the rsync/ssh setup for the user weewx is running as...

[Tom -KQ5S]

Hi Vince,

As mentioned earlier I do not have a .ssh/config file so perhaps that is the issue.  Would the file look like:

Host website

        Hostname kq5s.com

       User pi (or the name in my [Rsync] part of weewx.conf)

        Port 2222

[cric...@pobox.com]

So, here's one of mine:

renogy0_/home/crichmon> cat .ssh/config
# hosts with more than two components are probably external
PubkeyAuthentication yes
# relax checking on internal hosts/networks # hopefully this list will not grow and all new site's networks will be
Host 192.168.0.* 192.168.1.* 192.168.2.* 192.168.3.*
        StrictHostKeyChecking no
Host *.chaos.home *.csrhome.home *.garage.home
        StrictHostKeyChecking no
Host *.*
        StrictHostKeyChecking yes

Not particularly interesting.  I'd suggest looking at: man ssh_config

At the end of the day, this is not the likely problem.  As you said, the issue seems to be somewhat intermittent, so it isn't

a complete config failure, but something a bit harder to figure out.

Still, we need as many details as you can provide.  One thing... You said you are running weewx as userID pi.

Is that a fact or an assumption?  How do you know?  ps uaxw | grep weewx will give that answer.  Knowing that

will narrow down the possible problems.  The other bit is how your rsync is configured.  We can sort-of see that

from the logs, but nothing is clear yet.

You still haven't provided the RSYNC config.  One of mine is:

    [[RSYNC]]
        skin   = Rsync
        enable = false
        server = router1
        user   = crichmon
        path   = /var/www/html/solar
        delete = 0

Nothing secret here, at least for me.  This one is internal to my network.  I don't actually have weewx push externally.

That's done elsewhere.

The other bit is the mystery of how ssh is set up for both ends of your connection, and details of what happens

when running either ssh or rsync manually.

I'll step out here a bit:

renogy0_/home/crichmon> ls -la .ssh
drwx------  2 crichmon crichmon  4096 Jul 19 12:50 .
drwxr-xr-x 14 crichmon crichmon  4096 Oct 29 15:57 ..
-rw-------  1 crichmon crichmon  1029 Aug 24  2019 authorized_keys
-rw-------  1 crichmon crichmon   415 Jul 19 12:50 config
-rw-------  1 crichmon crichmon   883 Jan 16  2009 id_rsa
-rw-------  1 crichmon crichmon   226 Jan 16  2009 id_rsa.pub
-rw-------  1 crichmon crichmon  1675 Jan 18  2019 id_rsa_r1
-rw-------  1 crichmon crichmon   398 Jan 18  2019 id_rsa_r1.pub
-rw-------  1 crichmon crichmon  2558 Jul  7  2019 identity
-rw-------  1 crichmon users    26801 Sep 27 15:35 known_hosts

The same from you might also eliminate a possible source of error.  We'd need this for all your users and all

ends of the connections, or at least confirmation that your permissions are reasonable.

Chris

[Tom -KQ5S]

Thanks Chris.

ps uaxw | grep weewx
root     10111  2.6  0.6 117148 52824 ?        Ssl  11:31   9:09 /usr/bin/python3 /home/weewx/bin/weewxd /home/weewx/weewx.conf
pi       13411  0.0  0.0   7444   580 pts/4    S+   17:21   0:00 grep --color=auto weewx

[[RSYNC]]

        # rsync'ing to a webserver is treated as just another report
        skin = Rsync
        enable = true
        server = kq5s.com
        user = XXXXX ( not sure if I should post this or not.  It is a user on my internet site)
        path = public_html
        port = 2222
        

        delete = 0

[vince]

So yet again - what is the ~/.ssh/config entry for kq5s.com for the user weewx is running as.

Typically that is root based on the paths you've provided.

• Are you 'sure' you are running weewx specifically as user pi ?

• if you're running as the non-standard user pi, show us how you did that

• What is in /root/.ssh/config ?

• What is in /home/pi/.ssh/config ?

• Are you 'sure' you do not have any cron entries trying to do rsync outside weewx ?
  

You should be able to do something like "ssh XX...@kq5s.com date" successfully if you have shell access there.

[Tom -KQ5S]

 sudo ssh xxx...@kq5s.com -p2222 -v
OpenSSH_8.4p1 Raspbian-5+deb11u2, OpenSSL 1.1.1w  11 Sep 2023
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to kq5s.com [199.250.194.206] port 2222.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type 0
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa_sk type -1
debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_ed25519_sk type -1
debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.4p1 Raspbian-5+deb11u2
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug1: Authenticating to kq5s.com:2222 as 'xxxxx'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes128-ctr MAC: umac-...@openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: umac-...@openssh.com compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:p4xX8yxXsJ+3l/Xg8B29Qbdc5r9iHXvoX0bpFCpCwes
debug1: Host '[kq5s.com]:2222' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:2
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /root/.ssh/id_rsa RSA SHA256:PCS+HvqfV/Y0X/UuplMbVw+cfzk+v5Bk+pC7hQubdbU
debug1: Will attempt key: /root/.ssh/id_dsa
debug1: Will attempt key: /root/.ssh/id_ecdsa
debug1: Will attempt key: /root/.ssh/id_ecdsa_sk
debug1: Will attempt key: /root/.ssh/id_ed25519
debug1: Will attempt key: /root/.ssh/id_ed25519_sk
debug1: Will attempt key: /root/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /root/.ssh/id_rsa RSA SHA256:PCS+HvqfV/Y0X/UuplMbVw+cfzk+v5Bk+pC7hQubdbU
debug1: Server accepts key: /root/.ssh/id_rsa RSA SHA256:PCS+HvqfV/Y0X/UuplMbVw+cfzk+v5Bk+pC7hQubdbU
debug1: Authentication succeeded (publickey).
Authenticated to kq5s.com ([199.250.194.206]:2222).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-...@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostk...@openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending env LC_ALL = en_US.UTF-8
[kxxxx@ecbiz246 ~]$ 

[Tom -KQ5S]

No cron jobs running rsync.

The only files in my /root/.ssh directory are the rsa keys and a known_hosts file.  There is no config file.

ps uaxw | grep weewx
root     10111  2.6  0.6 117148 52824 ?        Ssl  11:31   9:09 /usr/bin/python3 /home/weewx/bin/weewxd /home/weewx/weewx.conf
pi       13411  0.0  0.0   7444   580 pts/4    S+   17:21   0:00 grep --color=auto weewx

[Tom -KQ5S]

Without root:

ssh xx...@kq5s.com -p2222 -v

OpenSSH_8.4p1 Raspbian-5+deb11u2, OpenSSL 1.1.1w  11 Sep 2023
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to kq5s.com [199.250.194.206] port 2222.
debug1: Connection established.

debug1: identity file /home/pi/.ssh/id_rsa type 0
debug1: identity file /home/pi/.ssh/id_rsa-cert type -1
debug1: identity file /home/pi/.ssh/id_dsa type -1
debug1: identity file /home/pi/.ssh/id_dsa-cert type -1
debug1: identity file /home/pi/.ssh/id_ecdsa type -1
debug1: identity file /home/pi/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/pi/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/pi/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/pi/.ssh/id_ed25519 type -1
debug1: identity file /home/pi/.ssh/id_ed25519-cert type -1
debug1: identity file /home/pi/.ssh/id_ed25519_sk type -1
debug1: identity file /home/pi/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/pi/.ssh/id_xmss type -1
debug1: identity file /home/pi/.ssh/id_xmss-cert type -1

debug1: Local version string SSH-2.0-OpenSSH_8.4p1 Raspbian-5+deb11u2
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002

debug1: Authenticating to kq5s.com:2222 as 'xxxxxx'

debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes128-ctr MAC: umac-...@openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: umac-...@openssh.com compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:p4xX8yxXsJ+3l/Xg8B29Qbdc5r9iHXvoX0bpFCpCwes
debug1: Host '[kq5s.com]:2222' is known and matches the ECDSA host key.

debug1: Found key in /home/pi/.ssh/known_hosts:2

debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks

debug1: Will attempt key: /home/pi/.ssh/id_rsa RSA SHA256:PCS+HvqfV/Y0X/UuplMbVw+cfzk+v5Bk+pC7hQubdbU
debug1: Will attempt key: /home/pi/.ssh/id_dsa
debug1: Will attempt key: /home/pi/.ssh/id_ecdsa
debug1: Will attempt key: /home/pi/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/pi/.ssh/id_ed25519
debug1: Will attempt key: /home/pi/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/pi/.ssh/id_xmss

debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey

debug1: Offering public key: /home/pi/.ssh/id_rsa RSA SHA256:PCS+HvqfV/Y0X/UuplMbVw+cfzk+v5Bk+pC7hQubdbU
debug1: Server accepts key: /home/pi/.ssh/id_rsa RSA SHA256:PCS+HvqfV/Y0X/UuplMbVw+cfzk+v5Bk+pC7hQubdbU

debug1: Authentication succeeded (publickey).
Authenticated to kq5s.com ([199.250.194.206]:2222).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-...@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostk...@openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending env LC_ALL = en_US.UTF-8

[xxxxxx@ecbiz246 ~]$

[cric...@pobox.com]

OK, still confused, but for sure, doing things as root to a remote server you don't own is likely to fail.

It looks like the second attempt with:  ssh xx...@kq5s.com -p2222 -v

seems to work, but you cut off the log with ...

Anyhow, weewx is running as root (typical, not bad), and you've configured rsync user as:

 user = XXXXX ( not sure if I should post this or not.  It is a user on my internet site)

So... does user XXXX have ssh keys, and do the contents of the key files and the like match

on the other end?  Verify with cksum of sha256sum on each of the files.  Are all the permissions

correct.  ssh is very picky about tight permissions.  From the second log, it looks like everything

is OK as pi.  Seems like sudo ssh <stuff> also works.  Do you have matching keys in ~/root/.ssh,

XXX.ssh, and ~/pi/.ssh, and on the remote end in ~/XXX/.ssh?  All that has to match up for things

to work.  Do you actually have an XXX account on your local machine?  Maybe that's a missing piece.

Chris

[cric...@pobox.com]

Need to better proofread before sending:

"Verify with cksum of sha256sum" should be   Verify with cksum or sha256sum

"Do you have matching keys in ~/root/.ssh, XXX.ssh, " should be: Do you have matching keys in ~root/.ssh, ~XXX/.ssh, 
"the remote end in ~/XXX/.ssh?  " should be the remote end in ~XXX/.ssh?  

[vince]

Tom - I'm at a loss at this point as to why you're not working (or failing) consistently.

I did a setup of rsync over ssh on a clean pi and here are the steps.

Worked first time for me :-)

(this example uses weewx v5.0b15 as user 'pi')

# reimaged a raspberry pi
# installed rsyslog to get traditional syslogging rather than systemd
    sudo apt-get install -y rsyslog

# logged into the pi as unprivileged user pi (for this example)

# installed weewx with the simulator driver

# created a new ssh keypair for the user pi
# taking the defaults to make a passwordless keypair
# which will be ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub
    ssh-keygen

# added this key to the appropriate user@host for my server host

    ssh-copy-id vince@nuc2

# verified I could log into the remote server
    ssh vince@nuc2
# at this point ssh is known to work using the new ssh key....

# edited weewx.conf to set debug=1 for more verbose logging

# edited weewx.conf to configure the RSYNC uploader
# so this will upload the default public_html to the specified path

   [[RSYNC]]
        skin = Rsync
        enable = true
        server = nuc2
        user = vince
        path = /home/vince/weehtml
        log_success = true
        log_failure = true
        delete = 0

# restarted weewx, waited 5 minutes, I see the successful rsync of the Seasons skin in syslog

    2023-10-29T19:00:20.236994-07:00 raspberrypi weewx[2416] DEBUG weeutil.rsyncupload: rsyncupload: cmd: [['rsync', '--archive', '--stats', '-e', 'ssh', '/home/pi/weewx-data/public_html/', 'vince@nuc2:/home/vince/weehtml']]

    2023-10-29T19:00:20.680310-07:00 raspberrypi weewx[2416] INFO weeutil.rsyncupload: rsync'd 73 files (526,454 bytes) in 0.44 seconds

# Notes:
#  - no /home/pi/.ssh/config file was needed on a pi as long as the specified 'server' resolves
#       because ssh on the pi defaults to using ~/.ssh/id_rsa as the private key

#  - weewx assumes the local user is the one it is running as

#  - weewx uses the RSYNC 'user' as the remote user it connects to

[Tom -KQ5S]

Thanks to Vince and Chris for the help.  At this point I am giving up.  I'll try a fresh install and see what happens. 

Tom.  

[Tom -KQ5S]

As a follow-up I did a clean install of bookworm and WeeWX 5, using apt-get,  on my Pi.  I did nothing else except set up Rsync and I am getting the same sporadic errors as before.  No crons.  No extensions.  Nothing was added to the base install.

[Tom -KQ5S]

I guess I installed V5.  The config file shows version = 4.10.2.  I followed the install instructions at https://weewx.com/docs/5.0/quickstarts/debian/.

[cric...@pobox.com]

At the end of the day, we're still back to debugging the interaction between you and your web hosting service.

It is unlikely, but not impossible, that weewx contributes to the issue.  Debug logs are required, and maybe the logs

again from your hosting service.

As an alternative, do you have another computer/VM you can set up as your rsync destination to see if that works

cleanly or not?  It's possible to set up two "Reports" that use the rsync skin with different configs, just name them

differently.  Like this:

[StdReport]
    [[RSYNCa]]

    [[RSYNCb]]

Once basic ssh connectivity is set up, the rest of all this shouldn't be so difficult to make work.  For you, debugging outside

of weewx is likely more expedient, since it feels like a you verses web hosting service issue.  You should have the rsync

command lines from the logs when debug is set > 0.  Just run those a few times in a row (try to work around weews's

time slot) and see what results you get.

Chris