Realtime-_gauge-data.txt

[philip....@gmail.com]

Hi Guys

Ive tried and failed to get the rsync to work using realtimeguagedata

I never get passed gauge-data.txt file will not be exported.

Here is an extract of the log.

May 17 11:38:34 raspberrypi weewx-weewx[352] INFO user.rtgd: version is 0.5.5

May 17 11:38:34 raspberrypi weewx-weewx[352] INFO user.rtgd: RealTimeGaugeData scroller text will use a fixed string
May 17 11:38:34 raspberrypi weewx-tides[363] INFO weewx.engine: Using archive interval of 600 seconds (specified in weewx configuration)
May 17 11:38:34 raspberrypi weewx-weewx[352] INFO user.rtgd: gauge-data.txt will not be exported.
May 17 11:38:34 raspberrypi weewx-weewx[352] INFO user.rtgd: '/home/weewx/public_html/gauge-data.txt' will be generated. min_interval is 15 seconds

I have the following in the weewx.conf file

 rsync_server = SynLusher
  rsync_user = guest
  rsync_remote_rtgd_dir = /volume1/web/SheringhamDDNS
   rsync_log_success = True

weewx is version 4.5.1

The HTTP POST is not being used.

I have set debug = 0, 1, 2 and 3 to try and find where it might be failing but its always the same.

The server is a Synology NAS and I can copy from the rasberry pi to the rtgd_dir OK

Just wondered if its possible to get more logs so give me a clue to where is failing

Thanks

Phil

[philip....@gmail.com]

Just to update

Have a cron job working Ok with

rsync -v /home/weewx/public_html/gauge-data.txt Lush...@xxx.xxx.x.xxx:/volume1/web/SheringhamDDNS so the key/passwordless is working

Changed

 rsync_server = xxx.xxx.x.xxx
 rsync_user = LushEr21

And it still doesnt work

If somebody has tried this and got it to work your ideas on why I cant get rsync to work with RTGD would be most welcome

Thanks 

[John Kline]

You need to specify the rsync parameters in the RealtimeGaugeData section of weewx.conf.  Are you doing that?  Perhaps send the RealtimeGaugeData section of weewx.conf so we can have a look.  From your previous emails, it isn’t clear that you have the parameters in the correct section.  I can tell you that this worked for me for a long time (in fact, I submitted the pull request for this functionality).

# Fill out the following fields:

#   rsync_server             : The server to which gauge-data.txt will be copied.

#   rsync_user               : The userid on rsync_server with write

#                              permission to rsync_remote_rtgd_dir.

#   rsync_remote_rtgd_dir    : The directory on rsync_server where

#                              gauge-data.txt will be copied.

#   rsync_compress           : True to compress the file before sending.

#                              Default is False.

#   rsync_log_success        : True to write success with timing messages to

#                              the log (for debugging).  Default is False.

#   rsync_ssh_options        : ssh options Default is '-o ConnectTimeout=1'

#                              (When connecting, time out in 1 second.)

#   rsync_timeout            : I/O timeout. Default is 1.  (When sending,

#                              timeout in 1 second.)

#   rsync_skip_if_older_than : Don't bother to rsync if greater than this

#                              number of seconds.  Default is 4.  (Skip this

#                              and move on to the next if this data is older

#                              than 4 seconds.

On May 19, 2022, at 6:37 AM, philip....@gmail.com <philip....@gmail.com> wrote:

Just to update

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/39fff649-3aa4-4a7b-aab5-cd6628080faen%40googlegroups.com.

[philip....@gmail.com]

[RealtimeGaugeData]
    date_format = %Y.%m.%d %H:%M
    rtgd_path = /home/weewx/public_html
    min_interval = 15
    response_text = success

    rsync_server = xxx.xxx.x.xxx
    rsync_user = LushEr21

    rsync_remote_rtgd_dir = /volume1/web/SheringhamDDNS
    rsync_compress = False
    rsync_log_success = True
    rsync_ssh_options = "-o ConnectTimeout=1"
    rsync_timeout = 1
    rsync_skip_if_older_than = 4    

The key/passwordless to the server works with

rsync -v /home/weewx/public_html/gauge-data.txt Lush...@xxx.xxx.x.xxx:/volume1/web/SheringhamDDNS so the key/passwordless so am assuming the above works

Thanks for your help

Phil

[John Kline]

I believe the rsync_xxx entries need to be under an Rsync section:

[RealtimeGaugeData]

.

.

.

  [Rsync]

    rsync_server=xxx

    Rsync_user=LushEr21

    etc.

If that doesn’t work, I will install the extension and get it working.

On May 21, 2022, at 3:18 AM, philip....@gmail.com <philip....@gmail.com> wrote:

[RealtimeGaugeData]

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/a68f9075-80c3-4980-a3bc-e80f3dfb2ce1n%40googlegroups.com.

[John Kline]

BTW, this is the commit that moved the rsync parms (and, it looks like HTTP post parms) into a separate section:

https://github.com/gjr80/weewx-realtime_gauge-data/commit/534032cef8f1bfba45b90cf7ef939c16084a5b2e0

The README should probably change to match.

On May 21, 2022, at 6:48 AM, John Kline <jo...@johnkline.com> wrote:



[philip....@gmail.com]

Thanks John

Can I just confirm how many bracket Rsync would have please 

Have tried 1 bracket and got lots of log failures and then tried it with 2 brackets as [[StringFormats]] below has 2 brackets , and got a different failures.

Both were critical errors and stopped RTGD from working.

In both cases the line raspberrypi weewx-weewx[352] INFO user.rtgd: gauge-data.txt will not be exported. had gone.

Phil

[John Kline]

Oh, yes, two brackets.  What failures did you get with two brackets?

On May 22, 2022, at 2:59 AM, philip....@gmail.com <philip....@gmail.com> wrote:

Thanks John

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/dbca6111-df73-45be-8d3f-487a90e8bb7en%40googlegroups.com.

[philip....@gmail.com]

Just find out 1 bracket stops weewx

With 2 brackets I get 

raspberrypi weewx-weewx[3036] DEBUG user.rtgd: rtgdthread: **** Traceback (most recent call last):
May 22 14:40:17 raspberrypi weewx-weewx[3036] DEBUG user.rtgd: rtgdthread: ****   File "/home/weewx/bin/user/rtgd.py", line 1824, in run
May 22 14:40:17 raspberrypi weewx-weewx[3036] DEBUG user.rtgd: rtgdthread: ****     self.process_packet(_package['payload'])
May 22 14:40:17 raspberrypi weewx-weewx[3036] DEBUG user.rtgd: rtgdthread: ****   File "/home/weewx/bin/user/rtgd.py", line 1903, in process_packet
May 22 14:40:17 raspberrypi weewx-weewx[3036] DEBUG user.rtgd: rtgdthread: ****     self.exporter.export(data)
May 22 14:40:17 raspberrypi weewx-weewx[3036] DEBUG user.rtgd: rtgdthread: ****   File "/home/weewx/bin/user/rtgd.py", line 1425, in export
May 22 14:40:17 raspberrypi weewx-weewx[3036] DEBUG user.rtgd: rtgdthread: ****     packet_time = datetime.datetime.fromtimestamp(data['dateTime'])
May 22 14:40:17 raspberrypi weewx-weewx[3036] DEBUG user.rtgd: rtgdthread: **** KeyError: 'dateTime'
May 22 14:40:17 raspberrypi weewx-weewx[3036] CRITICAL user.rtgd: Thread exiting. Reason: 'dateTime'

Phil

[John Kline]

Never mind.  I installed it and found the issue.  It’s been broken since it was refactored.

Below is the patch to get it working.  I’ll also follow up with a URL to grab the file from github; as well as a PR to Gary (who may take it if he notices it).

diff --git a/bin/user/rtgd.py b/bin/user/rtgd.py

index 9f631c9..09ddd24 100644

--- a/bin/user/rtgd.py

+++ b/bin/user/rtgd.py

@@ -1300,7 +1300,7 @@ class HttpPostExport(object):

         # response text from remote URL if post was successful

         self.response = post_config_dict.get('response_text', None)

 

-    def export(self, data):

+    def export(self, data, dateTime):

         """Post the data."""

 

         self.post_data(data)

@@ -1419,10 +1419,11 @@ class RsyncExport(object):

         self.rsync_skip_if_older_than = to_int(rsync_config_dict.get('rsync_skip_if_older_than',

                                                                      4))

 

-    def export(self, data):

+    def export(self, data, dateTime):

         """Rsync the data."""

 

-        packet_time = datetime.datetime.fromtimestamp(data['dateTime'])

+        log.info("export: data: %r" % data)

+        packet_time = datetime.datetime.fromtimestamp(dateTime)

         self.rsync_data(packet_time)

 

     def rsync_data(self, packet_time):

@@ -1900,7 +1901,7 @@ class RealtimeGaugeDataThread(threading.Thread):

                     self.last_write = time.time()

                     # export gauge-data.txt if we have an exporter object

                     if self.exporter:

-                        self.exporter.export(data)

+                        self.exporter.export(data, packet['dateTime'])

                     # log the generation

                     if weewx.debug == 2:

                         log.info("gauge-data.txt (%s) generated in %.5f seconds" % (cached_packet['dateTime'],

On May 22, 2022, at 6:08 AM, John Kline <jo...@johnkline.com> wrote:



[John Kline]

OK, you could also replace rtgd.py (in weewx’s bin/user directory) with the file at:

https://github.com/chaunceygardiner/weewx-realtime_gauge-data/blob/master/bin/user/rtgd.py

I’ve also sent Gary a PR at:

https://github.com/gjr80/weewx-realtime_gauge-data/pull/28

On May 22, 2022, at 10:07 AM, John Kline <jo...@johnkline.com> wrote:



[philip....@gmail.com]

Thanks John

 I now get 

May 23 10:59:03 raspberrypi weewx-weewx[1438] ERROR weeutil.rsyncupload: [['rsync', '--archive', '--stats', '--timeout=4', '-e ssh -o ConnectTimeout=1', '/home/weewx/public_html/gauge-data.txt', 'Lush...@192.xxx.x.xxx:/volume1/web/SheringhamDDNS/gaugedata/gauge-data.txt']] reported errors: ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory. [sender] io timeout after 4 seconds -- exiting. rsync error: timeout in data send/receive (code 30) at io.c(204) [sender=3.1.3]
May 23 10:59:03 raspberrypi weewx-weewx[1438] INFO weeutil.rsyncupload: rsync executed in 6.03 seconds

I had to extend the Timeout = 4 as it would time out.

I dont appear to have a file usr/bin/ssh_askpass When the passwordless key was set up I believe it saved the info to ssh-keygen

rsync -v /home/weewx/public_html/gauge-data.txt Lush...@192.xxx.x.xxx:/volume1/web/SheringhamDDNS works OK in a cron job

Phil

[John Kline]

I don’t recommend extending the timeout.  It’s there so that requests don’t build up.

Are you sure you have ssh set up correctly?  In particular, did you set it up for the same user as weewx is running as?  That’s often root.  If weewx is running as root for you, please try to ssh in without a password when you are logged in as root.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/fd71e6f8-fa8b-4fab-bfe9-e9d9d0abd6cen%40googlegroups.com.

[philip....@gmail.com]

For ssh 

I used when I was root

ssh-keygen -f C:\Users\<username>\.ssh\id_rsa -t rsa -P ""

ssh-copy-id -i ~/.ssh/id_rsa.pub <username>@<server-ip-address> -o StrictHostKeyChecking=No

Should I have a file /usr/bin/ssh-askpass

Sorry to be a pain

Phil

[John Kline]





I’m assuming that <username> in ssh-copy-id -i ~/.ssh/id_rsa.pub <username>@<server-ip-address> - is:

LushEr21

Since that what you specified in weewx.conf.

Please confirm that the following works (i.e., you are logged in to <server-ip-address> as LushEr21 without having to enter a password or an [empty] passphrase:

(As root:)

ssh LushEr21@<server-ip-address>

On May 23, 2022, at 7:00 AM, philip....@gmail.com <philip....@gmail.com> wrote:

For ssh 

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/2db45023-d08b-4691-8609-e43b5ce2fba6n%40googlegroups.com.

[philip....@gmail.com]

Im using LushEr21 for both and ssh LushEr21@server-ip-address logs on to the server without a password

[John Kline]

I’m curious, are you using rsync to publish weewx?  I’m asking because rtgd is calling the same weewx method.

If you are using rsync to publish weewx, please send that section of your weewx.conf.  There must be some difference.

Also, what OS are you using?  I’m worried that it is Windows from your paths.  If it is, I’m unlikely to be able to debug this.

On May 23, 2022, at 10:56 AM, philip....@gmail.com <philip....@gmail.com> wrote:

Im using LushEr21 for both and ssh LushEr21@server-ip-address logs on to the server without a password

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/3af547de-13a5-4e41-9bb0-aadf108b4a52n%40googlegroups.com.

[philip....@gmail.com]

Im using rsync to copy the public_html folder to my server using a cron job every 10mins ===== rsync -a /home/weewx/public_html/ Lush...@192.xxx.x.xxx etc etc

Then thought I would use rsync in rtgd, from weewx.conf to update the gauges in real time.

OS is Buster 10 using a Raspberry Pi

[John Kline]

That rsync command you listed, it is in root’s crontab?

And why aren’t you doing the rsync in weewx?

On May 23, 2022, at 1:15 PM, philip....@gmail.com <philip....@gmail.com> wrote:

Im using rsync to copy the public_html folder to my server using a cron job every 10mins ===== rsync -a /home/weewx/public_html/ Lush...@192.xxx.x.xxx etc etc

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/0cf5d4d4-b044-420d-8808-306365b69b76n%40googlegroups.com.

[philip....@gmail.com]

No idea why I wasn't using Rsync in weewx.conf but it was over 3 years ago. 

Have changed it to the weewx.conf which didnt work at first so followed the example in USING RYSNC SKIN AS A BACKUP to create the password-less access.

Now backing up the public_html and RTGD now work, without the cron jobs.

I most have set it up as a non-root password although what threw me was that the cron job work.

Thanks for your help John

Phil