FTP Upload

[Karen K]

From the logs of the web server I realize that huge amounts of data are uploaded that did not change. Today I looked into the code of the FTP uploader. I saw that some kind of check is included which file needs to be uploaded and which not. This is done by comparing the file timestamp.

As the Cheetah generator re-creates a file every time it runs, unfortunately the file timestamp does not say anything whether the file content changed or not.

So I wonder whether it it would be better to calculate a hash of the file contents and use that hash instead of file timestamp to decide about upload.

The actual FTP Uploader does already maintain a file containing the file names of the uploaded files. So, may be, it is not that big step to include a hash into that file for comparison reasons.

An example for a file that seldom changes is the about page. It only changes if an update of WeeWX or some skin was installed for the version number. Other examples are imprint, privacy statement, FAQs, or other general information.

[Tom Keffer]

Good idea. A PR would be welcome.

If you have access to rsync, it effectively does this. it is a much faster alternative to FTP.

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/1282406b-4ce2-46bb-a0ae-4c47d68e9d28n%40googlegroups.com.

[Ryan Stasel]

Thank you for mentioning rsync option Tom! I had completely forgotten this or missed it when I first set up weewx and yes indeed. It’s about an order of magnitude faster (about 18 seconds for for ftp upload and only 2sec or so for rsync). 

Thanks!

On Mar 7, 2021, at 07:27, Tom Keffer <tke...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/CAPq0zECvGedUCJSs9W%2B8N%2BX6r%3DRSEkGg3gNwfATOkQfx7TNCBg%40mail.gmail.com.

[Karen K]

Unfortunately, my provider does not offer rsync.

[Ryan Stasel]

Can you SSH to your host?

On Mar 9, 2021, at 00:50, Karen K <kk44...@gmail.com> wrote:

Unfortunately, my provider does not offer rsync.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/b7edcbb5-0e53-47b0-93b9-bc93dbf5f5e4n%40googlegroups.com.

[Karen K]

rcst...@gmail.com schrieb am Dienstag, 9. März 2021 um 15:45:26 UTC+1:

Can you SSH to your host?

Unfortunately, no.

 

[Ryan Stasel]

Wow. That’s unfortunate. FTP works great and has for a while for me! Adding the time stamp check would probably have the same results as rsync I would imagine. 

On Mar 9, 2021, at 06:54, Karen K <kk44...@gmail.com> wrote:

rcst...@gmail.com schrieb am Dienstag, 9. März 2021 um 15:45:26 UTC+1:

Can you SSH to your host?

Unfortunately, no.

 

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/978fd433-ba74-4d19-826b-992ddf4d6e9dn%40googlegroups.com.

[Joel Bion]

FTP is full of problems as far as security goes, not the least of which is passwords get sent as cleartext across the internet. This is one of many reasons why ftp/telnet are often disabled, and instead sftp and ssh are used instead. I am hoping Weewx offers SFTP uploads, but I never have checked. 

Sent from my iPhone

On Mar 9, 2021, at 7:14 AM, Ryan Stasel <rcst...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/46C3CF79-D53B-4B63-83F6-4A6677DEA780%40gmail.com.

[Ryan Stasel]

It does support SFTP. :)

On Mar 9, 2021, at 07:50, Joel Bion <jpb...@gmail.com> wrote:

FTP is full of problems as far as security goes, not the least of which is passwords get sent as cleartext across the internet. This is one of many reasons why ftp/telnet are often disabled, and instead sftp and ssh are used instead. I am hoping Weewx offers SFTP uploads, but I never have checked. 

[Tom Keffer]

On Tue, Mar 9, 2021 at 7:52 AM Ryan Stasel <rcst...@gmail.com> wrote:

It does support SFTP. :)

Out of the box, WeeWX supports FTPS, not SFTP. The latter is available as an extension. You must install pysftp in order for it to work.

FTPS also encrypts passwords, but it uses TLS, not SSH. 

-tk

[Les Niles]

The timestamp check isn’t as good.  When rsync decides to transfer a file, it does so by just sending the parts that have changed.  (Like running a diff/patch, but it works on binary files as well.)  The result is very little data is actually transferred for files that just have small changes.  

I did a little experiment: I’m running the belchertown and seasons skins.  In a typical cycle, rsync “transfers” — updates — 1.2-1.3MB of files.  The total data actually flowing on the network between the two machines is only half of this, 650K, and that includes all the handshaking, protocol overhead, file metadata, and even the SSL setup — everything above the TCP layer.  The update includes 340K of webcam jpegs that most likely have to be fully copied each time, so the remaining 1MB is updated with only 300K of network traffic. 

As invoked by weewx, rsync makes the update decision by looking for changes in file size and modification time.  Rsync has an option to make the decision by comparing checksums.  I tweaked rsyncupload.py to use this, and found that it didn’t make a significant difference in the amount of network data.  Apparently the contents of most of the files really does change.

  -Les

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/46C3CF79-D53B-4B63-83F6-4A6677DEA780%40gmail.com.

[Karen K]

Tom Keffer schrieb am Sonntag, 7. März 2021 um 16:27:58 UTC+1:

Good idea. A PR would be welcome.

I did it.