weewx ftp to Digital Ocean host not connecting

[Stephen Rogers]

I changed my server for hosting weewx reports from A2Hosting to Digital Ocean (Ubuntu 20.04).

I can't get weewx ftp to successfully connect to the new server.  This is from Raspberry Pi which wass running until recently fine when A2Hosting was the target.  I have combed through and made necessary adjustments to weewx.conf to accommodate the new host, e.g. target directory, port (DigitalOcean uses port 22), username, password etc.

Using Filezilla from the same Pi, no problem connecting to the target.  With weewx I have tried all permutations of active/passive, secure_ftp etc. in weewx.conf [[FTP]] settings since I don't actually know a couple of these details for the target.  Each time restart and wait 5 min for weewx to retry.  No dice.

I always get the following log error:

      ftpupload: Unable to connect or log into server : timed out

What else can I try to get this working again...?

Thanks, Stephen

[vince]

Usual answer applies - set debug=1 in weewx.conf, restart weewx, wait 10 minutes, and post the log for a couple passes through the 5 minute archive period.

A quick search showed one user a while back who had this issue on a pi when they had Weather Underground rapidfire enabled.  Their pi was just too slow to finish the FTP upload in time from within weewx.

FWIW, I'd recommend using the 'rsync' uploader rather than FTP.   It is 'much' faster.  Like amazingly much faster.  I'm running weewx on a little ARM box that is much slower than a pi and my rsync uploads to AWS finish in two seconds according to the syslog.

[Tom Keffer]

Agree with what vince said, except I would set debug=2. This will cause a lot of the FTP chatter to be put in the log. 

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/e6e73831-4ce1-4188-97cd-2cbb9da75ae7n%40googlegroups.com.

[Stephen Rogers]

I upped the debug level and below is a couple of 5-minute log output cycles.

Also of note, I installed vsftpd to give that a try, since it has good documentation of how to configure it and is highly regarded.  I attached vsftpd.conf for reference.  Filezilla connects fine from my desktop to the DO FTP server, using ftpuser instead of root.   Here is the weewx.conf ftp section as modified for vsftp.conf upgrade.

    [[FTP]]
        # FTP'ing the results to a webserver is treated as just another report,
        # albeit one with an unusual report generator!
        skin = Ftp
       
        # If you wish to use FTP, set "enable" to "true", then
        # fill out the next four lines.
        # Use quotes around passwords to guard against parsing errors.
        enable = true
        #user = root
        #password = BoomRoot1$z
        user = ftpuser
        password = boomrock
        server = intraprem.com    # The ftp server name, e.g, www.myserver.org
        path = /home/nashwoods/weather    # The destination directory, e.g., /weather
       
        # Set to True for an FTP over TLS (FTPS) connection. Not all servers
        # support this.
        secure_ftp = False
       
        # To upload files from something other than what HTML_ROOT is set
        # to above, specify a different HTML_ROOT here.
        #HTML_ROOT = /var/www/html/weewx
       
        # Most FTP servers use port 21
        port = 22
       
        # Set to 1 to use passive mode, zero for active mode
        passive = 0

-----

Neither the root user nor the new ftpuser works.  I feel like I've tried every permutation of ftp parameters...

Stephen

log output:

Jul 26 13:05:15 raspberrypi weewx[3712]: vantage: Getting archive packets since 2022-07-26 13:00:00 PDT (1658865600)
Jul 26 13:05:15 raspberrypi weewx[3712]: vantage: Gentle wake up of console successful
Jul 26 13:05:15 raspberrypi weewx[3712]: vantage: Retrieving 1 page(s); starting index= 1
Jul 26 13:05:15 raspberrypi weewx[3712]: manager: Added record 2022-07-26 13:05:00 PDT (1658865900) to database 'weewx.sdb'
Jul 26 13:05:15 raspberrypi weewx[3712]: manager: Added record 2022-07-26 13:05:00 PDT (1658865900) to daily summary in 'weewx.sdb'
Jul 26 13:05:15 raspberrypi weewx[3712]: vantage: DMPAFT complete: page timestamp 2022-07-17 15:55:00 PDT (1658098500) less than final timestamp 2022-07-26 13:05:00 PDT (1658865900)
Jul 26 13:05:15 raspberrypi weewx[3712]: vantage: Catch up complete.
Jul 26 13:05:15 raspberrypi weewx[3712]: reportengine: Running reports for latest time in the database.
Jul 26 13:05:15 raspberrypi weewx[3712]: vantage: Requesting 200 LOOP packets.
Jul 26 13:05:15 raspberrypi weewx[3712]: reportengine: Running report 'SeasonsReport'
Jul 26 13:05:15 raspberrypi weewx[3712]: vantage: Gentle wake up of console successful
Jul 26 13:05:15 raspberrypi weewx[3712]: reportengine: Found configuration file /etc/weewx/skins/Seasons/skin.conf for report 'SeasonsReport'
Jul 26 13:05:16 raspberrypi weewx[3712]: cheetahgenerator: using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras']
Jul 26 13:05:16 raspberrypi weewx[3712]: manager: Daily summary version is 2.0
Jul 26 13:05:16 raspberrypi weewx[3712]: restx: Wunderground-PWS: Published record 2022-07-26 13:05:00 PDT (1658865900)
Jul 26 13:05:16 raspberrypi weewx[3712]: restx: MQTT: Published record 2022-07-26 13:05:00 PDT (1658865900)
Jul 26 13:05:20 raspberrypi weewx[3712]: cheetahgenerator: Generated 8 files for report SeasonsReport in 4.92 seconds
Jul 26 13:05:21 raspberrypi weewx[3712]: manager: Daily summary version is 2.0
Jul 26 13:05:22 raspberrypi weewx[3712]: imagegenerator: Generated 14 images for SeasonsReport in 1.26 seconds
Jul 26 13:05:22 raspberrypi weewx[3712]: copygenerator: copied 5 files to /weewx_generated_reports/html/weewx
Jul 26 13:05:22 raspberrypi weewx[3712]: reportengine: Report 'SmartphoneReport' not enabled. Skipping.
Jul 26 13:05:22 raspberrypi weewx[3712]: reportengine: Report 'MobileReport' not enabled. Skipping.
Jul 26 13:05:22 raspberrypi weewx[3712]: reportengine: Report 'StandardReport' not enabled. Skipping.
Jul 26 13:05:22 raspberrypi weewx[3712]: reportengine: Running report 'FTP'
Jul 26 13:05:22 raspberrypi weewx[3712]: reportengine: Found configuration file /etc/weewx/skins/Ftp/skin.conf for report 'FTP'
Jul 26 13:05:22 raspberrypi weewx[3712]: ftpupload: Attempting connection to intraprem.com
Jul 26 13:06:02 raspberrypi weewx[3712]: ftpupload: Unable to connect or log into server : timed out
Jul 26 13:06:42 raspberrypi weewx[3712]: ftpupload: Unable to connect or log into server : timed out
Jul 26 13:07:22 raspberrypi weewx[3712]: ftpupload: Unable to connect or log into server : timed out
Jul 26 13:07:22 raspberrypi weewx[3712]: ftpupload: Attempted 3 times to connect to server intraprem.com. Giving up.
Jul 26 13:07:22 raspberrypi weewx[3712]: ftpgenerator: ftp'd 0 files in 120.66 seconds
Jul 26 13:07:22 raspberrypi weewx[3712]: reportengine: Report 'RSYNC' not enabled. Skipping.
                                                                            ^[[A^[Jul 26 13:10:15 raspberrypi weewx[3712]: vantage: Getting archive packets since 2022-07-26 13:05:00 PDT (1658865900)
Jul 26 13:10:15 raspberrypi weewx[3712]: vantage: Gentle wake up of console successful
Jul 26 13:10:15 raspberrypi weewx[3712]: vantage: Retrieving 1 page(s); starting index= 2
Jul 26 13:10:15 raspberrypi weewx[3712]: manager: Added record 2022-07-26 13:10:00 PDT (1658866200) to database 'weewx.sdb'
Jul 26 13:10:15 raspberrypi weewx[3712]: manager: Added record 2022-07-26 13:10:00 PDT (1658866200) to daily summary in 'weewx.sdb'
Jul 26 13:10:15 raspberrypi weewx[3712]: vantage: DMPAFT complete: page timestamp 2022-07-17 16:00:00 PDT (1658098800) less than final timestamp 2022-07-26 13:10:00 PDT (1658866200)
Jul 26 13:10:15 raspberrypi weewx[3712]: vantage: Catch up complete.
Jul 26 13:10:15 raspberrypi weewx[3712]: reportengine: Running reports for latest time in the database.
Jul 26 13:10:15 raspberrypi weewx[3712]: vantage: Requesting 200 LOOP packets.
Jul 26 13:10:15 raspberrypi weewx[3712]: reportengine: Running report 'SeasonsReport'
Jul 26 13:10:15 raspberrypi weewx[3712]: vantage: Gentle wake up of console successful
Jul 26 13:10:15 raspberrypi weewx[3712]: reportengine: Found configuration file /etc/weewx/skins/Seasons/skin.conf for report 'SeasonsReport'
Jul 26 13:10:15 raspberrypi weewx[3712]: cheetahgenerator: using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras']
Jul 26 13:10:15 raspberrypi weewx[3712]: manager: Daily summary version is 2.0
Jul 26 13:10:16 raspberrypi weewx[3712]: restx: Wunderground-PWS: Published record 2022-07-26 13:10:00 PDT (1658866200)
Jul 26 13:10:16 raspberrypi weewx[3712]: restx: MQTT: Published record 2022-07-26 13:10:00 PDT (1658866200)
Jul 26 13:10:18 raspberrypi weewx[3712]: cheetahgenerator: Generated 8 files for report SeasonsReport in 2.79 seconds
Jul 26 13:10:18 raspberrypi weewx[3712]: manager: Daily summary version is 2.0
Jul 26 13:10:19 raspberrypi weewx[3712]: imagegenerator: Generated 14 images for SeasonsReport in 1.27 seconds
Jul 26 13:10:19 raspberrypi weewx[3712]: copygenerator: copied 0 files to /weewx_generated_reports/html/weewx
Jul 26 13:10:19 raspberrypi weewx[3712]: reportengine: Report 'SmartphoneReport' not enabled. Skipping.
Jul 26 13:10:19 raspberrypi weewx[3712]: reportengine: Report 'MobileReport' not enabled. Skipping.
Jul 26 13:10:19 raspberrypi weewx[3712]: reportengine: Report 'StandardReport' not enabled. Skipping.
Jul 26 13:10:19 raspberrypi weewx[3712]: reportengine: Running report 'FTP'
Jul 26 13:10:19 raspberrypi weewx[3712]: reportengine: Found configuration file /etc/weewx/skins/Ftp/skin.conf for report 'FTP'
Jul 26 13:10:19 raspberrypi weewx[3712]: ftpupload: Attempting connection to intraprem.com
Jul 26 13:11:00 raspberrypi weewx[3712]: ftpupload: Unable to connect or log into server : timed out
Jul 26 13:11:40 raspberrypi weewx[3712]: ftpupload: Unable to connect or log into server : timed out
Jul 26 13:12:20 raspberrypi weewx[3712]: ftpupload: Unable to connect or log into server : timed out
Jul 26 13:12:20 raspberrypi weewx[3712]: ftpupload: Attempted 3 times to connect to server intraprem.com. Giving up.
Jul 26 13:12:20 raspberrypi weewx[3712]: ftpgenerator: ftp'd 0 files in 120.60 seconds
Jul 26 13:12:20 raspberrypi weewx[3712]: reportengine: Report 'RSYNC' not enabled. Skipping.
Jul 26 13:15:15 raspberrypi weewx[3712]: vantage: Getting archive packets since 2022-07-26 13:10:00 PDT (1658866200)
Jul 26 13:15:15 raspberrypi weewx[3712]: vantage: Gentle wake up of console successful
Jul 26 13:15:15 raspberrypi weewx[3712]: vantage: Retrieving 1 page(s); starting index= 3
Jul 26 13:15:15 raspberrypi weewx[3712]: manager: Added record 2022-07-26 13:15:00 PDT (1658866500) to database 'weewx.sdb'
Jul 26 13:15:15 raspberrypi weewx[3712]: manager: Added record 2022-07-26 13:15:00 PDT (1658866500) to daily summary in 'weewx.sdb'
Jul 26 13:15:15 raspberrypi weewx[3712]: vantage: DMPAFT complete: page timestamp 2022-07-17 16:05:00 PDT (1658099100) less than final timestamp 2022-07-26 13:15:00 PDT (1658866500)
Jul 26 13:15:15 raspberrypi weewx[3712]: vantage: Catch up complete.
Jul 26 13:15:15 raspberrypi weewx[3712]: vantage: Requesting 200 LOOP packets.
Jul 26 13:15:15 raspberrypi weewx[3712]: reportengine: Running reports for latest time in the database.
Jul 26 13:15:15 raspberrypi weewx[3712]: reportengine: Running report 'SeasonsReport'
Jul 26 13:15:15 raspberrypi weewx[3712]: vantage: Gentle wake up of console successful
Jul 26 13:15:15 raspberrypi weewx[3712]: reportengine: Found configuration file /etc/weewx/skins/Seasons/skin.conf for report 'SeasonsReport'
Jul 26 13:15:15 raspberrypi weewx[3712]: cheetahgenerator: using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras']
Jul 26 13:15:15 raspberrypi weewx[3712]: manager: Daily summary version is 2.0
Jul 26 13:15:16 raspberrypi weewx[3712]: restx: Wunderground-PWS: Published record 2022-07-26 13:15:00 PDT (1658866500)
Jul 26 13:15:16 raspberrypi weewx[3712]: restx: MQTT: Published record 2022-07-26 13:15:00 PDT (1658866500)
Jul 26 13:15:18 raspberrypi weewx[3712]: cheetahgenerator: Generated 8 files for report SeasonsReport in 2.68 seconds
Jul 26 13:15:18 raspberrypi weewx[3712]: manager: Daily summary version is 2.0
Jul 26 13:15:19 raspberrypi weewx[3712]: imagegenerator: Generated 14 images for SeasonsReport in 1.28 seconds
Jul 26 13:15:19 raspberrypi weewx[3712]: copygenerator: copied 0 files to /weewx_generated_reports/html/weewx
Jul 26 13:15:19 raspberrypi weewx[3712]: reportengine: Report 'SmartphoneReport' not enabled. Skipping.
Jul 26 13:15:19 raspberrypi weewx[3712]: reportengine: Report 'MobileReport' not enabled. Skipping.
Jul 26 13:15:19 raspberrypi weewx[3712]: reportengine: Report 'StandardReport' not enabled. Skipping.
Jul 26 13:15:19 raspberrypi weewx[3712]: reportengine: Running report 'FTP'
Jul 26 13:15:19 raspberrypi weewx[3712]: reportengine: Found configuration file /etc/weewx/skins/Ftp/skin.conf for report 'FTP'
Jul 26 13:15:19 raspberrypi weewx[3712]: ftpupload: Attempting connection to intraprem.com
Jul 26 13:16:00 raspberrypi weewx[3712]: ftpupload: Unable to connect or log into server : timed out
Jul 26 13:16:40 raspberrypi weewx[3712]: ftpupload: Unable to connect or log into server : timed out
Jul 26 13:17:01 raspberrypi CRON[3771]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 26 13:17:20 raspberrypi weewx[3712]: ftpupload: Unable to connect or log into server : timed out
Jul 26 13:17:20 raspberrypi weewx[3712]: ftpupload: Attempted 3 times to connect to server intraprem.com. Giving up.
Jul 26 13:17:20 raspberrypi weewx[3712]: ftpgenerator: ftp'd 0 files in 120.51 seconds
Jul 26 13:17:20 raspberrypi weewx[3712]: reportengine: Report 'RSYNC' not enabled. Skipping.
Jul 26 13:20:15 raspberrypi weewx[3712]: vantage: Getting archive packets since 2022-07-26 13:15:00 PDT (1658866500)

[Stephen Rogers]

Vince:  I will also try RSYNC based on your recommendation but I'd like to get through this first since it has worked fine for me for a few years.  Since FTP is working fine from Filezilla on multiple machines it seems like the issue must be on the weewx side somewhere.

[Tom Keffer]

If I understand correctly, you're getting the same symptoms with two different FTP servers. Correct?

1. What version of WeeWX? Judging from the logging style, it looks like it's pre v4.0.

2. Any particular reason why you're using FTP active mode?

If you wish, you can send me your login credentials and I'll try and debug it on my end.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/cbe53951-9257-401b-80da-514803cb0e60n%40googlegroups.com.

[Stephen Rogers]

Thanks for the help, Tom.  

As for FTP servers, first I tried the FTP server that "came with" the Ubuntu 20.04 distro that digital ocean installs.  Today to widen my debugging scope I decided to try vsftpd and installed it, updated its config file (which I can't find how to control for the built-in ftp I'd been using).  weewx hasn't worked with either, same inability to connect.  Filezilla works with both.

weewx version from the .conf = 3.9.1

I haven't updated Weewx since I started using it for the Davis weather station about 3 years ago.  It's been working without a hitch.

DO Server:

intraprem.com and nashwoods.com point to the server droplet (Ubuntu 20.04).

root user: root

root pwd:   BoomRoot1$z

ftp user: ftpuser

ftp user pwd: boomrock

ftp port is 22 which means it uses SSH I think.  I don't have a ton of experience with managing devops.

I don't have a special reason for using active mode.  That is what I've been using on this server (which  is also my dev server for Intraprem, the water tracking system I've been working on for the last couple of years.  I haven't had need for anything other than Filezilla until I needed to retarget weewx reports.

Stephen

        # FTP'ing the results to a webserver is treated as just another report,
        # albeit one with an unusual report generator!
        skin = Ftp
       
        # If you wish to use FTP, set "enable" to "true", then
        # fill out the next four lines.
        # Use quotes around passwords to guard against parsing errors.
        enable = true
        #user = root
        #password = BoomRoot1$z
        user = ftpuser
        password = boomrock
        server = intraprem.com    # The ftp server name, e.g, www.myserver.org
        path = /home/nashwoods/weather    # The destination directory, e.g., /weather
       
        # Set to True for an FTP over TLS (FTPS) connection. Not all servers
        # support this.
        secure_ftp = False
       
        # To upload files from something other than what HTML_ROOT is set
        # to above, specify a different HTML_ROOT here.
        #HTML_ROOT = /var/www/html/weewx
       
        # Most FTP servers use port 21
        port = 22
       
        # Set to 1 to use passive mode, zero for active mode
        passive = 0

[vince]

you REALLY want to not post usernames/passwords here especially root.

Change your posted passwords immediately.  Really.

This is a bug reason why rsync is better.  It uses keypairs.

[Stephen Rogers]

My bad, passwords are changed.

Given that Tom tried to break through on ftp with my server and also had issues, that's enough of that pursuit.  I will read up on rsync and try that now.  Thank you for the advice Vince.