FTP downloads are not finishing

[Chris]

I've seen this happening for several days. When I perform a download command, the file will start to download, then it stops downloading and never completes (usually around 300MB).

This is a simple command line FTP.  No errors are reported, but when I check the size of the file being downloaded, it stops updating at some point (before entire file is downloaded). 

[Michael McDonald]

Please try again now. We think that the culprit was a miscoded check script that would trigger an autofs cache flush/refresh when it encountered an unreadable directory anywhere the FTP tree. A user had uploaded dirs/files to their /pub folder that were not world readable and therefore was causing all systems to think that something was wrong and flush the cache every hour. 

The script has been corrected to now only *report* the error and not invoke the autofs restrart.

Please let us know if this fix lets your FTP downloads complete.

 

--
--
You received this message because you are a member of HYCOM.org
To ask a question, send an email to fo...@hycom.org

--

Michael McDonald
HYCOM.org Administrator

[Chris]

Thanks for your help, however I tried a couple of different times today, but still received the same results.

[Akio Hansen]

Hey everybody,

 

thanks for reporting the issue, because so far I thought my scripts were wrong or something similar. But I also still have the same issue and have actually big problems at downloading the data. With ncftp it works, but with a simple wget or python command, it breaks down after roughly 300 MB…

If I could help you, please let me know! A solution for the issue would be great!

 

Thank you in advance and all the best,

Akio

---
You received this message because you are subscribed to the Google Groups "HYCOM.org Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to forum+un...@hycom.org.

[Michael McDonald]

What internet connection are you downloading these files via (.edu/commercial/home)?

Can you try running a speedtest from our "Florida State University" node located in Tallahassee, FL?

https://www.dropbox.com/s/r1vurkq3d5vtu7l/Screenshot%202017-02-01%2008.40.47.png?dl=0

http://www.speedtest.net/

Then report back on your results (you can share/save the image of the speed test).

/mike

[Chris]

My server is commercial and hosted in a datacenter in Phoenix, Az. I have a 1 GBs public network interface. 

My speedtest is below

[Michael McDonald]

Wow! I cannot believe more HYCOM users have not piled on to this thread to complain. We now *clearly* see this issue and is likely related to a recent network change at our campus border.

We just tested/verified from the University of Florida and from the Navy. At both places, we see these stalled FTP downloads after a few hundred MB.

A temporary workaround would be to use the HTTP method that our THREDDS service provides.

e.g.,

http://tds.hycom.org/thredds/fileServer/datasets/GLBu0.08/expt_91.2/forecasts/hycom_glb_912_2017020100_t120_ts3z.nc

The equivalent FTP for this would be

ftp://ftp.hycom.org/datasets/GLBu0.08/expt_91.2/forecasts/hycom_glb_912_2017020100_t120_ts3z.nc

i.e., substitute "http://tds.hycom.org/thredds/fileServer/" for "ftp://ftp.hycom.org/" and you can download in the browser or via "wget" at the command line.

We sincerely apologize for this issue. We have elevated the problem to our Network/Security teams to hopefully rectify ASAP!

[Michael McDonald]

Our ftp.hycom.org servers have all been trust listed on our campus IPS as of Friday 5 PM EST (2017-02-03). Please let us know if this has *not fixed* the FTP data stalling issue (originally reported on Jan 30th). You should also see a slight boost in data transfer speeds as a result of the zero traffic inspection. We will be working this and next week on trust listing all our other data portals so that our users get the best access to the data we can offer.