Image Cutout Script No Longer Working

[Juan]

Hello,

I have a script I have used for a while now to download image cutouts that seems to stop working after the recent server outage. The bash script is:

xargs < "image_urls.txt" -P 8 -L 1 wget -q -O

where "image_urls.txt" contains the file name in the first column and the url for downloading the image in the second column. This has worked with 0 issues in the past and I wondering if something with the recent server outage (maybe a rate limit?) might be affecting it now.

[Dustin Lang]

What's the URL pattern you're using?

There is a rate limit (maybe 5/second?).  You'll get an HTTP error code (429, maybe?).

cheers,

dustin

--
You received this message because you are subscribed to the Google Groups "DECam Legacy Survey" group.
To unsubscribe from this group and stop receiving emails from it, send an email to decam-legacy-su...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/decam-legacy-survey/a4bc7761-752e-47d3-8729-49aa804460f0n%40googlegroups.com.

[Juan]

Hi Dustin,

Thanks for the quick reply!

Here are some rows from the txt file I am using:

2.jpg http://legacysurvey.org/viewer/cutout.jpg?ra=36.791325&dec=0.630722&pixscale=0.262&layer=sdss&size=128
3.jpg http://legacysurvey.org/viewer/cutout.jpg?ra=120.402008&dec=0.525899&pixscale=0.262&layer=sdss&size=128
4.jpg http://legacysurvey.org/viewer/cutout.jpg?ra=120.75692&dec=0.585627&pixscale=0.262&layer=sdss&size=128
5.jpg http://legacysurvey.org/viewer/cutout.jpg?ra=348.902557&dec=1.271853&pixscale=0.262&layer=sdss&size=128
6.jpg http://legacysurvey.org/viewer/cutout.jpg?ra=145.801208&dec=-0.211129&pixscale=0.262&layer=sdss&size=128
7.jpg http://legacysurvey.org/viewer/cutout.jpg?ra=188.974274&dec=0.079262&pixscale=0.262&layer=sdss&size=128
8.jpg http://legacysurvey.org/viewer/cutout.jpg?ra=145.196304&dec=-0.87998&pixscale=0.262&layer=sdss&size=128
9.jpg http://legacysurvey.org/viewer/cutout.jpg?ra=145.463226&dec=-0.844223&pixscale=0.262&layer=sdss&size=128
10.jpg http://legacysurvey.org/viewer/cutout.jpg?ra=145.618866&dec=-0.972686&pixscale=0.262&layer=sdss&size=128

The bash command I sent actually sets how many urls to try at once. It is the "8" value, but even if I lower it to "1" I get just empty files being downloaded. The script also only creates files for the number of urls to try at once, it seems to not continue to the other rows like it used to before.

[Dustin Lang]

Hi,

I just ran your script on my computer and it worked for me.

If you open those URLs in your browser, or just run a single wget from the command line (with the -q so you can see any error messages), does it work?

cheers,

dustin

To view this discussion visit https://groups.google.com/d/msgid/decam-legacy-survey/dbfe7642-2156-4af3-a3b6-88027b874a13n%40googlegroups.com.

[Juan]

Hello,

So I am downloading these images on a remote server and when I try to open the links on it in a browser it does not load and the connection times out (it does load on my laptop). And removing the -q I see I am getting 443 errors. I think maybe the ip of my server got flagged

[Dustin Lang]

Yeah, could be, NERSC network operations does that sometimes when they see what looks like DOS attacks.

To view this discussion visit https://groups.google.com/d/msgid/decam-legacy-survey/3251614b-7426-47cf-acb3-4855f48a4488n%40googlegroups.com.