Fwd: Error while trying to download lev1.fits images from successfull requests from jsoc.stanford.edu repository

[Felipe Meza]

Hello folks,

We’ve been experiencing an issue now that we’re trying to download the data from a successful request from jsoc.stanford.edu,

This issue is replicable in multiple environments and different requests,

We’re sending a request for 220 files for some trials, we can only download spikes.fits files (110 out of 220), all of image_lev1.fits files fail using the Fido.fetch method

In past runs, before April 3rd, 2023, We were able to download every single file successfully to work on them,

Yesterday, April 13th in the afternoon, while trying to replicate the error, we were able to download 22 spikes.fits and lev1.fits pairs out of 220 files, 110 pairs.

But then later that day, at nighttime, we weren’t able to download a single image_lev1.fits file again, and it’s still the same today.

Is this a known issue? If so, is there a workaround or fix?

Best regards,

[Nabil Freij]

Hello,

What is the error message?

We are not aware of any issues with downloading from the JSOC at the moment.

They recently up a proxy server and I think that now limits parallel downloads to 2.

[Felipe Meza]

Thanks Nabil, this is the message we are receiving:

110/0 files failed to download. Please check `.errors` for details

however it's been not possible to identify such .errors file (if exist). By the way If you folks have an idea about how to find this file, really appreciated If you can share how.

Returning to the original problem, it's an interesting behavior, first we are using the same date and attempt with different dates, but the problem remains, second we were able to download all the files several times like two weeks ago, third, sometimes we are able to download 2 or 5 files, but most if the time the result is the same, able to download the spikes.fits but not the image_lev1.fits files. 

We have tried to run this on weekdays, weekends in the morning, in the nights and nothing.

Any idea about what could be happening?

--
You received this message because you are subscribed to the Google Groups "SunPy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sunpy+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sunpy/b38e1a91-61c8-4adc-b53a-19255c14bdc6n%40googlegroups.com.

[Nabil Freij]

Hello,

`.errors.` refers to the attribute on the return from the fetch method, in your case, `download_files.errors` should tell you the errors.

Recently the JSOC added a proxy server in front of their servers, we noticed a bug in our JSOC code that was triggering 429s which could be the issue you are hitting.

When we confirm the bugfix is working, we will be doing a release soon. 

To view this discussion on the web visit https://groups.google.com/d/msgid/sunpy/CADhWRJCRyeLzAPepWL%3D4R9Sk1%3DnMMSW5rSKpu2FyvSpNhkHrSA%40mail.gmail.com.