Unknown solve failure when using url

[Howard Edin]

I'm not exactly sure how to track down an issue I have run into, it appears to come from astrometry.net but I am open to any suggestions. I first encountered the problem using the Global Meteor Network utility SkyFit2. This program uses imgur to store the generated file and passes the URL to astrometry.net to solve (in code).

However it has become clear in testing that manually using a imgur image link fails for the images SkyFit2 would be using. 

One I just tested today, the image source example: https://i.imgur.com/Loj7gpc.jpeg

Resulting failure message:

...ile "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 429 Client Error: Unknown Error for url: https://i.imgur.com/Loj7gpc.jpeg

Another imgur example setup just for testing is : https://i.imgur.com/yWlIFYr.jpeg

Any thoughts on why this is happening? Thanks for any feedback.

Howard

[Dustin Lang]

Hi,

It looks like imgur is intentionally blocking retrieval with the "wget" command we use to fetch URLs.  I see that when I use my browser, it works, but with "wget" it fails with error "429 Unknown Error".  Oddly, it works with "curl".  And based on the help message from solve-field, curl is the default, and you can use wget instead by passing the "--use-wget" command.

The wget and curl utilities identify themselves to the imgur server using the "UserAgent" header.  If imgur is refusing to send image files based on the UserAgent setting, that's their choice.  You can work around it in different ways depending on which utility is being used.  Since you have only included a tiny snippet from the error message, I can't tell where that error is actually coming from.  I've never heard of skyfit2 so can't tell you any specifics.

cheers,

dustin

--
You received this message because you are subscribed to the Google Groups "astrometry" group.
To unsubscribe from this group and stop receiving emails from it, send an email to astrometry+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/043c2cfb-4710-495d-9cc8-e3068760da19n%40googlegroups.com.

[Howard Edin]

Thank you for feedback Dustin, the error snippet is coming from nova.astrometry.net so that is all that I could provide. It is the actual online https://nova.astronomy.net/upload website generating the error message, not our code. But I understand what you area saying: fault is at imgur in that they are rejecting the request to pull the image.

However, you are saying that curl is the default option and it works (for now). Is that for the source code or for the nova website? I'm not sure I understand as I don't think we have the ability to specify wget as an option (see below).

I am not the author of SkyFit2 but I am assisting in this case, since I found the error ;)  For reference SkyFit2 is a part of the Global Meteor Network, in the RMS station code here:  https://github.com/CroatianMeteorNetwork/RMS

I don't expect you to look at that, but for reference a small piece of the code in AstrometryNetNova.py is:

        # Upload the image to imgur
        image_url = imgurUpload('skyfit_image.jpg', image_data=img_data)

  c = Client()

    # Log in to nova.astrometry.net
    if api_key is None:
        api_key = API_KEY

    c.login(api_key)

    # Add keyword arguments
    kwargs = {}
    kwargs['publicly_visible'] = 'y'
    kwargs['crpix_center'] = True
    kwargs['tweak_order'] = 3

    # Add the scale to keyword arguments, if given
    if fov_w_range is not None:
        scale_lower, scale_upper = fov_w_range
        kwargs['scale_lower'] = scale_lower
        kwargs['scale_upper'] = scale_upper

    # Upload image or the list of stars
    if file_handle is not None:
        upres = c.url_upload(image_url, **kwargs)

    elif x_data is not None:
        upres = c.upload(x=x_data, y=y_data, **kwargs)

    else:
        print('No input given to the funtion!')

[Dustin Lang]

Thanks for the details.

I added a fake user-agent header and it now seems to work with your second example URL - I get this solution

https://nova.astrometry.net/user_images/9608319#annotated

cheers,

dustin

To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/38bf7d7b-0eab-4e11-90ba-2e1b856ba10cn%40googlegroups.com.

[Howard Edin]

Awesome Dustin!    Appreciate the adjustment, thanks.

Howard