Rate limits/ Requests in queue

[Luke Scott]

Hiya!
I just want to say a huge thank you to makers and maintainers of this service, as it has been vital in my masters project work (via the astroquery module sending source lists via the api).
Slowly over the last week or so, I've noticed a gradual increase in rate limiting (or what i can only assume is rate limiting), which i've counteracted using a 10 second cool down between requests or submission checkups.
Today however, I've had images sit in a queue indefinitely, which is new. I am operating off the university network currently, so I am unsure if it is limiting based on the sum traffic coming from the Uni, or if it is my account specifically?
Any thoughts or advice would be hugely welcome, as I'm now stuck unable to continue my masters work

Luke

[Dustin Lang]

Hi,

This is two totally separate things --

- the web server does rate limiting.  This is grouped by IP address, and it limits to 5 requests/second per URL, and 50 requests per 10 seconds across the whole site.  Depending on the network setup, many users may be coming from the same IP.  I think you should now get an HTTP error code 429 to indicate rate limiting.

- once an image is submitted, it goes into a queue to be solved.  Currently there seems to be a large backlog -- 1700 images from 170 distinct users.

(I think I caused some of this, because I was running a process to compress our database backups, but it looks like that took a huge amount of memory and caused things to grind to a halt for a couple of hours...)

The backlog should clear over the next few hours.  The rate limiting is here to stay.

What exactly are you doing?  You're batch processing some big data set?  Using some script to submit things via the API?  Images?  xy-lists?

cheers,

dustin

--
You received this message because you are subscribed to the Google Groups "astrometry" group.
To unsubscribe from this group and stop receiving emails from it, send an email to astrometry+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/astrometry/a0494e69-a06d-48f2-a34e-eeb0b74ac62en%40googlegroups.com.

[Luke Scott]

Im on my home wifi right now and just used it for the first time this evening, immediately had the error in my code and on browser a 429 error

I haven't touched it for hours, and no one is on my network except for my flatmate who's definetly not spamming requests to the website.

I'm processing some old comet imagery, and I am locally performing source extraction and uploading those lists to the server with a 10 second cool down between request (i used to not need this). I'm only trying to plate solve a handful of images and I never got this for the few months I've been working on this project. 
I am using the astroquery library to send things via the API, and again it has been working flawlessly until the last few weeks, and I am very mindful of not spamming requests, so it took me by suprise. My earlier Images have now made it through the queue. Just a bit frustrating to pin down the issue as even if it eventually solves the image web server side, my code has already terminated and quit, and trying to download the WCS from the site and match the exact image in the request out of the huge library i've got saved would be a nightmare...

[Dustin Lang]

Hi,

You can try again now -- it turns out that the mod_evasive module I was using ... doesn't exactly do what it says it does.  Instead of limiting to N requests within T seconds, it actually requires a gap of at least T seconds to reset its counter, and if the counter ever hits N, you get blocked.

I now switched to the much more complicated mod_qos, where it is supposed to be limiting to 100 requests per minute.

cheers,

dustin

[Paul Leyland]

May I suggest that others do what I do?

Install astronomy.net locally. This has at least two major benefits: there is zero load on the public server and turn-around time is generally *much* shorter.

Paul

On 20/11/2025 14:12, astro...@googlegroups.com wrote:

astro...@googlegroups.com

Google Groups

Topic digest
View all topics

• Rate limits/ Requests in queue - 1 Update

Rate limits/ Requests in queue

Dustin Lang <dstn...@gmail.com>: Nov 19 01:17PM -0500

Back to top

You received this digest because you're subscribed to updates for this group. You can change your settings on the group membership page. To unsubscribe from this group and stop receiving emails from it send an email to astrometry+...@googlegroups.com.

[Bryan]

Ditto

Bryan

[Bryan]

You can also plate solve when you do not have an Internet connection.  You can fine-tune the config file to use only those index files that match your particular camera/scope setup  well.  Solutions are in seconds, not minutes.

Bryan