Bypassing the latency of calling the solve-field binary?

76 views
Skip to first unread message

Akarsh Simha

unread,
Jun 22, 2021, 5:11:07 AM6/22/21
to astro...@googlegroups.com
Hello all,

I'm trying to use astrometry.net for a personal hobby project, and for my use case, I need the plate-solving process to be fast. However, I find that although the solving process itself is very snappy, the time it takes to load solve-field and transform the xylist to the format needed by the actual solve engine takes up a lot of the time. I'm aware of a project called StellarSolver that addresses this by providing direct C++ library bindings into astrometry.net -- but my language of preference is Python. I was wondering if there exists any Python bindings to directly feed an xylist into the astrometry.net engine without having to incur the overhead of making a system call.

Thank you!

Regards
Akarsh

Astro Keith

unread,
Jun 22, 2021, 6:38:51 AM6/22/21
to astrometry

Hi Akarsh,
I'm interested by what you mean as fast? I'm using a local copy of astrometry.net on my Pi and accessing it via a Python code and would say that it is fast by my standards.

Dustin Lang

unread,
Jun 22, 2021, 8:39:44 AM6/22/21
to Akarsh Simha, astrometry
Hi,
Try running solve-field with the "-v" verbose option to see where the time is going.
You probably want to try turning off a couple of steps:
--uniformize 0
--no-remove-lines
Converting xylists to axy files should be very fast.  It's really not the cost of a system call that you're seeing.
cheers,
--dustin


--
You received this message because you are subscribed to the Google Groups "astrometry" group.
To unsubscribe from this group and stop receiving emails from it, send an email to astrometry+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/CA%2B9k5twV1_QNJqNhH9%3DfLddFxr_jawY6UELeQFshYZ3aLCSzMA%40mail.gmail.com.

Akarsh Simha

unread,
Jun 22, 2021, 4:57:29 PM6/22/21
to Dustin Lang, astrometry
Thank you Dustin! You solved my problem -- The --uniformize 0 and --no-remove-lines options made the initial "wait" before solving vanish. I'm not sure yet what they do and will look up the docs.

Running with -v indicated that some time was spent in identifying 2MASS indexes which are not relevant for my problem. Now that I've removed all of them, I can solve some xylists in as little as 200ms!

Regards
Akarsh

Dustin Lang

unread,
Jun 22, 2021, 5:15:57 PM6/22/21
to Akarsh Simha, astrometry
Glad to hear it!

The "--uniformize 0" option turns off runing a python script that splits your image into boxes and selects one star from each box in turn.  This avoids problems if you have a large number of stars (or artifacts!) detected in one part of the image.
The "--no-remove-lines" option turns off a python script that tries to detect straight lines of sources (eg from bad columns in your detector, satellite trails, etc) in your image.

cheers,
--dustin

Alex Whittemore

unread,
Jul 31, 2023, 7:22:34 PM7/31/23
to astrometry
I'm on a similar journey and I thank you all for asking your excellent questions and stating your excellent answers in public!

Just to add a little to the discussion, it sure looks like the most time-intensive aspect of each of those process steps is merely importing the FITS library. I somewhat accidentally discovered that uninstalling astropy sped up the end-to-end solve process dramatically, it turns out because that library took something north of a second to import. Now that these steps are relying on PyFITS, they're quite a lot faster, but even that seems to take ~400ms to load!

With --no-remove-lines and --uniformize 0, I'm down to consistent sub-second JPG-input-to-orientation-out solves. 

@Akarsh, I suppose when you say that you can solve "some xylists in as little as 200ms" that's because you're feeding pre-extracted data and not waiting for image processing and source extraction? The best I've seen so far is ~450ms but of course maybe that's just my input image scale/catalog/etc. 

Dustin Lang

unread,
Jul 31, 2023, 7:30:22 PM7/31/23
to Alex Whittemore, astrometry
Hi all,

As an alternative to pyfits, I would also suggest looking at "fitsio", https://github.com/esheldon/fitsio, pip installable.  That being said, calling Python definitely does have some overhead too!

cheers,
dustin


--
You received this message because you are subscribed to the Google Groups "astrometry" group.
To unsubscribe from this group and stop receiving emails from it, send an email to astrometry+...@googlegroups.com.

Alex Whittemore

unread,
Jul 31, 2023, 7:33:18 PM7/31/23
to astrometry
Excellent info, thanks! Luckily I believe I can avoid it entirely and just hang with --uniformize 0 and --no-remove-lines :)

Dustin Lang

unread,
Jul 31, 2023, 7:34:42 PM7/31/23
to Alex Whittemore, astrometry
Yep, those really just clean up some edge cases.


Akarsh Simha

unread,
Jul 31, 2023, 8:14:52 PM7/31/23
to Alex Whittemore, astrometry

@Akarsh, I suppose when you say that you can solve "some xylists in as little as 200ms" that's because you're feeding pre-extracted data and not waiting for image processing and source extraction? The best I've seen so far is ~450ms but of course maybe that's just my input image scale/catalog/etc. 

That’s correct. An xy list is a list of (x, y) coordinates of sources with a flux value for sorting bright sources. So it is pre-extracted. I’m doing my source extraction using SEP, the Python front end to SExtractor.

And a scale hint is one of the best things you can give to astrometry.net if you are solving wide fields where you’re using the shallow 4100 indices and have enough RAM to place all the indices in memory and use the “inparallel” option. I think my measurement was on a Core i5 laptop and not a raspberry pi, I’ve noticed that it takes longer on a raspberry pi.

Of course, I’m not an expert here. Others are free to correct me where I’m wrong.

Regards
Akarsh



On Tuesday, June 22, 2021 at 2:15:57 PM UTC-7 dstn...@gmail.com wrote:
Glad to hear it!

The "--uniformize 0" option turns off runing a python script that splits your image into boxes and selects one star from each box in turn.  This avoids problems if you have a large number of stars (or artifacts!) detected in one part of the image.
The "--no-remove-lines" option turns off a python script that tries to detect straight lines of sources (eg from bad columns in your detector, satellite trails, etc) in your image.

cheers,
--dustin

On Tue, Jun 22, 2021 at 4:57 PM Akarsh Simha <akars...@gmail.com> wrote:
Thank you Dustin! You solved my problem -- The --uniformize 0 and --no-remove-lines options made the initial "wait" before solving vanish. I'm not sure yet what they do and will look up the docs.

Running with -v indicated that some time was spent in identifying 2MASS indexes which are not relevant for my problem. Now that I've removed all of them, I can solve some xylists in as little as 200ms!

Regards
Akarsh

--
You received this message because you are subscribed to the Google Groups "astrometry" group.
To unsubscribe from this group and stop receiving emails from it, send an email to astrometry+...@googlegroups.com.

Alex Whittemore

unread,
Jul 31, 2023, 8:38:18 PM7/31/23
to Akarsh Simha, astrometry
Thank you, super helpful! Of course I hadn’t thought about the intrinsic speed of the platform, that makes perfect sense. I also just came across SEP the first time today but hadn’t yet figured out what it was!

Out of curiosity, what’s your interest/purpose in controlling the pipeline yourself in python, vs simply calling “solve-field” with the right options to let it do everything?

I’d certainly love a good way to cut out the 250ms middleman of image2pnm. 

Alex
--

--
Alex Whittemore
617-213-0890
Reply all
Reply to author
Forward
0 new messages