Problem with local photultil source detection

[RSAA Plates]

Hi,

I have several thousand images as FITS files (scans of old glass plates) and I would like to use astrometry.net to get WCS info for them. My approach has been to write a Python script that uses astroquery.astrometry_net.  I have installed photutil and I'm calling solve_from_image in my script like this:

                wcs_header = ast.solve_from_image(fits_file,
                                                  submission_id=submission_id,
                                                  downsample_factor=2, parity=2,                                                 
                                                  scale_units='degwidth', scale_type='ul', scale_lower=0.1, scale_upper=180.0)

The problem is that I can never get a successful outcome from the solver this way (it always times out).  However, I can upload one of my images and get a solution fairly quickly that way.  I have chosen the parameters in the above call to match those which seem to be the default for an image upload so I could compare the log files but this has not really helped me and I've decided I need an expert!

My image upload is job 8297115 and my attempt via my python script is job 8297662

I've tried setting values for FWHM and the plate scale but this has made no difference.  I suspect that photutil might be finding too many stars or else is finding sources that are really noise, but I have not been able to change the outcome by fiddling with parameters like detect_threshold and FWHM.

I tried installing sextractor and then using "use_sextractor=True" but this made no difference (I think I have realised that this setting only applies to image uploads?)

I assume that my approach is valid - i.e. that NOT uploading several thousand images is a good idea and that trying to locally extract sources is the best way to go.  Please let me know if there is a better way I could or should be doing this. Each image is about 200MB.

Regards,

Bill Roberts

[Dustin Lang]

Hi,

It looks like your uploaded source list has 36,027 stars in it.  That's *way* overkill for astrometry.net.  50-100 of the brightest stars is plenty.  The challenge will likely be that Astrometry.net cares about the *brightest* stars, AND you still need good centroids on them, even if they're saturated.  What we do in the web version, by default, is downsample (bin) the image 2x2, then run source detection, then scale the positions back up (carefully, because FITS pixel indexing).  (That's the --downsample 2 or downsample_factor=2 you have -- but that only applies to images, not source lists).  This tends to improve the detection & localization of saturated sources.  Maybe it will work for yours?  The other thing is, you *must* sort the sources to have the brightest ones first in the list that you upload.

Oh, and I would have expected that if you don't want to set one of the settings to a specific value, that you could leave it blank, ie you shouldn't need to set scale_lower=0.1 scale_upper=180 (the defaults).

Hope that helps!

-dustin

--
You received this message because you are subscribed to the Google Groups "astrometry" group.
To unsubscribe from this group and stop receiving emails from it, send an email to astrometry+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/f9b550fe-b533-41d2-afd1-e8fcb0224b96n%40googlegroups.com.

[RSAA Plates]

Thanks Dustin, that helps.  Maybe my problem is with the use of the astroquery.astrometry_net Python module.  I need to work out how to control the underlying use of photutil for source extraction.  I had assumed that the parameters in the call to solve_from_image would affect what photutil does (it certainly changes the number of stars detected, although not the end result when the table is uploaded to astrometry.net).  I'm pretty sure the Python module sorts the list correctly - I just have to work out how to shorten it considerably.

Thanks again for your response - it has helped with my understanding of what is going on.

Just a follow-up question (or three) as I'm new to all this and not an astronomer: Does what I'm doing make sense to you?  If you had several thousand scanned glass plates as FITS files with no RA/Dec etc, what would you do?  ie. what software would you use?  Maybe I've gone down the wrong path with astroquery.astrometry_net.

Cheers, Bill

[Dustin Lang]

Hi Bill,

That sounds like a totally reasonable approach to me!  Glass plate archives are like a perfect example of where Astrometry.net should be useful.

If I were you, I think I would try to continue fiddling with the FWHM and detect_threshold parameters (increase both of them considerably from their defaults which look to be FWHM=3, detect_threshold=5.  If you increase that to, say, 6 and 20, how many stars get detected?

( https://github.com/astropy/astroquery/blob/main/astroquery/astrometry_net/core.py#L323 )

If you do that -- maybe you have already -- then I would try to check whether the first, say, 50 sources, are actually finding the brightest stars you can see in the image.  If that's working, then it could be that you just need to cut down the number of stars submitted, which looks like it's not currently in the astroquery code, but shouldn't be too hard to add?

By the way, it looks like this is how they do the source detection in astroquery --

https://github.com/astropy/astroquery/blob/main/astroquery/astrometry_net/core.py#L400

-- not *too* complicated, so could try copying that and modifying it to your needs.

So, another option, if you are comfortable downloading and compiling code...  is to download the Astrometry.net code yourself and run it on your computer.  With relatively wide-angle images like the example one you sent, it looks like index scales 08 through maybe 12 would be required, so those files aren't too huge (ie 4108 through 4112 here -- http://data.astrometry.net/4100/).

Good luck, I'm sure you'll be able to get this working one way or another!

-dustin

To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/5df2299f-3f69-43e8-9a65-8dfad384f192n%40googlegroups.com.

[RSAA Plates]

Hi Dustin,

Thank you very much for those suggestions.  I had tried increasing the FWHM and detect_threshold but obviously not far enough!  Your suggested values are already giving me results.

If I continue to have issues, I will install the code.

Cheers, Bill