Correlation file

[Michael Zhang]

Hi everyone,

This might be a very stupid question, but I just can't find anything about it in the docs.  I ran solve-field on the command line, and plotted the resulting .corr file's x and y on the initial image.  The .corr file had 1300 stars, and all of the matched stars fell in big circle; there were no correlated stars in the rightmost 1/8 of the image or in the upper left & lower left corners.

Then I plotted the .axy and indx.xyls files, with around 4700 and 2200 stars respectively.  The stars in both files were distributed evenly (as they should be), and the WCS transform was so good that the star locations matched perfectly.  However, the stars around the right 1/8 of the image are not in the .corr file.

Why is this the case?  Does astrometry.net not use the stars at the edge to derive the WCS transform?  How do I get a .corr file containing all stars in the catalogue, or at least all stars in the indx.xyls file?

[Maik Riechert]

I'd like to join and also ask: Can I use the .corr file to determine how accurate the solution is (within some uncertainty)? If I understood correctly, it maps sources to reference stars and gives a kind of probability for each match.

[Dustin Lang]

Hmm, could you send an example?  There is some logic that tries to avoid extrapolating a match in one part of the image to areas of the image that are too far away, but I thought that was relaxed once it had matched more stars.  Maybe not.

As you've seen:
axy contains stars detected in your image, in x,y coordinates.
index.xyls are the x,y positions of reference stars (from the "index files", hence the "index" part)
corr contains matched stars in your image and the reference catalog; it has both x,y and ra,dec positions for both.

cheers,
--dustin

[Dustin Lang]

Yes, you could try that.  The match probability is not very smart; it does not adapt to the actual distribution of scatters it sees; it's based on the distance from the matched quadrangle and the assumed scatter of the input positions.

Some time I would love to sink some more time into getting the fine-tuning code working really well...

--dustin

[Maik Riechert]

Dustin Lang wrote:
> Yes, you could try that. The match probability is not very smart; it
> does not adapt to the actual distribution of scatters it sees; it's
> based on the distance from the matched quadrangle and the assumed
> scatter of the input positions.

In the end, I would try to recreate my brain logic when looking at the
red/green plots: If there are many perfect red/green matches across the
whole image (or in my case, across the whole masked image area), then I
know two things: a) the solution is very accurate, b) the lens
distortion profile I used for that image is of very good quality. That's
all I need (is perfect/is not perfect) and probably all I can really get
at the moment.

> Some time I would love to sink some more time into getting the
> fine-tuning code working really well...

Well, without distortion it already works quite well I would say. I
imagine it's really hard to handle distortion in general, especially
with the very general SIP model. Maybe have a look at the rather simple
lensfun models
(http://lensfun.sourceforge.net/manual/group__Lens.html#gaa505e04666a189274ba66316697e308e)
which only care about radius and don't handle x,y separately, e.g. poly3
has only one parameter, poly5 two, and the common ptlens model three. I
would guess that these models work for the majority of the uploaded images.

Maik

[Michael Zhang]

I created an example, but I also found a solution.

I attached a screenshot that shows the problem.  In circles are the stars in .corr; in red, the stars in indx.xyls.

The solution I found was to download a massive number of index files (far more than I actually need to solve the field) and run:

solve-field --verify input.wcs input.fits --continue --tweak-order 3

after running:

solve-field input.fits --tweak-order 3

The verification step seems to take stars out of every index file and put it in .corr.

@Maik Riechert: I did something like that yesterday.  The .corr file contains the x,y of detected stars (presumably as reported by the star detector), the RA/Dec of the stars they correspond to (calculated by transforming detected x,y to RA/Dec), the x,y of those stars (calculated by projecting RA/Dec from the catalogue back to image position), and the real RA/Dec (from the catalogue).  So you can plot field_x, from the star detector, to index_x, calculated by transforming catalogue RA/Dec, and see what the difference is.  The standard deviation of field_x - index_x that I get is 0.13 pixels--so low that it's probably limited by the star detector's accuracy and not the accuracy of the astrometric solution.

[Michael Zhang]

Actually, never mind about the last post; I didn't find a solution.  When I tried solve-field on an image very dense in stars, I get a lot of correlated stars in the upper-left quarter of the image, much fewer in the lower-left image, and none at all in the right half of the image.  How do I make astrometry.net use every star it finds to derive the WCS? 

Strangely, even the stars in the .axy file are heavily clustered towards the top of the image, with very few in the bottom half.  Why is this the case?  The stars are pretty much evenly distributed in this frame.

In my attachments: Screenshot-1.png shows the distribution of stars in .corr, while detected_stars.png shows the stars in .axy.

The commands I ran were:

solve-field --overwrite -p --scale-low 9 --scale-high 10 --scale-units degwidth --parity pos -N none --ra <RA> --dec <DEC> --radius 7 --tweak-order 3 <filename.fits>
solve-field -N none --continue -t 3 -q 0.01 -V <filename.wcs> filename.fits

[David W Hogg]

Try downsample = 2 or = 4 or higher in the options and see if you solve.  We don't deal well with very crowded fields.  Obviously this is a fixable problem!  And we accept pull requests.

On Sun, Apr 6, 2014 at 11:42 PM, Michael Zhang <mzz...@princeton.edu> wrote:

Actually, never mind about the last post; I didn't find a solution.  When I tried solve-field on an image very dense in stars, I get a lot of correlated stars in the upper-left quarter of the image, much fewer in the lower-left image, and none at all in the right half of the image.  How do I make astrometry.net use every star it finds to derive the WCS? 

Strangely, even the stars in the .axy file are heavily clustered towards the top of the image, with very few in the bottom half.  Why is this the case?  The stars are pretty much evenly distributed in this frame.

In my attachments: Screenshot-1.png shows the distribution of stars in .corr, while detected_stars.png shows the stars in .axy.

On Sunday, April 6, 2014 12:25:57 PM UTC-4, Michael Zhang wrote:

--
You received this message because you are subscribed to the Google Groups "astrometry" group.
Visit this group at http://groups.google.com/group/astrometry.

--
David W. Hogg ~ CCPP, NYU; MPIA ~ http://cosmo.nyu.edu/hogg/

[Dustin Lang]

I think there are some hard-coded limits on the number of stars that can be detected.  I'm not sure why you are seeing that strange pattern of stars.  Do you get the same if you include "--uniformize 0" on the solve-field command-line?

You could also try using "--use-sextractor" to use SourceExtractor rather than our star detector.  It is far more configurable; ours is probably more robust given arbitrary input images.

cheers,
--dustin

[Michael Zhang]

Yes, I get the same problem with --uniformize 0.  Downsampling by a factor of 2 didn't help either, and I'm wary of downsampling more because the stars only have a FWHM of 2 pixels or so.  I've thought about using SourceExtractor, but since my team has developed a custom star detector, maybe I'll use that instead.

I took a glance at the code of simplexy, but couldn't spot the problem.  Maybe I'll do some debugging later and see why it's producing this pattern of stars.

Thanks for all your help!

[David W Hogg]

wary or not, it is useful for debugging to know what happens as you downsample further.

--
You received this message because you are subscribed to the Google Groups "astrometry" group.
Visit this group at http://groups.google.com/group/astrometry.

[Michael Zhang]

I finally got around to debugging this issue, and managed to fix half of it.  I'm stuck on the second half and would really appreciate some help.

The first problem is that simplexy detects only a fraction of the stars in images of high stellar density.  This is because when the image has too many stars, the blobs of pixels that simplexy associates with stars start becoming all connected to each other.  In my case, one blob covered half the image, and that blob was ignored because SIMPLEXY_DEFAULT_MAXSIZE in simplexy.h was too low.  I changed that value to something ridiculously high, and got a second problem: exactly 10,000 stars were being detected, and they were all in the top 2/3 of the image.  This is because SIMPLEXY_DEFAULT_MAXNPEAKS is 10,000.  Changing it to a ridiculously high value made simplexy detect all 15,000 bright stars in the image.

After the star detector runs, the astrometry engine does a great job in producing a .corr file.  The stars in .corr are distributed uniformly across the image.  However, when I try to verify the .wcs file using:

solve-field -V filename.wcs -t 3 filename.axy -w 2601 -e 1734

All of the stars in the .corr file are in the left 40% of the image, and absolutely no stars are in the right 60% (!!)  If I randomly choose only 2000 of the 15000 detected stars and make solve-field verify on that subset, the problem disappears: the stars are uniformly distributed.  I suspect this is due to another hard-coded limit, but I just can't find anything that could possibly be related.  Anyone have ideas?

[Robert Siverd]

It sounds like the star-finder is sorting by X position and stopping after a fixed number of detections. Have you tried using SExtractor? This might provide a straightforward way to accomplish what you want.

Even with tons of stars, you can maintain very good spatial sampling by extracting many, sorting by brightness, and then take the top N you want. I use roughly this procedure with images spanning 26x26 degrees (many many stars) and it works very well.

-Rob

[Michael Zhang]

The star finder actually works fine; I fixed it, and it now finds all the stars uniformly across the image.  The problem is the "verification" step of solve-field, which only considers stars on the left half of the image.

I'll use your technique as a last resort.  What N do you use, or do you determine that case by case?

[Robert Siverd]

On Monday, May 26, 2014 12:16:18 PM UTC-5, Michael Zhang wrote:

The star finder actually works fine; I fixed it, and it now finds all the stars uniformly across the image.  The problem is the "verification" step of solve-field, which only considers stars on the left half of the image.

I'll use your technique as a last resort.  What N do you use, or do you determine that case by case?

The number of stars I need to solve the image varies but is usually somewhere between 300 - 1000. Solutions become hard with <~ 200 stars. In practice, there seems to be a trade-off between numbers and precision. More stars are helpful provided the centroids are reliable. Usually I just specify a SExtractor extraction threshold and use whatever it finds. I suspect the optimal number of stars varies from instrument to instrument. In terms of SExtractor config, a threshold of ~200-300 (in units of background RMS) works well for my highest quality images. For poor conditions, I drop the threshold until N >~ 250.

SExtractor has lots of options, some of which are quite useful. The "windowed" positional parameters (XWIN_IMAGE, YWIN_IMAGE) seem to be much more reliable than the normal X_IMAGE, Y_IMAGE, particularly for odd PSFs. You can also extract additional parameters to help clean up your list before astrometry.

My procedure is roughly:
1) Run SExtractor, extracting parameters: XWIN_IMAGE, YWIN_IMAGE, MAG_ISO, ELLIPTICITY, FWHM_IMAGE
2) Clean catalog:
-- eliminate detections with ELLIPTICITY > 0.5 (often saturated stars with blooms)
-- eliminate "stars" with FWHM <~ 1.5 (cosmic rays or SE fit failure)
3) Feed catalog to solve-field

Verification is something I do not currently do (but plan to start soon). If I understand correctly, verification works better with more stars as long as positions are reliable. I think the trick is to run verification with more (smaller scale) index files than you use for the initial solution to maximize matches.

Hope this helps,
Rob