Astrometry on small field

[Stefan]

Hello,

after several days of work I finally managed to install astrometry.net and think I got the right index files. But somehow it doesn't work. So here is what I've done. Hopefully you will find the mistake(s):

1.I got the coordinates and K-Fluxes for the field I'm interested in from the paper by Close et al. 2012. These I wrote into trapezium_close12.csv  so it looks like:
RA,DEC,FLUX_K,MAIN_ID
83.83,-5.34,4.57,* tet01 Ori C
and so on. The ones without a FLUX_K measured I entered 1 as default.

2. Converting to a .fits file: I used the ipython-console (based on 2.7):
In [1]: %run text2fits.py -s , -f fffs trapezium_close12.csv trapezium_close12.fits

3. Making the index file for astrometry. The output file i moved into /usr/local/astrometry/data/ where astrometry searches for it and finds it:

/usr/local/astrometry/bin/build-astrometry-index -i trapezium_close12.fits -o trapezium_close12_astrometry5.fits -E -P -5 -S FLUX_K -f -I 2016-03-24

4. I then run astrometry, using sextractor. Sextractor finds the right 5 sources in my image (I can look at the trapezium_agpm_reduced.png where they are circeled in red).

/usr/local/astrometry/bin/solve-field --overwrite  --use-sextractor --sextractor-config default.sex --x-column X_IMAGE --y-column Y_IMAGE --sort-column MAG_AUTO --sort-ascending --no-remove-lines trapezium_agpm_reduced.fits

5. I get the output:
****************************************************************************

/usr/local/astrometry/bin/solve-field --overwrite  --use-sextractor --sextractor-config default.sex --x-column X_IMAGE --y-column Y_IMAGE --sort-column MAG_AUTO --sort-ascending --no-remove-lines trapezium_agpm_reduced.fits Reading input file 1 of 1: "trapezium_agpm_reduced.fits"...
Header has 539 cards
/scisoft/python/lib/python2.7/site-packages/pyfits/card.py:1246: VerifyWarning: Card is too long, comment will be truncated.
  VerifyWarning)
Found an existing WCS header, will try to verify it.
Extracting sources...
----- SExtractor 2.19.5 started on 2016-03-24 at 16:37:33 with 1 thread

> Setting catalog parameters
> WARNING: lNUMBER catalog parameter unknown

----- Measuring from: tmp.sanitized.oEInDW
      "Object name not set" / no ext. header / 1024x1024 / 32 bits (floats)
(M+D) Background: -0.16479   RMS: 1.78735    / Threshold: 18.7672   
      Objects: detected 7        / sextracted 5              

> All done (in 0.0 s: 26837.8 lines/s , 131.0 detections/s)
Reading sort column "MAG_AUTO"
Sorting sort column
mmapping input file
Copying table header.
Writing row 0
Done
Solving...
Reading file "./trapezium_agpm_reduced.axy"...
Verifying WCS using indices with quads of size [0.0464043, 0.464043] arcmin
Verifying WCS with index 1 of 1 (/usr/local/astrometry/data/trapezium_close12_astrometry5.fits)
Verifying WCS of field 1.
 --> log-odds 0.881754
Got 0 solutions.
Field 1 did not solve (index trapezium_close12_astrometry5.fits, field objects 1-10).
Field 1 did not solve.
Field 1 did not solve.
Field 1 did not solve.
Field 1 did not solve.
Field 1 did not solve.
Field 1 did not solve.
Field 1 did not solve.
Field 1 did not solve.
Field 1 did not solve.
Field 1 did not solve.
Field 1 did not solve.
Field 1 did not solve.
Field 1 did not solve.
Field 1 did not solve.
Field 1 did not solve.
Field 1 did not solve.
Field 1 did not solve.
Field 1 did not solve.
Field 1 did not solve.
Field: trapezium_agpm_reduced.fits
Did not solve (or no WCS file was written).

******************************************************************************************************
So why doesn't it find any results? I guess it has something to do that I don't have an all-sky map (just some arcmin), or it looks for a wrong plate-scale. But also setting this to some predefined values (--scale-units arcsecperpix --scale-low 10.0 --scale-high 25.0) where it has to lie inbetween doesn't give a better result.

Thank you for your help,
Stefan

[Dustin Lang]

5 objects is not enough for Astrometry.net to find a match.  Usually 15-20 are required.

cheers,
--dustin

[Stefan]

Oh, this means there is no way to use astrometry.net for my purposes? I never have more than 5-10 targets. And I thought astrometry.net makes triangles (or rectangles, you can choose) to match :(

Well, thank you than. I'll probably write my own code then..
Thanks,
Stefan

[Dustin Lang]

For any quadrangle of stars, there is >= 1 match somewhere on the sky, so extra stars are needed to confirm matches.

Sorry...

--dustin

[Dustin Lang]

You know, one thing you could try quickly would be to add to your solve-field command,

solve-field --odds-to-solve 1.0

to lower the threshold for confirming a match.  If you add --verbose (maybe twice) it may tell you the log - odds ratios of matches it finds.  Or maybe even set --odds-to-solve to, say, 1e-6


If that doesn't work, it might be possible to change the Astrometry.net code so that it doesn't use any extra stars to confirm the match, but just accepts a match if it finds one. 

The place to change would be here,
https://github.com/dstndstn/astrometry.net/blob/master/blind/solver.c#L1331

which gets called each time we find a match between your image and the index.  That function is supposed to use extra stars to confirm the match.  You could change it to just always accept, ie, delete the lines where it returns FALSE if the match isn't good enough, lines 1387-8 and 1395-6 .

cheers,
--dustin

[Stefan]

Hey Dustin,

thanks a lot for your thoughts. So I played around with the --odds-to-solve and for values <= 3 I really get a result :) But unfortunately it is a very poor match (independent of the value only two stars match quite poorly). I check the quality of the fits by following the *-indv.png file (red and green circles showing sextractor identified ones and index stars) . What is strange is, that in the *-ngc.png there is even a star marked which was not identified by sextractor a a star ^^ (I also flipped the images in all directions before the match, but i just gives other stupid looking results :(  )
So probably you're right and I need to change the solver.c
However, I couldn't find this file in my astrometry folder...there was only a solver.h (which is required by solver.c), but it is completely different. So do I need to edit this file before the installation and then re-install? (I'm a bit afraid of this as it took the IT and me more than a day to get astrometry.net running on my machine and they will hate me if I blow it up again by re-installing ;)

Thanks again,
Stefan

[Dustin Lang]

Hi Stefan,

If the "-ngc.png" plot shows labelled stars where there is nothing in your image, you probably got a false match.  Remember how I said that you can match any quadrangle to some place on the sky?

You don't have any other index files in /usr/local/astrometry/data, do you?

For tiny fields, you need to tell solve-field to use a smaller lower-bound on the image scale -- its default is 6 arcmin.  For example, if your images are 3 arcminutes wide,

solve-field --scale-low 2.5 --scale-high 3.5 --scale-units amw   [...]

This will also reduce the possibility of false matches.

You may also want to repeat the indexing step you did with more scale presets (-P -4, -P -3) to build index files with different ranges of scales of quadrangles.

Oh, you may also want to try building index files using triangles rather than quadrangles:

build-astrometry-index -d 3 [...]

cheers,
--dstn

[Dustin Lang]

PS, I don't think you've ever said how big (small) your fields are.

[Stefan]

Hey Dustin,
the image has a FOV of about 28 arcsec ( 1024x1024 at a platescale of about 27 mas/ pix). So it is a very small one.
I already used
--scale-units arcsecperpix --scale-low 0.01 --scale-high 0.06
but changing numbers doesn't have an effect :(
Also -d 3 doesn't change a thing

But what I noticed when I retried to do the Index with only the 7 closest stars (out of which 5 are in the image), I think the problem is that the FOV is too small as -P - 5 still corresponds to a quad scale of 15 arcsec. Do you think this causes the problems? And if yes, what could I do as -P -6 is not available.

Output of build-astrometry_index :
********************************************************************************************
Nside=9957.  Nside^2=99141849.  Number of healpixes=1189702188.  Healpix side length ~ 0.353313 arcmin.
Star tree contains 7 objects.
Healpix radius 14.9898 arcsec, quad scale 15 arcsec, total 30.2897 arcsec
Scanning 7 input stars...
Will check 2 healpixes.
Pass 1 of 16.
Trying 2 healpixes.
.
Made 0 quads (out of 2 healpixes) this pass.
Made 0 quads so far.
Merging quads...
Pass 2 of 16.
Trying 2 healpixes.
.
Made 0 quads (out of 2 healpixes) this pass.
*******************************************************************************************

[Dustin Lang]

Hi,

Ooof, that is indeed a TINY field.


One side question: you said above that your CSV file contains values like:
83.83,-5.34,4.57,* tet01 Ori C
for RA,Dec, ...  Presumably you need WAY more digits of accuracy in your RA,Dec positions, right?

You are right that the preset scale of -5 is too big for your tiny images.  Unfortunately, it's not easy to support -6 because the number of healpixes becomes too large (I forget exactly, but it maybe overflows an int?).

There *is* something we can try, though.... would you mind sending your FITS reference catalog?

thanks,
--dustin

[Stefan]

Hey,

sure. You should have gotten a message.
Stefan

[Dustin Lang]

Just bringing part of this conversation back on list:

Hi,

After fixing up some old code, I got the 'allquads' program working again.  This just goes through a reference catalog and enumerates all possible quadrangles or triangles within certain scales.  It should work reasonably well for your indexing needs.  You will need a github checkout of the code if you want to play with it, and after you've built the code as usual, "cd blind && make allquads".

allquads -i trap.fits -o trap.index -I 42 -u 10 -l 0.1 -d 3

where trap.fits is the RA,Dec reference catalog, and "-l" and "-u" are the lower and upper bounds of quadrangles to build (or in this case, triangles because "-d 3").

cheers,
--dustin

[Sergio Velasco Muñoz]

Dear Dustin,

I am facing the same problem. I would like to know what to do after generating the .index file. Thanks.

[Dustin Lang]

Hi,

Once you have generated the .index file, you need to tell the solver to use it, by either creating a new astrometry.cfg file, or adding it to the directory where solve-field normally reads the index files.

To create the config file, create a text file "new.cfg" with:

inparallel
add_path /PATH/TO/INDEX/FILE
index FILENAME.index

or

inparallel
index /PATH/TO/FILENAME.index

or

inparallel
index FILENAME.index

if the file is in the current directory.

Then run

solve-field --config new.cfg  [... usual arguments...]


cheers,
--dustin

[Sergio Velasco Muñoz]

Dear Dustin,

I am trying to astrometrize a very small FoV image (30 arcsec^2) based on this catalogue:

http://vizier.cfa.harvard.edu/viz-bin/Cat?J/AJ/124/3255

which I have downloaded from vizier and computed the J2000 coordinates. I get it as a votable file and I edit the header of the _DEJ2000 and _RAJ2000 to DEC and RA
after that I use the votabletofits converter to get it as a fit file.

When I run the allquads with this options: -I 1 -u 1 -l 0.1 -d 3

I get the error:

  data 31982: ok 0, RA,Dec 322.504, 12.1426
Wrote star kdtree to allquads.skdt
Running allquads...
Reading star kdtree allquads.skdt ...
Star tree contains 31983 objects.
Will write to quad file allquads.quad and code file allquads.code
Warning: skdt does not contain "HEALPIX" header.  Code and quad files will not contain this header either.
quad-builder.c:167:quadbuilder_create: quad-builder: failed to malloc qb->inbox or qb->pquads.  Nstars=25833.

quad-builder.c:167:quadbuilder_create: quad-builder: failed to malloc qb->inbox or qb->pquads.  Nstars=25953.

I would like to know what I am doing wrong, as I need to astrometrize a region of this globular cluster.

Thanks a lot for your time.

[Dustin Lang]

Hi,

I'm not sure off-hand, but it may be trying to allocate memory for 25833 ^ 3 objects...  Do you need that many catalogue objects?

Would you be able to post the FITS table (with RA,Dec, not the original votable) so I ca try to reproduce and debug this problem?

How many objects are in your images?  Would you be able to post an image somewhere?  I'm just curious what your images look like.

Thanks,
--dustin