An minimalistic build

[Danko Dnevic]

I'm trying to build a minimalistic Astrometry.net solver for Android. The input parameters are:

• XY source list (magnitude up to +5.0), sorted by decreasing flux,
• very wide field (68x53 degrees), index 4215+
• negative parity (starting in the upper left corner).

From the output data, I need:

• XY Ra Dec pairing, and probability
• Ra0 and Dec0 position of the optical axis on the tangent plane, and the rotation matrix
• SIP coefficients (polynomial  degree  3)

My questions are:

• What is the minimum An skeleton for the required functionality?
• What are the minimum package dependencies?
• Is there any earlier example of the described use?

[Dustin Lang]

Hi,

If you're starting from source lists (in FITS format), then you don't need any of the image conversion utilities, or netpbm.  The Makefiles should automatically detect this and do the right thing.  You *may* need to do "make -k" to continue even if some programs fail to build because dependencies are missing - but I don't think that should be necessary.  You also don't need python.

You will need to add to your solve-field command: --no-removelines --uniformize 0

to avoid trying to run python programs.

Other than that, what you describe is basically what is included in the core solver engine, so there is no real need to strip out anything.

Basically, this should just work.  Try it and find out and report back!

You could experiment by using Docker (docker/solver/Dockerfile), remove all the ubuntu packages except for probably build-essentials, make, gcc, file, pkg-config, git.

cheers,

dustin

--
You received this message because you are subscribed to the Google Groups "astrometry" group.
To unsubscribe from this group and stop receiving emails from it, send an email to astrometry+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/9f119bd7-68aa-47fb-9752-e90b7707c014n%40googlegroups.com.

[Danko Dnevic]

I would like to skip FITS, because sorted sources XY are already in the array. What would be solver entry point in that case? Also, I can collect results from arrays w/o saving to the files.

[Dustin Lang]

If you want to call the code as a library rather than running the solve-field program, then have a look at the solver/control-program.c program.  That should have most of what you want.

To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/cb61f4b5-c4ed-4838-978d-4c831885c208n%40googlegroups.com.

[Danko Dnevic]

Hi Dastin,

I'm amazed with effort you invested in AN. One can realize that with just trying to comprehend the amount of code i.e. kdtree_internal.c

With that and with magic you used to handle/mangle all possible data types, comes my insufficient skills to see through it.

My attempt to build AN, particularly libkd, while staying within Android Studio and CMakeLists.txt, failed. Mostly, because of poor preprocessor handling of your versatile macros.

Last option, that I have, is to reduce data types used in libkd. As I mentioned earlier, I have liberty to present source data (x,y,flux) in any data type (float/double), that guarantees minimum recasting need for AN.

My question is: what would be best default kdint_xxx? Assuming that downloadable very wide field index files (4117, 4118, 4119) are Ra/Dec/Mag floats.

Thanks, DD

[Dustin Lang]

Hi,

It looks like the index-411*.fits files use both "dss" and "duu" kd-trees.

Feel free to send the error messages that you're getting when you try to compile, someone might be able to comment.  Is it maybe failing to glue together the symbols?  That part is fancy footwork.

Sorry for the trouble!

cheers,

dustin

To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/a5d719a8-a93e-4933-8453-0e6e9d161b91n%40googlegroups.com.

[Danko Dnevic]

Yes, it's all about gluing symbols together.

I restored all sources, in CMakeLists.txt declared all libkd sources as in Makefile.min

set(KD_SOURCES
       #Internals
       ${ASTROMETRY_DIR}/libkd/kdint_ddd.c
       ${ASTROMETRY_DIR}/libkd/kdint_fff.c
       ${ASTROMETRY_DIR}/libkd/kdint_ddu.c
       ${ASTROMETRY_DIR}/libkd/kdint_duu.c
       ${ASTROMETRY_DIR}/libkd/kdint_dds.c
       ${ASTROMETRY_DIR}/libkd/kdint_dss.c
       #kd
       ${ASTROMETRY_DIR}/libkd/kdtree.c
       ${ASTROMETRY_DIR}/libkd/kdtree_dim.c
       ${ASTROMETRY_DIR}/libkd/kdtree_mem.c
       #kd fits
       ${ASTROMETRY_DIR}/libkd/kdtree_fits_io.c
       #dt
       ${ASTROMETRY_DIR}/libkd/dualtree.c
       ${ASTROMETRY_DIR}/libkd/dualtree_rangesearch.c
       ${ASTROMETRY_DIR}/libkd/dualtree_nearestneighbour.c
)

ld: error: undefined symbol: kdtree_build_2_lll
>>> referenced by kdtree_dim.c:25 (C:/astrometry/libkd/kdtree_dim.c:25)

ld: error: undefined symbol: kdtree_update_funcs_lll
>>> referenced by kdtree.c:78 (C:/astrometry/libkd/kdtree.c:78)

ld: error: undefined symbol: kdtree_node_node_mindist2_lll
>>> referenced by kdtree.c:595 (C:/astrometry/libkd/kdtree.c:595)

...

It's clear that AS fails to preprocess macros mangling all those variations. Question is, if solution is to choose "one for all", which one fits naturally?

[Dustin Lang]

Oh, it looks like kdint_lll.c is not included in your list (or in Makefile.min -- sorry!)

--dustin

To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/4b5d2f1f-fb93-4eb5-b8cb-8d1f9975db79n%40googlegroups.com.

[Danko Dnevic]

LOL, it's my blunder also. Now, libkd errors disappeared, with just some implicit conversions warnings, like:

In file included from C:/astrometry/libkd/kdint_ddu.c:41:

C:/astrometry/libkd/kdtree_internal.c:604:27: warning: implicit conversion from 'unsigned long' to 'double' changes value from 18446744073709551615 to 18446744073709551616 [-Wimplicit-const-int-float-conversion]

        } else if (dtl2 < BIGTTYPE_MAX) {

                        ~ ^~~~~~~~~~~~

C:/astrometry/libkd/kdint_ttype_u.h:9:22: note: expanded from macro 'BIGTTYPE_MAX'

#define BIGTTYPE_MAX UINT64_MAX

                     ^~~~~~~~~~ 

Thanks, without your insight, I would wander across the libkd source.

BR, DD

[Danko Dnevic]

I hope, that you have some insight magic for gsl-an?

This time I selected all files:

file(GLOB GSL_BASE_SOURCES "${ASTROMETRY_DIR}/gsl-an/*.c") file(GLOB GSL_BLAS_SOURCES "${ASTROMETRY_DIR}/gsl-an/blas/*.c") file(GLOB GSL_BLOCK_SOURCES "${ASTROMETRY_DIR}/gsl-an/block/*.c") file(GLOB GSL_CBLAS_SOURCES "${ASTROMETRY_DIR}/gsl-an/cblas/*.c") file(GLOB GSL_ERR_SOURCES "${ASTROMETRY_DIR}/gsl-an/err/*.c") file(GLOB GSL_LINALG_SOURCES "${ASTROMETRY_DIR}/gsl-an/linalg/*.c") file(GLOB GSL_MATRIX_SOURCES "${ASTROMETRY_DIR}/gsl-an/matrix/*.c") file(GLOB GSL_MULTIROOTS_SOURCES "${ASTROMETRY_DIR}/gsl-an/multiroots/*.c") file(GLOB GSL_PERMUTATION_SOURCES "${ASTROMETRY_DIR}/gsl-an/permutation/*.c") file(GLOB GSL_SYS_SOURCES "${ASTROMETRY_DIR}/gsl-an/sys/*.c") file(GLOB GSL_VECTOR_SOURCES "${ASTROMETRY_DIR}/gsl-an/vector/*.c") # Add sources from the Makefile set(GSLAN_SOURCES        ${GSL_BASE_SOURCES}        ${GSL_BLAS_SOURCES}        ${GSL_BLOCK_SOURCES}        ${GSL_CBLAS_SOURCES}        ${GSL_ERR_SOURCES}        ${GSL_LINALG_SOURCES}        ${GSL_MATRIX_SOURCES}        ${GSL_MULTIROOTS_SOURCES}        ${GSL_PERMUTATION_SOURCES}        ${GSL_SYS_SOURCES}        ${GSL_VECTOR_SOURCES} )

And included:

#target_include_directories(astrolib include_directories(        PRIVATE ${ASTROMETRY_DIR}        PRIVATE ${ASTROMETRY_DIR}/include        PRIVATE ${ASTROMETRY_DIR}/include/astrometry        PRIVATE ${ASTROMETRY_DIR}/util        PRIVATE ${ASTROMETRY_DIR}/solver        PRIVATE ${ASTROMETRY_DIR}/libkd        PRIVATE ${ASTROMETRY_DIR}/qfits-an        PRIVATE ${ASTROMETRY_DIR}/gsl-an        PRIVATE ${ASTROMETRY_DIR}/gsl-an/gsl

PRIVATE ${ASTROMETRY_DIR}/gsl-an/cblas PRIVATE ${ASTROMETRY_DIR}/gsl-an/multiroots PRIVATE ${ASTROMETRY_DIR}/gsl-an/vector PRIVATE ${ASTROMETRY_DIR}/gsl-an/matrix )

It seems that preprocessor is unable to resolve some macros:

C:/astrometry/gsl-an/block/init_source.c:20:18: error: expected function body after function declarator

TYPE (gsl_block) *

C:/astrometry/gsl-an/matrix/submatrix_source.c:21:1: error: expected function body after function declarator

FUNCTION (gsl_matrix, submatrix) (QUALIFIED_TYPE(gsl_matrix) * m,

^

C:/astrometry/gsl-an/vector/subvector_source.c:21:1: error: expected function body after function declarator

FUNCTION(gsl_vector, subvector) (QUALIFIED_TYPE(gsl_vector) * v, size_t offset, size_t n)

^

BTW, I transfered gsl-an config.h from my WSL machine, but to no avail.

[Dustin Lang]

Hmm, I'm afraid I can't help much with that!  The "gsl-an" directory contains a subset of the GNU Scientific Library (https://www.gnu.org/software/gsl/).  If you can install GSL normally, you can use it that way (should get auto-detected by pkg-config).

It may be helpful to know that the gsl-an directory is self-contained -- it does not require any include files or anything outside the gsl-an directory.

For what it's worth, I think it just compiles each of the .c files and then links them into a single library -- here is the list of .o files shown in the library linking step.

ar rc libgsl-an.a blas/blas.o block/block.o block/init.o cblas/caxpy.o cblas/ccopy.o cblas/cdotc_sub.o cblas/cdotu_sub.o cblas/cgemm.o cblas/cgemv.o cblas/cgerc.o cblas/cgeru.o cblas/chemm.o cblas/chemv.o cblas/cher.o cblas/cher2.o cblas/cher2k.o cblas/cherk.o cblas/cscal.o cblas/csscal.o cblas/cswap.o cblas/csymm.o cblas/csyr2k.o cblas/csyrk.o cblas/ctrmm.o cblas/ctrmv.o cblas/ctrsm.o cblas/ctrsv.o cblas/dasum.o cblas/daxpy.o cblas/dcopy.o cblas/ddot.o cblas/dgemm.o cblas/dgemv.o cblas/dger.o cblas/dnrm2.o cblas/drot.o cblas/drotg.o cblas/drotm.o cblas/drotmg.o cblas/dscal.o cblas/dsdot.o cblas/dswap.o cblas/dsymm.o cblas/dsymv.o cblas/dsyr.o cblas/dsyr2.o cblas/dsyr2k.o cblas/dsyrk.o cblas/dtrmm.o cblas/dtrmv.o cblas/dtrsm.o cblas/dtrsv.o cblas/dzasum.o cblas/dznrm2.o cblas/icamax.o cblas/idamax.o cblas/isamax.o cblas/izamax.o cblas/sasum.o cblas/saxpy.o cblas/scasum.o cblas/scnrm2.o cblas/scopy.o cblas/sdot.o cblas/sdsdot.o cblas/sgemm.o cblas/sgemv.o cblas/sger.o cblas/snrm2.o cblas/srot.o cblas/srotg.o cblas/srotm.o cblas/srotmg.o cblas/sscal.o cblas/sswap.o cblas/ssymm.o cblas/ssymv.o cblas/ssyr.o cblas/ssyr2.o cblas/ssyr2k.o cblas/ssyrk.o cblas/strmm.o cblas/strmv.o cblas/strsm.o cblas/strsv.o cblas/xerbla.o cblas/zaxpy.o cblas/zcopy.o cblas/zdotc_sub.o cblas/zdotu_sub.o cblas/zdscal.o cblas/zgemm.o cblas/zgemv.o cblas/zgerc.o cblas/zgeru.o cblas/zhemm.o cblas/zhemv.o cblas/zher.o cblas/zher2.o cblas/zher2k.o cblas/zherk.o cblas/zscal.o cblas/zswap.o cblas/zsymm.o cblas/zsyr2k.o cblas/zsyrk.o cblas/ztrmm.o cblas/ztrmv.o cblas/ztrsm.o cblas/ztrsv.o err/error.o err/stream.o err/strerror.o linalg/bidiag.o linalg/cholesky.o linalg/householder.o linalg/lu.o linalg/qr.o linalg/svd.o matrix/copy.o matrix/init.o matrix/matrix.o matrix/rowcol.o matrix/submatrix.o matrix/swap.o matrix/view.o multiroots/broyden.o multiroots/convergence.o multiroots/dnewton.o multiroots/fdfsolver.o multiroots/fdjac.o multiroots/fsolver.o multiroots/gnewton.o multiroots/hybrid.o multiroots/hybridj.o multiroots/newton.o permutation/init.o permutation/permutation.o permutation/permute.o sys/coerce.o sys/fdiv.o sys/infnan.o sys/ldfrexp.o vector/copy.o vector/init.o vector/oper.o vector/prop.o vector/subvector.o vector/swap.o vector/vector.o

cheers,

dustin

To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/7d9cfbaa-b99b-455f-931f-c337df91d314n%40googlegroups.com.

[Danko Dnevic]

It's fine, I know that GSL is out of your responsibility. 

I ported GSL 2.7, using AMPL modification, that use CMake (w/o AMPL). If anyone in the future reads this, here is the command line:

cmake .. -G "Ninja" -DCMAKE_TOOLCHAIN_FILE=<path_to_toolchain>/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-21 -DANDROID_NDK=<path_to_NDK> -DCMAKE_BUILD_TYPE=Release -DGSL_DISABLE_TESTS=1 -DNO_AMPL_BINDINGS=1 -S .. -B .

It builds without errors, but app now crashes. I'm going to investigate causes. I hope, that I can take rain check, if some AN issue appears.

Thanks, DD 

[Danko Dnevic]

Hi Dustin,

I'm back in your domain again. By using my adaptation of control-program.c, I got SIGSEGV in solver_run() (particularly during 3rd iteration of for(newpoint=...) loop). Because I'm not sure, did I prepare 'solver' structure properly, I had to give my best shoot, imitating control-program.c. This is 'solver' structure before solver_run(solver) call:

indexes: 0xb400007a09615930

fieldxy: 0x0

pixel_xscale: 0.000000

predistort: 0x0

fieldxy_orig: 0xb400007a0976eae0

funits_lower: 23.437500

funits_upper: 70.312500

logratio_toprint: 0.000000

logratio_tokeep: 27.631021

logratio_totune: 1000000000000000019884624838656.000000

record_match_callback: 0x799e13c24c

userdata: 0x7a08ffb2b0

distance_from_quad_bonus: 1

verify_uniformize: 1

verify_dedup: 1

do_tweak: 0

tweak_aborder: 3

tweak_abporder: 3

verify_pix: 1.000000

distractor_ratio: 0.250000

codetol: 0.010000

quadsize_min: 307.200000

quadsize_max: 0.000000

startobj: 0

endobj: 0

parity: 1

use_radec: 0

centerxyz: [0.000000, 0.000000, 0.000000]

r2: 0.000000

logratio_bail_threshold: -230.258509

logratio_stoplooking: 1000000000000000019884624838656.000000

maxquads: 0

maxmatches: 0

set_crpix: 0

set_crpix_center: 0

crpix: [0.000000, 0.000000]

mo_template: 0x0

timer_callback: 0x799e13c3f4

quit_now: 0

numtries: 0

nummatches: 0

numscaleok: 0

last_examined_object: 0

num_cxdx_skipped: 0

num_meanx_skipped: 0

num_radec_skipped: 0

num_abscale_skipped: 0

num_verified: 0

index: 0x0

minminAB2: 0.000000

maxmaxAB2: 0.000000

rel_index_noise2: 0.000000

rel_field_noise2: 0.000000

abscale_low: 0.000000

abscale_high: 0.000000

field_minx: 0.000000

field_maxx: 0.000000

field_miny: 0.000000

field_maxy: 0.000000

field_diag: 0.000000

cxdx_margin: 0.000000

starttime: 0.000000

timeused: 0.000000

best_logodds: 0.000000

best_match: 0xb400007a097278b0

best_index: 0x0

best_match_solves: 0

have_best_match: 0

vf: 0x0 

Please, on the first sight, can you find, what I made wrong?

Thanks, 

DD

[Dustin Lang]

Hahaha, it has been a very long time since I wrote that code!  I have no idea!  Can you find out what line your segfault is coming from?

thanks,

dustin

To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/6de5523a-4cce-4352-a7d7-5bc1ad579c44n%40googlegroups.com.

[Danko Dnevic]

On 3rd entrance from outside for loop)

for (newpoint = solver->startobj; newpoint < numxy; newpoint++) {

...

...

Fault happens in this loop:

// Now iterate through the different indices

            for (i = 0; i < num_indexes; i++) {

                index_t* index = pl_get(solver->indexes, i);

                int dimquads;

                set_index(solver, index);

                dimquads = index_dimquads(index);

                for (field[A] = 0; field[A] < newpoint; field[A]++) {

                    // initialize the "pquad" struct for this AB combo.

                    pquad* pq = pquads + field[B] * numxy + field[A];

                    if (!pq->scale_ok)

                        continue;

                    if ((pq->scale < minAB2s[i]) ||

                        (pq->scale > maxAB2s[i]))

                        continue;

                    // set code tolerance for this index and AB pair...

                    solver->rel_field_noise2 = pq->rel_field_noise2;

                    tol2 = get_tolerance(solver);

                    // Now look at all sets of (C, D, ...) stars (subject to field[C] < field[D] < ...)

                    // ("dimquads - 2" because we've set stars A and B at this point)

                    add_stars(pq, field, C, dimquads-2, 0, newpoint, dimquads, solver, tol2);

                    if (solver->quit_now)

                        goto quitnow;

                }

            }

...

[Dustin Lang]

:)  Sorry, I still don't know, keep adding printf statements :)

[Danko Dnevic]

Update: I narrow down fault, to the add_stars function invoked from first iteration of for (field[A] = 0;...) loop in solver_run.

[Dustin Lang]

Are you calling solver_set_field() before this?  Looks like "fieldxy" is null in your struct

To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/21065ed9-f740-4a01-9adf-3d960d20c1c8n%40googlegroups.com.

[Danko Dnevic]

I did it imitating control-program.c, as much as I could:

// Get the next image... racenter = 1.0/0.0; deccenter = 1.0/0.0; radec_radius = 0.0; t0 = timenow(); // Feed the image source coordinates to the solver... field = starxy_new(nstars, TRUE, FALSE); starxy_set_x_array(field, starx); starxy_set_y_array(field, stary); starxy_set_flux_array(field, starflux); starxy_sort_by_flux(field); solver_set_field(solver, field); free(starx); free(stary); free(starflux);

// center of the image in pixels, according to FITS indexing.

imagecx = (imagew - 1.0)/2.0; imagecy = (imageh - 1.0)/2.0; // How big of a search radius should we use?  radec_radius // plus the maximum image size. radec_radius += arcsec2deg(app_max * hypot(imagew, imageh) / 2.0);

What I missed?

[Danko Dnevic]

I saw that before, but taught that fieldxy_orig is the one set by solver_set_field()?

[Dustin Lang]

But shouldn't this line run and call solver_preprocess_field(), setting fieldxy?

https://github.com/dstndstn/astrometry.net/blob/main/solver/solver.c#L458

To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/ccd8ed10-64e3-4bb8-b074-9435455cc409n%40googlegroups.com.

[Danko Dnevic]

I listed 'solver' structure before solver_run(solver) call.

[Danko Dnevic]

void solver_set_field(solver_t* s, starxy_t* field) {

    solver_free_field(s);

    s->fieldxy_orig = field;

    // Preprocessing happens in "solver_preprocess_field()".

}

solver_run() at the start has preprocessing.

[Danko Dnevic]

If you ment on this part of control-program.c

if (sip && !ignore) {

            logmsg("Trying to verify existing WCS...\n");

            solver_verify_sip_wcs(solver, sip);

            if (solver->best_match_solves) {

                // Existing WCS passed the test.

                logmsg("Existing WCS pass the verification test with odds ratio %g\n",

                       exp(solver->best_match.logodds));

                // the WCS is solver->best_match.wcstan

                solved = TRUE;

            } else {

                logmsg("Existing WCS failed the verification test...\n");

            }

 }

I skipped that part, because 'sip' is not initialized at this point or NULL. Should I initialize sip?

[Danko Dnevic]

UPDATE:

Skip my previous posts.

I traced further through solver.c (and digged past the Balrog :-) to the try_permutations(), where this call:

          // Search with the code we've built.

            *presult = kdtree_rangesearch_options_reuse(solver->index->codekd->tree, *presult, code, tol2, options);

Leads to the segfault. In kdtree_dim.c function kdtree_rangesearch_options_reuse() is defined as:

kdtree_qres_t* KDFUNC(kdtree_rangesearch_options_reuse)

     (const kdtree_t *kd, kdtree_qres_t* res, const void *pt, double maxd2, int options) {

    assert(kd->fun.rangesearch);

    return kd->fun.rangesearch(kd, res, pt, maxd2, options);

}

[Danko Dnevic]

Here is reason for the crash:

solver->index pointer: 0xb400007a04ea38c0, solver->index->codekd pointer: 0x0, *presult pointer: 0x0, tol2: 0.000100, options: 281

Unfortunately, I don't know how to fix it. 'codekd' is not mentioned in control-program.c, nor any function which sets it (using that name).

[Dustin Lang]

Hi,

The codekd is part of the index file.  In the index-4111.fits file I'm looking at, it is a "duu" kd-tree.  It should get loaded when you load the index file(s) -- in control-program.c it looks like that happens in engine_parse_config_file().

cheers,

dustin

To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/3767e042-002c-4673-a14d-425ae0aaa476n%40googlegroups.com.

[Danko Dnevic]

I'm loading 4118 and 4119 (it's very vide field), by autoindex in astrometry.cfg and my starx,y and flux are all double arrays (as in control-program). Is there any type mismatch?

Thanks,

DD

[Dustin Lang]

The star x,y, and flux types are not relevant.  You need to make sure that the libkd tree type used in the index files are supported by the libkd code that you compiled.

Are you getting logging information out?  You could try turning the logging up to verbose -- log_init(LOG_VERB).

cheers,

dustin

To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/efe81645-298a-423d-9821-89b09030231dn%40googlegroups.com.

[Danko Dnevic]

I have to use Android logging, with format  __android_log_print(ANDROID_LOG_INFO,...
Just tell me what and where to check?

[Danko Dnevic]

BTW, wasn't all that code mangling, just to have all possible codes available? ;-)

[Dustin Lang]

I forgot, I thought you only did the minimal ones.  But if you did all, then no problem.

To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/e65cd357-6dd4-409e-838c-84756626c0edn%40googlegroups.com.

[Danko Dnevic]

This is what went to the build:

set(KD_SOURCES        #Internals        ${ASTROMETRY_DIR}/libkd/kdint_ddd.c        ${ASTROMETRY_DIR}/libkd/kdint_fff.c        ${ASTROMETRY_DIR}/libkd/kdint_ddu.c        ${ASTROMETRY_DIR}/libkd/kdint_duu.c        ${ASTROMETRY_DIR}/libkd/kdint_dds.c        ${ASTROMETRY_DIR}/libkd/kdint_dss.c        ${ASTROMETRY_DIR}/libkd/kdint_lll.c        #kd        ${ASTROMETRY_DIR}/libkd/kdtree.c        ${ASTROMETRY_DIR}/libkd/kdtree_dim.c        ${ASTROMETRY_DIR}/libkd/kdtree_mem.c        #kd fits        ${ASTROMETRY_DIR}/libkd/kdtree_fits_io.c        #dt        ${ASTROMETRY_DIR}/libkd/dualtree.c        ${ASTROMETRY_DIR}/libkd/dualtree_rangesearch.c        ${ASTROMETRY_DIR}/libkd/dualtree_nearestneighbour.c )

[Dustin Lang]

Yeah, so that looks like everything.

[Danko Dnevic]

What should I do about missing pointer? Segfault is triggered, by attempt to read kd->fun.rangesearch in  kdtree_rangesearch_options_reuse()   

I have the impression that the solution is within reach, I just don't know where to look?

[Dustin Lang]

Well, that pointer should get set when you read the index file.  So if you can figure out why that  isn't happening...   I would have expected the code to print an error message if there was a problem!

[Danko Dnevic]

Index is loaded at solver_run in loop "Now loop through different indices" and pointer index is identical to the slover->index I print before crash. That is all what is loaded in function set_index() called from solver_run()

static void set_index(solver_t* s, index_t* index) {

    s->index = index;

    s->rel_index_noise2 = square(index->index_jitter / index->index_scale_lower);

}

[Dustin Lang]

Yes, you should check the index struct right after it is read/loaded -- after engine_parse_config_file(), or look at right where it is loaded at https://github.com/dstndstn/astrometry.net/blob/main/solver/engine.c#L196

And again, if you turn on the logging, it will probably tell you what is happening.

Or, can you run your code on a normal linux platform where you have a debugger?

[Danko Dnevic]

I'll try that! 

[Danko Dnevic]

I turn the logging to the file, with LOG_VERB level:

Auto-indexing directory "/data/data/com.example.natapp/files/indices" ...

Skipping directory /data/data/com.example.natapp/files/indices/.

Skipping directory /data/data/com.example.natapp/files/indices/..

Checking file "/data/data/com.example.natapp/files/indices/index-4118.fits"

Index name "/data/data/com.example.natapp/files/indices/index-4118.fits" is readable, using as index filename

Checking file "/data/data/com.example.natapp/files/indices/index-4119.fits"

Index name "/data/data/com.example.natapp/files/indices/index-4119.fits" is readable, using as index filename

Checking file "/data/data/com.example.natapp/files/indices/index-4117.fits"

Index name "/data/data/com.example.natapp/files/indices/index-4117.fits" is readable, using as index filename

Trying to add index "/data/data/com.example.natapp/files/indices/index-4119.fits".

Index name "/data/data/com.example.natapp/files/indices/index-4119.fits" is readable, using as index filename

Index name "/data/data/com.example.natapp/files/indices/index-4119.fits" is readable, using as index filename

Loading metadata for /data/data/com.example.natapp/files/indices/index-4119.fits...

Index name "/data/data/com.example.natapp/files/indices/index-4119.fits" is readable, using as index filename

Index scale: [1400, 2000] arcmin, [84000, 120000] arcsec

Index has 1728 quads and 1080 stars

Trying to add index "/data/data/com.example.natapp/files/indices/index-4118.fits".

Index name "/data/data/com.example.natapp/files/indices/index-4118.fits" is readable, using as index filename

Index name "/data/data/com.example.natapp/files/indices/index-4118.fits" is readable, using as index filename

Loading metadata for /data/data/com.example.natapp/files/indices/index-4118.fits...

Index name "/data/data/com.example.natapp/files/indices/index-4118.fits" is readable, using as index filename

Index scale: [1000, 1400] arcmin, [60000, 84000] arcsec

Index has 3072 quads and 1920 stars

Trying to add index "/data/data/com.example.natapp/files/indices/index-4117.fits".

Index name "/data/data/com.example.natapp/files/indices/index-4117.fits" is readable, using as index filename

Index name "/data/data/com.example.natapp/files/indices/index-4117.fits" is readable, using as index filename

Loading metadata for /data/data/com.example.natapp/files/indices/index-4117.fits...

Index name "/data/data/com.example.natapp/files/indices/index-4117.fits" is readable, using as index filename

Index scale: [680, 1000] arcmin, [40800, 60000] arcsec

Index has 4800 quads and 3000 stars

Quad scale range: [574.233, 5132.07] pixels

object 1 of 20: 0 quads tried, 0 matched.

object 2 of 20: 0 quads tried, 0 matched.

object 3 of 20: 0 quads tried, 0 matched.

[Dustin Lang]

That all looks fine!

I'm puzzled about what is going wrong!

To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/7a800512-92e0-4b8a-a3fb-5d3b42fce84cn%40googlegroups.com.

[Danko Dnevic]

I'm puzzled also. I expected that cause will be behind some wild macro, unsupported on Android clang target, incompatible malloc, switched sortq(), etc., not basic nullptr reference... :-)

I tried loading 4117-4119 and 4215-4219, in multiple combinations (one index, two, three...), but to no avail.

Endianness shouldn't be the issue (both aarch64 and armv7 are little endian).

What can I do to verify the loaded index(es)? I can hexdump some range of addresses, for indexes you are familiar with?

I can't find pl_get() (probably augmented by some macros), to see layout and what goes where?

BR,

DD

[Dustin Lang]

Right!!

In your log I think it's printing this,

"Index has 1728 quads and 1080 stars"

so it's getting here,

https://github.com/dstndstn/astrometry.net/blob/main/util/index.c#L300

so index_reload() finished successfully,

so all this should have run,

https://github.com/dstndstn/astrometry.net/blob/main/util/index.c#L345

and index->{starkd, quads, codekd}

should all be set to non-null.

and I don't think anyone is calling index_close() on index_unload(), so all your index->starkd, index->quads, and index->codekd pointers should always be non-null... right?

And yes, pl_get is some silly macro thing, but it just gets you one element out of a list ("bl" stands for "block list", "pl" is "pointer list").

So if you can't step through in a debugger, add a bunch of prints and make sure that the index->starkd,quads,codekd pointers are getting set?

cheers,

dustin

[Danko Dnevic]

What is the template for pl_get? I'm missing that part to complete the chain of events...

[Dustin Lang]

sure, all the bl_ things are sort of like the C++ vector class -

void* pl_get(pl_t* list, int i);

returns element "i" from pointer-list "list".

So, eg,

            index_t* index = pl_get(engine->indexes, i);

grabs element "i" out of the engine->indexes list.

cheers,

dustin

[Danko Dnevic]

Dastin, that initialization, you saw in the log, of index->codekd, is invoked from start of control-program.c after engine initialization:

engine = engine_new();

    if (engine_parse_config_file(engine, configfn)) {

        logerr("Failed to parse (or encountered an error while interpreting)"

               "config file \"%s\"\n", configfn);

        exit( -1);

    }

That is when index.c code is executed.

Unfortunately after that, in control-program.c

solver = solver_new();

Clean any structure within solver, including solver->index->codekd

[Danko Dnevic]

I see now, that indices are part of the engine, but still can't find how they get associated into the solver?

[Danko Dnevic]

Dustin, 

control-program.c is, on many ways, tied to WCS in FITS files, to the level that I don't understand how to present relatively simple setup:

- nstars number of elements, 241 source extracted from the 4096x3072 (68x56 HxV degrees) negative parity (FLIPPED) image in double *starx, *stary and *starflux arrays, sorted by descending starflux. 

nova.astrometry.net already has that setup, for uploading .txt xy lists (without fluxes). How is it done there?

[Dustin Lang]

That happens here,

https://github.com/dstndstn/astrometry.net/blob/main/solver/control-program.c#L386

[Dustin Lang]

Here, starx, stary, and flux are just double* arrays

https://github.com/dstndstn/astrometry.net/blob/main/solver/control-program.c#L353

cheers,

dustin

[Danko Dnevic]

But this part of control-program.c code, related to FITS image, make me confused:

// Where is the center of the image according to the existing WCS?

        if (sip)

            sip_pixelxy2radec(sip, imagecx, imagecy, &racenter, &deccenter);

        // How big of a search radius should we use?  radec_radius

        // plus the maximum image size.

        radec_radius += arcsec2deg(app_max * hypot(imagew, imageh) / 2.0);

        // Which indexes should we use?  Use the WCS or RA,Dec

        // estimate to decide.

        N = pl_size(engine->indexes);

        for (i=0; i<N; i++) {

            index_t* index = pl_get(engine->indexes, i);

            if (!isinf(racenter) &&

                !index_is_within_range(index, racenter, deccenter,

                                       radec_radius))

                continue;

            logmsg("Adding index %s\n", index->indexname);

            solver_add_index(solver, index);

        }

        if (solver_n_indices(solver) == 0) {

            logmsg("No index files are within range of given RA,Dec center "

                   "and radius: (%g,%g), %g\n",

                   racenter, deccenter, radec_radius);

            goto skip;

        }

        // If you want the list of matched stars, you have to do this ugliness:

        solver->record_match_callback = record_match_callback;

        solver->userdata = &cb;

        if (sip && !ignore) {

            logmsg("Trying to verify existing WCS...\n");

            solver_verify_sip_wcs(solver, sip);

            if (solver->best_match_solves) {

                // Existing WCS passed the test.

                logmsg("Existing WCS pass the verification test with odds ratio %g\n",

                       exp(solver->best_match.logodds));

                // the WCS is solver->best_match.wcstan

                solved = TRUE;

            } else {

                logmsg("Existing WCS failed the verification test...\n");

            }

        }

        solver->timer_callback = timer_callback;

        cb.max_cpu_time = 10;

        cb.max_wall_time = 20;

        cb.wall_start = timenow();

        if (!solved) {

            /*

             // Now, if you wanted to ignore the WCS and check all indexes, you

             // could do this:

             solver_clear_indexes(solver);

             for (i=0; i<N; i++) {

             index_t* index = pl_get(engine->indexes);

             solver_add_index(solver, index);

             }

             */

            //

[Dustin Lang]

If you don't have a guess of the WCS, then sip = null.

If you don't have a RA,Dec and radius, then ignore the "index_is_within_range()" call and just add all indexes into the solver.

[Danko Dnevic]

Hi Dustin,

index_reload(index); 

//before line 386 in control_program.c 

solver_add_index(solver, index);

Solved segfault :-)

Tomorrow, I will test. Thanks a LOT!!!

BR,

DD

[Danko Dnevic]

Hi Dastin,

I think that I found the root cause of the segfault. It could be the bug, or feature:

As we can see, from the log above

Trying to add index "/data/data/com.example.natapp/files/indices/index-4119.fits".

Index name "/data/data/com.example.natapp/files/indices/index-4119.fits" is readable, using as index filename

Index name "/data/data/com.example.natapp/files/indices/index-4119.fits" is readable, using as index filename

Loading metadata for /data/data/com.example.natapp/files/indices/index-4119.fits...

Index name "/data/data/com.example.natapp/files/indices/index-4119.fits" is readable, using as index filename

Index scale: [1400, 2000] arcmin, [84000, 120000] arcsec

Index has 1728 quads and 1080 stars

index_load() is called with 'flags' INDEX_ONLY_LOAD_METADATA and is loading ONLY index metadata

if (flags & INDEX_ONLY_LOAD_METADATA)
        logverb("Loading metadata for %s...\n", indexname);

https://github.com/dstndstn/astrometry.net/blob/fbca48ebec403d4f954c97bc83d260ea40643577/util/index.c#L273

Same flag, few lines below, causes unloading index

https://github.com/dstndstn/astrometry.net/blob/fbca48ebec403d4f954c97bc83d260ea40643577/util/index.c#L310

which destroy all *codekd etc. stuff

https://github.com/dstndstn/astrometry.net/blob/fbca48ebec403d4f954c97bc83d260ea40643577/util/index.c#L357

    if (index->starkd) {
        startree_close(index->starkd);
        index->starkd = NULL;
    }
    if (index->codekd) {
        codetree_close(index->codekd);
        index->codekd = NULL;
    }
    if (index->quads) {
        quadfile_close(index->quads);
        index->quads = NULL;
    }

Now, when I investigated, why is the INDEX_ONLY_LOAD_METADATA flag set, anyway?

I came to

if (engine_parse_config_file(engine, configfn)) {

in control-program.c, which parses astrometry.cfg and in engine_parse_config_file_stream()

https://github.com/dstndstn/astrometry.net/blob/fbca48ebec403d4f954c97bc83d260ea40643577/solver/engine.c#L328

invokes engine_add_index(), which in line:

https://github.com/dstndstn/astrometry.net/blob/fbca48ebec403d4f954c97bc83d260ea40643577/solver/engine.c#L196

has this

ind = index_load(path, engine->inparallel ? 0 : INDEX_ONLY_LOAD_METADATA, NULL);

I wonder, why is it set, that if NOT inparallel (what is probably default on small systems), to load only metadata (what later crashes app)?

Please, forgive me for taking so much of your time.

With Best Regards,

DD

[Dustin Lang]

Ohhhhh, I'm sorry for missing the "Loading metadata" log message!

There is some text about 'inparallel' here,

https://github.com/dstndstn/astrometry.net/blob/main/solver/control-program.c#L263

but it sounds like it should either force "engine->inparallel = TRUE", or print an error message and exit.  And the text should say you MUST set inparallel!

I'm sorry for the trouble!

--dustin

[Danko Dnevic]

LOL, I assumed that "inparallel" is flag/switch for some internal parallel processing of indexes, because it crashed even with just one index loaded, what further reinforced my misbelief. :-)

In the original template astrometry.cfg-dist it was stated:

# -if you are using a 64-bit machine and you have enough physical memory to contain
#  the indices you are using, then you want this enabled.
#
# -otherwise, leave it commented-out.
#inparallel

https://github.com/dstndstn/astrometry.net/blob/fbca48ebec403d4f954c97bc83d260ea40643577/etc/astrometry.cfg-dist#L15

And this quote from control-program.c is essentially correct:

    // ... Either way, for
    // performance it's very helpful to have enough memory to hold all
    // the relevant index files, and you should set the
    //     inparallel
    // option in the config file.

When something works it has performance... LOL

I hope, that it is safe now, to remove that index_reload() from control-program.c adaptation and rely on inparallel config switch.

Thank you again for your time.

With Best Regards,

DD