Windows port of astrometry.net

[Sean Prange]

I'm in the process of porting the project to windows using GCC as the compiler.  I decided not to use Microsoft's or Intel's compilers as they will take a lot more effort.  I've had to use a few external code projects:

mman-win32 which provides the necessary mmap() function, slightly modified

regex for windows 2.7 compiled as is

mkstemp and mkdtemp() from git--can't determine who the author is

getpagesize() from GCC returns 4096 but for memory mapping on my system it needs to be 65536.  Windows provides a way to determine this value so nothing will have to be hardcoded but getpagesize() will have to be avoided

I've had to remove sigbus, and for now svn functions (the svn ones simply gave a link to astrometry.net's project).  Rlimits and any function related to them removed, and cpu/resource calculations.  All the changes are within blocks of code marked by #ifdef _WIN32.  I've compiled everything necessary to get control-program running.  On my end, even with a fresh install of Ubuntu 14.04 and a clean build of astrometry.net, control-program seg faults.  My cygwin version does too.  My windows native port dies at the same point too so at least it's consistent!  Fortunately I have the windows version compiled using Microsoft Visual Studio 2013 and VisualGDB--a very nice combination that allows all the features of the IDE and terrific 'visual' debugging.   Soon I'll upload the changes I've made and provide the project files for easier building.  Technically the default Makefiles will work BUT have fun setting up a proper build environment.  Cross compiling from cygwin or linux could work though.  Maybe we could eventually port enough of the code to build solve-field.  

Sean

[Sean Prange]

Regarding the segmentation fault problem, both seem to be caused by the solver index.  In one case:

*presult = kdtree_rangesearch_options_reuse

                (solver->index->codekd->tree, *presult, code, tol2, options);

solver->index->codekb == NULL so therefore solver->index->codekb->tree causes the problem

in another

mo->dimquads = quadfile_dimquads(sp->index->quads);

Again sp->index->quads == NULL so the same thing occurs

The error I receive depends on whether the input files has WCS information.  This occurs (for me) in control-program built in linux, cygwin, or windows.

Last night I also found that some index files (all recent downloads) cause an immediate error (tracking that one down now).  Haven't seen if this happens in linux.

[Dustin Lang]

Weird.  Could you please send a full log (including -v in the solve-field command)?  Could you also please narrow down to one index file, and tell me which file it is?  I wonder if your index files got corrupted during download.  But it looks like some fields are NULL so I would have expected the index-loading code to complain.

thanks,
--dustin

[Sean Prange]

With index file 4207-08 I encounter:

ERROR("Expected table size (%Iu => %i FITS blocks) is not equal to "

 "size of table \"%s\" (%Iu => %i FITS blocks).",

This is a repeatable error--I can erase the file, download and same thing happens.

Attached is the output if I attempt to solve a file. It segfaults at one of the two locations I mentioned.  If I do a test for a NULL pointer and return from the function the program will run, search for matches, but never seem to find a match.

[Dustin Lang]

Ok, could you please check the md5sum:

341c662971c9f1368619baf491c5d59c *index-4207-08.fits

The file size should be 13786560 bytes.

if the file is the wrong size then something is clearly wrong.  Perhaps whatever program you are using to download it is making some wrong decision about its file type.  I believe the web server reports it as octet-stream, so any well-behaved program should just save it as binary.

cheers,
--dustin

[Sean Prange]

The file size and checksum checks out.

With 4207-08 in the data path I get

Reading config file astrometry.cfg and loading indexes...

fitsbin.c:518:read_chunk: Expected table size (243360 => 85 FITS blocks) is not

equal to size of table "sweep" (1221120 => 424 FITS blocks).

[Sean Prange]

Dustin what version of cfitsio do you build with? I think that appears to be the only variable.

[Dustin Lang]

I see 3.29 and 3.37 on my machines.

Could you please send the result from running:

   liststruc index-4207-08.fits

(giving the full path to index-4207-08.fits)

thanks,
--dstn

[Sean Prange]

Well this is getting more complicated.  Ubuntu control-program still seg faults in the same location(s) as windows but it has no problem with the 4207-08 index file (correct size, checksum/etc--same file as in windows).  The cygwin control-program behaves exactly like the native windows one, including giving the error from the index file.  Attached is the output from cygwin liststruc.

[Dustin Lang]

I'm getting lost now.

Ubuntu: reads the index files without complaining, then segfaults?

What does solve-field on ubuntu (using the same config file and image to be solved, via solve-field --config XXX.cfg -v ...) do?

If you want to get gutsy, run control-program under valgrind and send the log file (it helps to do:
   valgrind control-program [args] > log 2>&1 &
and send log

That "liststruc" looks correct.  So I wonder why it thinks the file size is wrong...

--dstn

[Sean Prange]

On all platforms (ubuntu, cygwin, windows) control-program segfaults at the exact same locations as mentioned (NULL pointers).

Cygwin and windows complain about the index file as mentioned above, ubuntu doesn't.

Solve-field works fine on ubuntu and cygwin.  I can't build it for windows.

I may be able to get a valgrind log later, although when I ran it previously it just pointed out the memory access problem thanks to a null pointer.

Sean

[Sean Prange]

Here are a few valgrind logs from ubuntu.  Both 0.50 and git.master fail at the same point but valgrind won't work on git.master (complains about unknown instruction--my cpu is ivy-bridge-e).  Dustin do you have a relatively widefield fits image that can be solved with a smaller index file and is properly populated with files?  It would be awesome to see the control-program output from your end to compare as I swear your results when you posted successfully using it were different than mine (your output had more details after mentioning something about a CTYPE key).  I can also setup some sort of rdp/etc so you can access my ubuntu and windows build environments if your life isn't busy enough!

[Dustin Lang]

Thanks for the valgrind logs... not too much more information that before though.  Too bad.

One thing: it says it loaded 2 index files.  I thought you had isolated down to 4207-08.fits.  Could you please post the config file (or modify the attached one) and the 1.fit you are using?  I suspect it's the same as before, but since we're getting strange inconsistencies, let's try to make sure we're looking at exactly the same thing.

Thanks,
--dustin

(1.cfg):

# You may need to change this path...
add_path /usr/local/astrometry/data
index index-4207-08.fits
inparallel

I get:

> ./control-program -c 1.cfg 1.fit
Reading config file 1.cfg and loading indexes...
Loaded 1 indexes.
Reading image file 1.fit
simplexy: found 688 sources.
Trying to read WCS header from 1.fit...
Trying to verify existing WCS...
Existing WCS failed the verification test...
Failed to solve.
That took 4.02576 seconds
Sleeping...
Starting!

And with verboseness:

> ./control-program -c 1.cfg -v 1.fit
Reading config file 1.cfg and loading indexes...
Trying index index-4207-08.fits...
Trying path /data/INDEXES/4200/index-4207-08.fits...
Index name "/data/INDEXES/4200/index-4207-08.fits" is readable; assuming singe file.
Index name "/data/INDEXES/4200/index-4207-08.fits" is readable; assuming singe file.
Index name "/data/INDEXES/4200/index-4207-08.fits" is readable; assuming singe file.
Index name "/data/INDEXES/4200/index-4207-08.fits" is readable; assuming singe file.
Index scale: [22, 30] arcmin, [1320, 1800] arcsec
Index has 389376 quads and 243360 stars
Loaded 1 indexes.
Reading image file 1.fit
simplexy: nx=1931, ny=1451
simplexy: dpsf=1.000000, plim=8.000000, dlim=1.000000, saddle=5.000000
simplexy: maxper=1000, maxnpeaks=10000, maxsize=2000, halfbox=100
simplexy: median smoothing...
simplexy: measuring image noise (sigma)...
Sampling sigma at 7081 points
Nsigma=0.7, s=156.574
simplexy: found sigma=156.574.
simplexy: finding objects...
simplexy: found 606 blobs
simplexy: finding peaks...
Failed to find (5x5) centroid of peak 77, subpeak 0 at (1788,199)
...
simplexy: found 688 sources.
Trying to read WCS header from 1.fit...
Trying to verify existing WCS...
Existing WCS failed the verification test...
Quad scale range: [inf, nan] pixels
object 1 of 688: 0 quads tried, 0 matched.
object 2 of 688: 0 quads tried, 0 matched.
object 3 of 688: 0 quads tried, 0 matched.
...
object 688 of 688: 0 quads tried, 0 matched.
Failed to solve.
That took 4.66249 seconds
Sleeping...
Starting!
Reading image file 1.fit
...

[Sean Prange]

There definitely is something going on with the index file (even under ubuntu I am pretty certain that if I have 3 index files in the data path and one of them is 4207-08, then it will report 2 loaded).  However I'm still trying to determine why I get a segmentation fault in control-program.

I had mentioned that I built a version that tested for a null pointer and returned gracefully to the calling function (there were two functions that needed to be modified).  That version actually ran and tried solving (trying to match an ever increasing number of quads) but it failed to find a match.  That is when I decided to add more index files and discovered that 4207-08 seemed to have a problem.  If you have just 4208 as the index file you can solve my 1.fit correct ?

What version of gcc are you using and what other options are you using?  I think I have tried 4.8.2 and 4.9.0 (and 4.9.1).  Most of the time it was the default optimizations -O3 -march=native and the rest that are part of the project

In most of my windows test builds I had absolutely no optimizations and included debugging information.  

Oh, maybe I'm doing something wrong but to build with your makefile I had to add qsort_reentrant.o to anutils.  

[Dustin Lang]

Question, in your config file, do you have "inparallel" set?  Just looking through the control-program code, it looks like if it's NOT set, this bug could be triggered.  Looks like I override "inparallel", but too late in the processing...

--dstn

[Sean Prange]

I did NOT have inparallel set as I copied over my config file from a different installation.  I added it and now it looks like its working!  Well it's taking forever to find a match but it's not optimized (still has debugging info too) and only has one index file.  Terrific catch!

[Dustin Lang]

For that test image you sent before, you need index index-4207-02.fits .  4207-08 will never work; it covers the wrong part of the sky.

I just git pushed a fix for not having inparallel set.

cheers,
--dstn

PS -- qsort_reentrant.c gets #included in os-features.c if NEED_QSORT_R is set.

[Sean Prange]

Oh, and what I'm seeing is not that it isn't finding a solution, but it keeps finding the solution over and over (and I take a few words back, it is FAST even with the debug code and unoptimized..I just didn't realize that it found the solution).  Should be easy to fix (perhaps I accidentally removed a line of code).

I have a better build environment now so soon I'll be porting the rest of the programs required for solve-field, then windows users no longer have to deal with cygwin.  

[Dustin Lang]

Yeah, control-program is really a sketch of how you might code up the situation where you're getting a stream of images from a 'scope, solving them, and using the result to re-point the telescope, so that's why it loops over the input images.  It's not really a piece of code that performs a useful function per se :)

--dstn

[Sean Prange]

Cool..that is exactly one type project I'd like to flesh out.  I was thinking that it would load up all indexes and monitor a given path for new images.  It would then solve them and write the solution to a file.  Not as elegant as a built in library but a developer of say an autoguiding program could use it to quickly incorporate this feature.  Considering the speed at which it solves I think it should work well and be far more tolerant of astronomical seeing than even multi star based centroid measurements.  Thanks again for your help!!

[Bryan]

Sean:

I don't know if you are familiar with AstroTortilla.

It uses astrometry.net (locally) as the underlying engine and does blind-solves in Windows (via Cygwin).  It will re-direct the mount to a desired object within a few steps.

It's open source, if you wanted to see how they do i.

https://sourceforge.net/p/astrotortilla/home/Home/

Bryan

[Sean Prange]

Thanks Bryan,

I am familiar with Astrotortilla...I used it for a while and then coded my own project that could be scripted in MaximDL.  A windows port of astrometry.net will definitely help Astrotortilla however, and likely Sequence Generator Pro.  Both require cygwin which technically isn't that bad but to have tighter integration will require libraries/dlls/ or (shudder) C# objects and that isn't possible with cygwin.

Sean

[Bryan]

Agreed.

On Friday, July 18, 2014 12:52:42 PM UTC-6, Sean Prange wrote:

...snip...
 

[Sean Prange]

As an update I've run into one unexpected problem:  the binary output from GCC (mingw64 for windows) does not seem to be compatible with the predominant C compiler MS Visual Studio.  This means at best a Windows program would have to execute a command (solve-field, control-program/etc), not unlike what is necessary now with the dependency on Cygwin.  Perhaps somebody out there knows if it is possible to use a mingw64 dll in Visual Studio.

My goal was to be able to have the developer access DLLs or static libraries to take advantage of astrometry.net's solving functions so I've switched to the more difficult task of porting to Intel's C compiler.  It is C99 compliant but it lacks a ton of GCC-specific features that the astrometry.net codebase uses so even mundane tasks (such as reading the contents of a directory) require a lot of changes.  The open source community has a lot of drop-in replacements luckily but some require a bit of effort and it will be interesting to see what happens.  Anyway with the change in compiler it's been a setback but I am already up to building control-program and am battling linking problems.

I will return to completing the GCC port of the project which will hopefully be incorporated into the trunk, but the Intel (C99) specific port will likely need to be forked as all the changes will annoy the main developers.

Sean   

[Sean Prange]

If anyone is following, I've managed to build control-program with Intel's compiler.  It took some code hacking but it works.  There are still some problems with indexes that I can't track down yet.  Depending on the index file used the program will fail with an assert() (due to "sweep") after being solved.  Removing the assert check takes care of the symptom but is obviously not ideal.  So far this code is better suited for a forked project as it introduces several more .c and .h files (regex, mman, getopt, etc).  Pretty much all of the programs should be able to be ported although the python/netpbm dependency will take some work.    

[Sean Prange]

Lonely topic here.  The GIT master code hasn't been too successful for me.  My Intel compiler project worked reasonably well with 0.50 but had quite a few problems with the GIT code.  I was quite a bit more aggressive with that port so perhaps that is the issue.  The GIT master code compiled fine with MSYS2 (using a cygwin GCC toolchain) but the same code with minimal changes caused a segfault in qsort_r no matter what I did.  This was using the makefile and GNU make.  Manually changing preprocessor defines (SWAP or not, use AN qsort or MS version) all segfaulted at a location that the 0.50 code didn't.  

At this point my Intel port is faster than the GCC (Mingw64-gcc-4.9.1) version although if index files are not being loaded from a ramdrive (I have 32GB for a scratch ramdrive) I'm not sure it will matter.  Apparently I can use a GCC compiled DLL to code a C# wrapper, so having a native library is no longer an issue.  The windows world only seems to care about coding in C#, and as such a static library isn't required. 

[Dustin Lang]

Thanks for continuing to post your progress here, Sean!  It'd be great if you could post your code also.  Git(hub) branches might be a nice way to keep track of all your different efforts.  You never know when someone will come out of the woodwork with a helpful comment or suggestion.

cheers,
--dustin

[Sean Prange]

Can somebody post a 'Github for dummies link'?  I've never worked on a collaborated project so I'm not sure where to start.

[Dustin Lang]

If you're not used to git, the learning curve might be steep, since you're doing a lot of branching.  But it's probably better than the alternatives.

The documentation for git itself is here,
http://git-scm.com/documentation

and then if you follow the "bootcamp" links on the github front page, you'll get more doco,
https://github.com/

Plus there are tons of git documents on the web.  I find that I need to "jump in" and then search stackoverflow when I get stuck.  The documentation sometimes leads off in directions I don't care about and it's hard to keep plugging away, but the first chapter or two is great for getting an outline and foundation.

I think what you'd want to do is "fork" the astrometry.net repository on github -- then you've got your own copy owned by you, and you can do with it as you like.  Then you'd create a branch for each effort (gcc, icc, whatever).  If anything pans out, it will then be easy to merge the changes in to the mainline astrometry.net, if possible.  Even if that never happens, you could give links to your work when reporting on it here, which would be cool.

cheers,
--dustin

[bade...@power.ms]

Any news in the windows port project? A windows version of astrometry.net would be extremly cool!-)

Uwe

[Maik Riechert]

Dustin, I think it would be a good idea to set up AppVeyor CI which is travis ci for windows. Then we would have a reproducible Windows build, or rather can work on one. If you need help, I already used that for building native Python extensions, see https://ci.appveyor.com/project/neothemachine/lensfunpy/branch/master for example.

Maik

[Dustin Lang]

Hi,

If it's like TravisCI, then enabling it involves setting up an account and then adding a small text file to the astrometry.net repo.  That second thing could/should be done by someone who wants to see a windows build working, by sending a pull request  :)

cheers,
--dustin

[Maik Riechert]

I can do the appveyor part but obviously only if I know which compilers/environments etc. to use/test. It would be nice if Sean Prange could get back to us with his work. I think it would be quite valuable to get it compiled with the latest available Visual Studio compiler. The MSVC compilers are preinstalled also. Then with a bit more work we could target other compilers if there is need. Dustin, can you flip the switch on http://www.appveyor.com ? Sean, are you still in?

Cheers
Maik

[Dustin Lang]

Done.

[Maik Riechert]

Wow, just had a look at the makefiles again and totally forgot what a
beast astrometry.net is. Maybe it's better to start with something unixy
then, like cygwin. On the other hand... do you know CMake? If not:
http://prateekvjoshi.com/2014/02/01/cmake-vs-make/ If astrometry.net
used CMake then it would be quite easy to target all sorts of platforms,
it can generate Visual Studio makefiles for example. What do you think?

[Dustin Lang]

Send a pull request ;)

I have extremely limited time to spend on astrometry.net and this is, frankly, not a priority for me.  I'm happy to support this work, but I am not leading it.  "Scratch your own itch", as they say.

cheers,
--dstn

[Dustin Lang]

As for "beast"lyness: I feel like the only nasty part is the "os-features" auto-config junk, which you would just skip and hard-code in Windows anyway (by creating an "include/astrometry/os-features-config.h" file).  Otherwise, it's just a bunch of C files.  They are organized into directories and built into separate sub-libraries partly for historical reasons and partly to keep the dependency tree clean.  Am I missing something?
--dstn

[Maik Riechert]

It seems the util/ folder contains lots of stuff, meaning multiple
libraries. Any reason why they are not in single folders each with
separate Makefiles? libanbase, libanutils, libanfiles, and then the
Python files as well. Also, there's a comment in util/Makefile "#
Actually there are ANFILES_H mixed in here too...." for ANUTILS_H.
Should that worry me?

Another question, it says that libanbase is a dependency of libqfits.
Why? I thought qfits-an just contains some patches to the original qfits
but should't be related to astrometry.net otherwise?

Maik

[Dustin Lang]

Aha, that *is* beastly!

Originally, qfits-an was just some patches on qfits.  But qfits is not under development any more, and I had to make a number of significant fixes to make it thread-safe and to handle large files.  While doing this, it made sense (to me...) to use some of the core astrometry.net library functions, hence the definition of libanbase.  I ended up defining new versions of many of the user-facing qfits functions (anqfits*), so one possible path forward would be to pull out those, plus the original qfits modules they depend on, and move them all to a monster-sized util/ directory, and then merge some of the libraries (eg, utils + qfits + base).

Like you say, it might make sense to pull the "anfiles" sources into a subdirectory.

There is really no good reason for so many libraries.  One could easily just treat the project as one big flat list of C files.  I assume you have the option of doing that when creating a windows build script, right?  I think that would be easier & more sensible.

Since all the header files get installed into the same place, mixing ANUTILS_H and ANFILES_H is not really a problem.  (Not *nice*, but doesn't break anything)

Thanks for this!
--dustin

[Maik Riechert]

Polluting util/ even more is not a good idea I think. Producing a single
library is fine for me. In the end it depends on whether you want to
extract one of those libraries and release them separately. I guess the
only candidate for that would be libkd, the rest could be compiled into
a single astrometry library. But I really don't like the util/ folder.
It's too much in there which could be separated. Simple things like
putting the python files into an own folder. Can you tell me what net/
is all about? Is that nova? If yes, then I'd they this doesn't belong
into that repository at all. Right?

Maik

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "astrometry" group.
> Visit this group at http://groups.google.com/group/astrometry.

[Dustin Lang]

Right, libkd is the only part that makes sense to release separately.

Yes, net/ contains the nova web service.

Where would you put the python bindings of the code in util/?  (that is, util.i , which becomes the horribly-named astrometry.util.util module)

What's your proposal for how to split up the util/ directory?

cheers,
--dustin

[Maik Riechert]

First, I have to correct the documentation, the Makefile says the
dependency stack is

blind/libastrometry
catalogs/libcatalogs
util/libanfiles
libkd/libkd
util/libanutils
gsl-an/libgsl-an
qfits-an/libqfits
util/libanbase

after looking at the files, I determined this one:


blind
catalogs
util/libanfiles
util/libanfiles
util/libanutils
libkd-an (optional, why?)
qfitsan
util/libanbase
util/libanutils
qfitsan
util/libanbase

So, the dependencies catalog -> anfiles, anfiles -> anutils, and libkd
-> qfits are not documented. Is my tree correct now?

Also, in the makefile of libanfiles it says libkd is optional, what's
the usecase here to leave it out?

I tried to get an #include dependency graph but the only tool I could
find doesn't quite work (http://flourish.org/cinclude2dot/) because it
tries to find all referenced #include ".." files by their relative path.

The python bindings, for which parts are these? All of the util
libraries? anfiles, anutils, anbase?

Cheers
Maik

[Dustin Lang]

Hi,

Sorry, I guess I meant "this is a dependency stack ordering that works", not "this is a full dependency graph".  Thanks for fixing that.

I think at one point I wanted to build libkd without qfits-an (only used for FITS I/O).  But I think we can drop that.

I thought you could get gcc to print the headers it reads, in nested order.  Not a nicely formatted dependency graph... but a start.

The python bindings (in util/util.i) use anfiles, anutils, qfits-an, and anbase.  In libkd/spherematch there is pyspherematch.c to provide bindings for libkd.  In blind/ there is blind.i and plotstuff.i .

thanks for this,
--dustin

[Maik Riechert]

Just a small note that this non-Windows related discussion continues in the thread "CMake and folder/library restructuring" (https://groups.google.com/d/topic/astrometry/T3DTN0bRFIw/discussion).