Command line flags or env variables to indicate cross compiling?

[John Filleau]

I'm cross compiling astrometry.net-0.50 for an armv7a. The architecture specifically isn't important. What is important is that when I run "make" in the main code directory, it eventually calls this line from gsl-an/Makefile:186

./configure --enable-shared=no --prefix=`pwd`/stage

The configure script, however, requires a "--host=whatever" flag be passed to it to indicate I'm cross compiling. Otherwise, it attempts to actually *run* an executable after compiling.

I was able to modify the gsl-an/Makefile file to just pass "--host=whatever" to ./configure, but I'm trying to do this elegantly, as I want to include it in a bit-bake recipe.

Any tips?

-John

[Dustin Lang]

Hi,

You could perhaps add something like this to gsl-an/Makefile:

GSL_CONFIG_ARGS ?=

config.h: configure config.h.in
    ./configure --enable-shared=no --prefix=`pwd`/stage $(GSL_CONFIG_ARGS)
    touch $@


And, I should also point out that you might encounter some trouble with the auto-config stuff we do in the main astrometry.net codebase.  It's called "os-features" and is a little home-baked config setup that tries to compile and run executables to find out things about the system.  It only has to figure out a few things, and writes a couple of files:

include/astrometry/os-features-config.h, containing, eg:

#define NEED_CANONICALIZE_FILE_NAME 1
#define NEED_DECLARE_QSORT_R 1
#define NEED_QSORT_R 0
#define NEED_SWAP_QSORT_R 0
#define HAVE_NETPBM 1

You can find out what those mean by looking at include/astrometry/os-features.h and util/os-features.c

It also writes  util/makefile.os-features , containing, eg:

# This file is generated by util/Makefile.
HAVE_NETPBM := yes

You might want to just figure out how each of those settings should be set and write your own os-features-config.h and makefile.os-features.  Happy to help with that if it's not clear.

If you can think of a better way of doing that when cross-compiling, I'm all ears.

cheers,
--dustin

[John Filleau]

The previous comments helped! To automate, I made a few patches to modify the makefiles as needed. I'll release all of this as a big chunk of data once I'm done. Until then, my cross-compiling saga continues. At some point in the "make install" part of compilation, we try to compile util.c, or util_wrap.c. I think as part of the process it calls util/setup.py, and that figures out which "-I" flags to set down the line. Now somewhere - SOMEWHERE - in the process, the build system figures that two of my local directories are the way to go when we compile: /usr/lib/python2.7/dist-packages/numpy/core/include & /usr/include/python2.7.

Any idea where these are set, and what I could modify to make them settable via command line or environment variables?

make[3]: Entering directory `/tmp/john_sysroot_build/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/astrometry-0.50-r0/astrometry.net-0.50/util'

LDFLAGS="-Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -g -Wall -ffinite-math-only -fno-signaling-nans -pthread  -O3 -fomit-frame-pointer -DNDEBUG -fpic -Winline -Wl,-O1 -Wl,--hash-style=gn    u -Wl,--as-needed -Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -g -Wall -ffinite-math-only -fno-signaling-nans -pthread  -O3 -fomit-frame-pointer -DNDEBUG -fpic -Winline -g -Wall -ffinite-math-only -fno-signaling-nans -pthread  -O3 -    fomit-frame-pointer -DNDEBUG -fpic -Winline" LDLIBS="          -lm        -lm" SLIB="../util/libanfiles.a ../libkd/libkd.a ../util/libanutils.a  ../qfits-an/libqfits.a ../util/libanbase.a     ../gsl-an/libgsl-an.a  ../util/libanutils    .a  ../qfits-an/libqfits.a ../util/libanbase.a     ../gsl-an/libgsl-an.a " \

   INC="     -I../include -I../include/astrometry -I../gsl-an     -I../include -I../include/astrometry -I../gsl-an " CFLAGS="  " \

   python setup.py build_ext -v --inplace --build-temp .

link: ['-Wl,-O1', '-Wl,--hash-style=gnu', '-Wl,--as-needed', '-Wl,-O1', '-Wl,--hash-style=gnu', '-Wl,--as-needed', '-g', '-Wall', '-ffinite-math-only', '-fno-signaling-nans', '-pthread', '-O3', '-fomit-frame-pointer', '-DNDEBUG', '-f    pic', '-Winline', '-Wl,-O1', '-Wl,--hash-style=gnu', '-Wl,--as-needed', '-Wl,-O1', '-Wl,--hash-style=gnu', '-Wl,--as-needed', '-g', '-Wall', '-ffinite-math-only', '-fno-signaling-nans', '-pthread', '-O3', '-fomit-frame-pointer', '-DN    DEBUG', '-fpic', '-Winline', '-g', '-Wall', '-ffinite-math-only', '-fno-signaling-nans', '-pthread', '-O3', '-fomit-frame-pointer', '-DNDEBUG', '-fpic', '-Winline', '-lm', '-lm']

objs: ['../util/libanfiles.a', '../libkd/libkd.a', '../util/libanutils.a', '../qfits-an/libqfits.a', '../util/libanbase.a', '../gsl-an/libgsl-an.a', '../util/libanutils.a', '../qfits-an/libqfits.a', '../util/libanbase.a', '../gsl-an/    libgsl-an.a']

inc: ['../include', '../include/astrometry', '../gsl-an', '../include', '../include/astrometry', '../gsl-an']

cflags: []

numpy_inc: ['/usr/lib/python2.7/dist-packages/numpy/core/include']

running build_ext

building '_util' extension

swigging util.i to util_wrap.c

C compiler: arm-poky-linux-gnueabi-gcc -march=armv7-a -mthumb-interwork -mfloat-abi=softfp -mfpu=neon -mtune=cortex-a9 --sysroot=/tmp/john_sysroot_build/tmp/sysroots/zynq-zx3-zamak -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -    Wstrict-prototypes -fPIC

compile options: '-I/usr/lib/python2.7/dist-packages/numpy/core/include -I../include -I../include/astrometry -I../gsl-an -I../include -I../include/astrometry -I../gsl-an -I. -I/usr/include/python2.7 -c'

arm-poky-linux-gnueabi-gcc: util_wrap.c

cc1: warning: include location "/usr/include/python2.7" is unsafe for cross-compilation [-Wpoison-system-directories]

In file included from /usr/include/python2.7/Python.h:58:0,

                 from util_wrap.c:125:

[Dustin Lang]

Hi,

That must be while building the python bindings.  You don't need that, almost certainly.  It's supposed to be optional -- try to install the "core", and then try to build and install "extras".

There is a "make install-core" target.

To your question, that's probably in util/setup.py ... get_numpy_include_dirs().  How to convince that to cross-compils is beyond my knowledge.

--dstn

[John Filleau]

I hadn't even considered it was optional!

The get_numpy_include_dirs() certainly does include the /usr/lib/python2.7/dist-packages/numpy/core/include directory. It does not, however, include /usr/include/python2.7

I'll try just installing core - thanks!

[Dustin Lang]

Hi,

 

I hadn't even considered it was optional!

The makefile is supposed to (a) not fail the whole "make install" and (b) print something pseudo-useful when it fails to build the optional stuff.  Did that fail?

 

The get_numpy_include_dirs() certainly does include the /usr/lib/python2.7/dist-packages/numpy/core/include directory. It does not, however, include /usr/include/python2.7

Probably the python include directory comes from the setup.py / distutils stuff.  It does its own build magic.

--dstn

[John Filleau]

Dustin,

Currently, the build is mostly succeeding:

"Make" runs/compiles just fine

"Make install" runs/compiles fine, except for the attempt at compiling util_wrap.c

Right now I have two options, depending on the utility of astrometry.net:

1. If I can use solve-field without the contents of util, then I'm good. I can load this on my arm and see if I can solve images

2. If solve-field requires util, and specifically the product of util_wrap.c, then I need to do some more hacking before I'm done

Thanks,

John

[Dustin Lang]

The part where it's failing is in building python bindings of the C code -- totally optional for solving.  All the stuff not in 'install-core' is optional w.r.t solving.

solve-field does try to run some python programs (uniformize.py, fits2fits.py), but none of those use the python bindings, and indeed they are also all optional (--no-fits2fits, --uniformize 0, --no-remove-lines?)

So you should be good to go to run on the ARM!

cheers,
--dustin

[John Filleau]

Excellent. I just need to integrate this into my build system to make sure it works, load it on the arm, and solve an image! The resultant product will be something called a "bitbake" file, which I'll push to the Yocto project (linked in my first message here). I'll post further instructions when I'm done, in the hopes that it's helpful to somebody who wants to do something as crazy as this in the future.

Thanks for your help.

-John

[John Filleau]

Still chugging along trying to get it to run - 

When I try to run solve-field on my ARM, there's still a lot of python dependencies that are making it crash. Of course I can go through and find these packages, cross compile those as required, and move them to my file system on the ARM, but I'd like to avoid any python use, if possible. Solve-field required subprocess, so I moved that over. But then subprocess needed something in the utilib package - which is where I am now. I wanted to ask this question before I got too deep in the rabbit hole. Can you think of a way to eliminate all run-time python requirements?

Also, it looks like "make install-core" invokes

$(MAKE) -C util  install

$(MAKE) -C catalogs install

$(MAKE) -C libkd install

$(MAKE) -C qfits-an install

$(MAKE) -C blind install

$(MAKE) -C sdss install

Where "make -C util install" is what was giving me the earlier issues with including local python libraries.

Any ideas? I understand this is probably not the most common set of questions. At the very least I'm learning a lot I didn't know before.

Thanks,

John

[Dustin Lang]

Hi,

I just added a new Makefile target, install-core in util;

https://github.com/dstndstn/astrometry.net/commit/a683acbdce923f6dd926c4b878443b25677bbe75

Ok, so there are a few pure-python programs called by solve-field.  Most, but not all, can be avoided.

--no-fits2fits    (fits2fits.py just tries to fix up slightly broken FITS files)
--no-remove-lines    (tries to removes lines of sources, eg due to bad columns)
--uniformize 0       (spatially uniformizes the detected sources)

However, one more mainline thing is 'image2pnm.py', which we use to get images in JPEG, etc formats into PNM and then into FITS.  If you have FITS images (or can convert your images to FITS outside of solve-field) then you can avoid this.

None of the compiled python stuff is necessary to run solve-field.

cheers,
--dustin

[John Filleau]

Thank you! I was just hacking away at this when I decided to check the discussion page. I was traveling for the past week, so I didn't see this until now. Reading the suggested command line options above (and the warning about image2pnm) - you were spot on about all of them. I really do appreciate how prompt you are with your responses. All of that helps - thanks.

[kiran kumar]

HI I am trying to build libmodbus on my fedora system. I was told to do suitable changes so that instead of GCC compiler my customised compiler used.

I did following steps:
./configure
make
Inside make file it said CC = gcc
 then i changed the environment variable/path in bashrc[ inside etc the path that contain compiler's binary]
When i type which 'CC' on on my terminal it say

 /usr/bin/which: no CC in (/usr/lib/qt-3.3/bin:/usr/lib/ccache:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/opt/buildroot/build_xtensa_c1lx2/staging_dir/usr/bin:/home/sachin/.local/bin:/home/sachin/bin:/opt/buildroot/build_xtensa_c1lx2/staging_dir/usr/bin:/opt/buildroot/build_xtensa_c1lx2/staging_dir/usr/bin)
 
Above highlighted is the path to my compiler and when I look inside my makefile it shows CC = gcc and LD = /usr/bin/ld i want them to say xtensa_c1lx2-linux-uclibc-gcc  that is my compiler. Any suggestions will be appreciated and i am new to linux environment.

[Dustin Lang]

Hi,

This is not the right forum for this question.

But you could try setting CC and LD in the shell rather than the Makefile.  If you're using bash,

export CC=whatever
export LD=whatld
make ...


--dustin