Building on Debian 8 - make complains of <netpbm/pam.h> not found

[Mike Bergenstjerna]

OK,

I've been working on this overnight and while I'm getting close to the line, I can't make the last 1/2 mile!  I've had astrometry.net working except for annotated plots, so I worked my way through all the applicable advice listed here on the group, specifically setting NETPBM_LIB and NETPBM_INC environmental variables prior to MAKE or MAKE RECONFIG.  I build netpbm from source (apparently there's a problem with the debian packages for netpbm) and it's installed to /usr/local/netpbm, in which directory there are directories bin, include, lib and man, so I've been setting NETPBM_LIB to "-L/usr/local/netpbm/lib -lnetpbm" and NETPBM_INC to "/usr/local/netpbm/include" but now, when running MAKE (from scratch) or MAKE RECONFIG (retrying existing build) I get a failure as follows:

rm -f util/os-features-config.h util/makefile.os-features
make -C util config
make[1]: Entering directory '/home/mike/code/astrometry.net/util'
Makefile:75: makefile.os-features: No such file or directory

---- Error messages in the next few commands are not necessarily bugs ----
     (we're checking how things works on your computer)
rm -f os-features-makefile.log
( \
 echo "# This file is generated by util/Makefile."; \
 ((gcc -o os-features-test-netpbm-make \
     -g -Wall -std=gnu89 -ffinite-math-only -fno-signaling-nans -pthread -march=native -O3 -fomit-frame-pointer -DNDEBUG -fpic -Winline -I../include -I../include/astrometry -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -DAN_GIT_REVISION='"0.57"' -DAN_GIT_DATE='"Wed_Sep_2_11:53:22_2015_-0400"' -DAN_GIT_URL='"https://github.com/dstndstn/astrometry.net"' -I../util -I/usr/include/cairo -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng12   -I/usr/include/libpng12           -I../include -I../include/astrometry -I../gsl-an     -I../include -I../include/astrometry -I../gsl-an  -I. -DTEST_NETPBM_MAKE  os-features-test.c  -g -Wall -std=gnu89 -ffinite-math-only -fno-signaling-nans -pthread -march=native -O3 -fomit-frame-pointer -DNDEBUG -fpic -Winline -L. -lnetpbm >> os-features-makefile.log && \
   ./os-features-test-netpbm-make >> os-features-makefile.log && \
   echo "HAVE_NETPBM := yes") \
|| (echo "# Astrometry.net didn't find netpbm; not setting HAVE_NETPBM."; \
    echo "# See os-features-makefile.log for details."; \
    echo "# To re-run this test, do 'make reconfig; make makefile.os-features' (in the 'util' directory)"; \
    echo "# Or to do it yourself, just uncomment this line:"; \
    echo "# HAVE_NETPBM := yes")) \
; \
echo) > makefile.os-features.tmp
os-features-test.c:88:24: fatal error: netpbm/pam.h: No such file or directory
 #include <netpbm/pam.h>
                        ^
compilation terminated.
--------------- End of expected error messages -----------------

mv makefile.os-features.tmp makefile.os-features

Config results:
------------------------------

cat makefile.os-features
# This file is generated by util/Makefile.
# Astrometry.net didn't find netpbm; not setting HAVE_NETPBM.
# See os-features-makefile.log for details.
# To re-run this test, do 'make reconfig; make makefile.os-features' (in the 'util' directory)
# Or to do it yourself, just uncomment this line:
# HAVE_NETPBM := yes

------------------------------

And, by the way, is WCSlib support being compiled in?

pkg-config --exists wcslib && echo yes || echo no
no

  WCSLIB_INC:
  WCSLIB_LIB:

------------------------------
make[1]: Nothing to be done for 'config'.
make[1]: Leaving directory '/home/mike/code/astrometry.net/util'

I've tried to work out what's changed in my config to stop this building now, since I had it running without annotated plots early last night.  I'm going to have one more try from scratch by removing the build code (astrometry.net gets installed to /usr/local/astrometry) and checking out a fresh copy of the source, but I suspect I'm going to be in the same bind again.  Anyone seen this before and been able to fix it?

Many thanks for your time,
Mike

[Dustin Lang]

Hi,

I don't see "-I/usr/local/netpbm/include" in the compile line that is failing, so somehow the NETPBM_INC that you set isn't making its way through.

If you're using bash, do

export NETPBM_INC=-I/usr/local/netpbm/include
export NETPBM_LIB="-L/usr/local/netpbm/lib -lnetpbm"

One hack thing you can do (I have been known to do it myself) is to just create a symlink from the astrometry directory to the netpbm include directory;

ln -s /usr/local/netpbm/include/netpbm netpbm

If you 'git pull' from the source, I added a debug print of the value that the makefile has for NETPBM_INC when it's doing that test... maybe that'll help figure out what's up.

cheers,
--dustin

[Ole Streicher]

Hi Mike,

I am working on packaging astrometry.net for Debian, and I am almost done with it.

Am Mittwoch, 9. September 2015 15:49:34 UTC+2 schrieb Mike Bergenstjerna:

I build netpbm from source (apparently there's a problem with the debian packages for netpbm)

Could you explain what problem you have here? For me, the Debian package works well (except the compilation problem which you describe below).
 

os-features-test.c:88:24: fatal error: netpbm/pam.h: No such file or directory
 #include <netpbm/pam.h>
                        ^
compilation terminated.

You could try to replace all appearances of "#include netpbm/XXX.h" with "#include XXX.h":

--- a/util/an-pnmtofits.c
+++ b/util/an-pnmtofits.c
@@ -22,7 +22,7 @@
 #include "os-features-config.h" // for HAVE_NETPBM.
 
 #if HAVE_NETPBM
-#include <netpbm/pam.h>
+#include <pam.h>
 #else
 #include
 #endif
--- a/util/os-features-test.c
+++ b/util/os-features-test.c
@@ -85,7 +85,7 @@
 #endif
 
 #if defined(TEST_NETPBM) || defined(TEST_NETPBM_MAKE)
-#include <netpbm/pam.h>
+#include <pam.h>
 int main(int argc, char** args) {
        struct pam img;
        pm_init(args[0], 0);
--- a/util/cairoutils.c
+++ b/util/cairoutils.c
@@ -32,7 +32,7 @@
 #include "os-features.h"
 
 #if HAVE_NETPBM
-#include <netpbm/ppm.h>
+#include <ppm.h>
 #endif
 
 #include "ioutils.h"

Alternatively, would you serve as a beta tester for my package? If so, I could give you a link to the packages and installation instructions.

Best regards

Ole

[Mike Bergenstjerna]

Hi Ole,
I'd be more than happy to test the package out for you, please feel free to ping me the link and instructions.  I'm running jessie amd64 on my "main" machine, and wheezy arm6 on my Raspberry Pi.  I'll start by trying it on jessie if that's good for you?  I expect some additional work will be needed to get it going on the pi and for me that's a low priority task right now.

Dustin,
I've done a git pull and make reconfig is now saying

Testing netpbm...
   NETPBM_INC_ORIG is -I/usr/local/netpbm/include
   NETPBM_LIB_ORIG is -L/usr/local/netpbm/lib -lnetpbm

So it seems the environmental variable is being set to what I tell it, and what I tell it is where the libraries are.  I suspect I've got a lot of things in a twist, so I'm going to spend a bit of time removing both netpbm and astrometry before continuing.

Thanks to both of you for yor time and kind assistance!
Mike

[Dustin Lang]

Hi Mike,

Could you please post the whole interaction when you run "make reconfig"?  Now I'm stumped why your NETPBM_INC and LIB settings aren't appearing in the compile line (if indeed they're not...)

Here's where I set NETPBM_INC_ORIG
https://github.com/dstndstn/astrometry.net/blob/master/util/Makefile#L65

and here's where it is used:
https://github.com/dstndstn/astrometry.net/blob/master/util/Makefile#L318

But maybe if you post the whole 'make reconfig' log, something will pop out?

thanks,
--dustin

[Mike Bergenstjerna]

Here's the complete output from make reconfig:

mike@tesla:~/code/astrometry.net$ make reconfig
rm -f util/os-features-config.h util/makefile.os-features
make -C util config
make[1]: Entering directory '/home/mike/code/astrometry.net/util'
Makefile:61: makefile.os-features: No such file or directory

---- Error messages in the next few commands are not necessarily bugs ----
     (we

're checking how things works on your computer)
rm -f os-features-makefile.log

Testing netpbm...
   NETPBM_INC_ORIG is -I/usr/local/netpbm/include
   NETPBM_LIB_ORIG is -L/usr/local/netpbm/lib -lnetpbm

( \
 echo "# This file is generated by util/Makefile."; \
 ((gcc -o os-features-test-netpbm-make \

     -g -Wall -std=gnu89 -ffinite-math-only -fno-signaling-nans -pthread -march=native -O3 -fomit-frame-pointer -DNDEBUG -fpic -Winline -I../include -I../include/astrometry -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -DAN_GIT_REVISION='"0.57-4-gb2fabc3"' -DAN_GIT_DATE='"Wed_Sep_9_11:29:08_2015_-0400"' -DAN_GIT_URL='"https://github.com/dstndstn/astrometry.net"' -I../util -I/usr/include/cairo -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng12   -I/usr/include/libpng12           -I../include -I../include/astrometry -I../gsl-an     -I../include -I../include/astrometry -I../gsl-an  -I. -DTEST_NETPBM_MAKE -I/usr/local/netpbm/include os-features-test.c  -g -Wall -std=gnu89 -ffinite-math-only -fno-signaling-nans -pthread -march=native -O3 -fomit-frame-pointer -DNDEBUG -fpic -Winline -L/usr/local/netpbm/lib -lnetpbm >> os-features-makefile.log && \

   ./os-features-test-netpbm-make >> os-features-makefile.log && \
   echo "HAVE_NETPBM := yes") \
|| (echo "# Astrometry.net didn'

t find netpbm; not setting HAVE_NETPBM."; \
    echo "# See os-features-makefile.log for details."; \

    echo "# To re-run this test, do 'make reconfig; make makefile.os-features' (in the 'util' directory)"; \
    echo "# Or to do it yourself, just uncomment this line:"; \
    echo "# HAVE_NETPBM := yes")) \
; \
echo) > makefile.os-features.

tmp
os-features-test.c:75:24: fatal error: netpbm/pam.h: No such file or directory
 #include <netpbm/pam.h>
                        ^
compilation terminated.

--------------- End of expected error messages -----------------

mv makefile.os-features.tmp makefile.os-features

Config results:
------------------------------

cat makefile.os-features
# This file is generated by util/Makefile.
# Astrometry.net didn't find netpbm; not setting HAVE_NETPBM.
# See os-features-makefile.log for details.
# To re-run this test, do 'make reconfig; make makefile.os-features' (in the 'util' directory)
# Or to do it yourself, just uncomment this line:
# HAVE_NETPBM := yes

------------------------------

And, by the way, is WCSlib support being compiled in?

pkg-config --exists wcslib && echo yes || echo no
no

  WCSLIB_INC:
  WCSLIB_LIB:

------------------------------
make[1]: Nothing to be done for 'config'.
make[1]: Leaving directory '/home/mike/code/astrometry.net/util'

This is identical to what I got from vanilla make with the newly-pulled astrometry.net source, having removed and rebuilt netpbm using all default settings from source.  The netpbm executables, includes, library and man folders are still being put in /usr/local/netpbm as before.  Is there a switch I can use on make reconfig to provide any more verbose output?  I've only just noticed the line Makefile:61: makefile.os-features: No such file or directory - is that not slightly alarming?

With thanks,
Mike

[Mike Bergenstjerna]

Hi again Ole, I just realised I neglected to answer your first question, about the netpbm on jessie.  There are 2 netpbm versions available as packages, and both are supposed to install the "lumped" version of netpbm which contains a single executable and a load of symlinks which are used to invoke (what would normally be) individual commands.  I've installed both of these packages, and could find absolutely no trace of the symlinks anywhere in my file system, and this made me very doubtful that things had gone well - no symlinks and no individual executables and apparently no lib or include folders to me meant that really netpbm wasn't going to be available to link against.  Also, the packages are somewhat older than the latest source.  Are you using the standard apt-get repos to get the package, or somewhere else?  I'd much rather use the package is I could!

[Dustin Lang]

Hi,

What is in
/usr/local/netpbm/include
is there a "netpbm" directory, or just the files?

If it's just the files, try the symlink hack I mentioned.

The "missing" makefile.os-features isn't alarming -- that's the file it's building.

cheers,
--dustin



[Mike Bergenstjerna]

Just the files - from bitio.h to shhopt.h - I'll try the symlink hack again.  Thanks again!

** EDIT **
Using ths symlink hack, pointing to /usr/local/netpbm/include results in exactly the same error.  Symlinking to /usr/local/netpbm/include/netpbm fails, since there's no directory below include to link to.

[Dustin Lang]

It looks like the 'astrometry' base source directory isn't included in the "-I" include paths, so I would suggest creating the symlink in the util directory; you want it so that
util/netpbm/pam.h exists, so the "netpbm" is a symlink to "/usr/local/netpbm/include".

--dstn

[Mike Bergenstjerna]

OK cool, I've made the symlink in util and the error is now gone.  Unfortunately, it's been replaced with a fresh and exciting error of a different flavor:

./os-features-test-netpbm-make: error while loading shared libraries: libnetpbm.so.10: cannot open shared object file: No such file or directory

Now, I tried something I'd seen suggested elsewhere, which was setting

export NETPBM_LIB="-L/usr/local/netpbm/lib/libnetpbm.a"

prior to running make reconfig.  This changes things again, here's the full output:

rm -f util/os-features-config.h util/makefile.os-features
make -C util config
make[1]: Entering directory '/home/mike/code/astrometry.net/util'
Makefile:61: makefile.os-features: No such file or directory

---- Error messages in the next few commands are not necessarily bugs ----
     (we

're checking how things works on your computer)

rm -f os-features-makefile.log
Testing netpbm...
   NETPBM_INC_ORIG is -I/usr/local/netpbm/include

   NETPBM_LIB_ORIG is -L/usr/local/netpbm/lib/libnetpbm.a

( \
 echo "# This file is generated by util/Makefile."; \
 ((gcc -o os-features-test-netpbm-make \

     -g -Wall -std=gnu89 -ffinite-math-only -fno-signaling-nans -pthread -march=native -O3 -fomit-frame-pointer -DNDEBUG -fpic -Winline -I../include -I../include/astrometry -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -DAN_GIT_REVISION='"0.57-4-gb2fabc3"' -DAN_GIT_DATE='"Wed_Sep_9_11:29:08_2015_-0400"' -DAN_GIT_URL='"https://github.com/dstndstn/astrometry.net"' -I../util -I/usr/include/cairo -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng12   -I/usr/include/libpng12           -I../include -I../include/astrometry -I../gsl-an     -I../include -I../include/astrometry -I../gsl-an  -I. -DTEST_NETPBM_MAKE -I/usr/local/netpbm/include os-features-test.c  -g -Wall -std=gnu89 -ffinite-math-only -fno-signaling-nans -pthread -march=native -O3 -fomit-frame-pointer -DNDEBUG -fpic -Winline -L/usr/local/netpbm/lib/libnetpbm.a >> os-features-makefile.log && \

   ./os-features-test-netpbm-make >> os-features-makefile.log && \
   echo "HAVE_NETPBM := yes") \
|| (echo "# Astrometry.net didn'

t find netpbm; not setting HAVE_NETPBM."; \
    echo "# See os-features-makefile.log for details."; \

    echo "# To re-run this test, do 'make reconfig; make makefile.os-features' (in the 'util' directory)"; \
    echo "# Or to do it yourself, just uncomment this line:"; \
    echo "# HAVE_NETPBM := yes")) \
; \
echo) > makefile.os-features.tmp

/tmp/ccvlqWD5.o: In function `main':
/home/mike/code/astrometry.net/util/os-features-test.c:78: undefined reference to `pm_init'
collect2: error: ld returned 1 exit status

--------------- End of expected error messages -----------------

mv makefile.os-features.tmp makefile.os-features

Config results:
------------------------------

cat makefile.os-features
# This file is generated by util/Makefile.
# Astrometry.net didn'

t find netpbm; not setting HAVE_NETPBM.

# See os-features-makefile.log for details.
# To re-run this test, do 'make reconfig; make makefile.os-features' (in the 'util' directory)
# Or to do it yourself, just uncomment this line:
# HAVE_NETPBM := yes

------------------------------

And, by the way, is WCSlib support being compiled in?

pkg-config --exists wcslib && echo yes || echo no
no

  WCSLIB_INC:
  WCSLIB_LIB:

------------------------------
make[1]: Nothing to be done for 'config'.
make[1]: Leaving directory '/home/mike/code/astrometry.net/util'

Some days, I could just cry ;)  I take it it will ALWAYS say it can't find netpbm if the build of os-features-test fails?  Should I just try and force it through?

[Dustin Lang]

Hi,

If you're using a dynamic library (.so), then at runtime it's got to be able to find the library.

If your library is in a non-standard place -- your is -- you've got to add that path to the environment variable LD_LIBRARY_PATH; eg,

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/netpbm/lib

(you could put that in your ~/.bashrc)


You *could* link to the static library; to do that, omit the "-L";

export NETPBM_LIB="/usr/local/netpbm/lib/libnetpbm.a"

and if you do that, you don't need to set LD_LIBRARY_PATH.

cheers,
--dustin




[Mike Bergenstjerna]

OK that solved the problem of not being able to find the library, beautiful!  Insamely though I now have a new error when trying to solve a field:

mike@tesla:~$ solve-field M57.png
Reading input file 1 of 1: "M57.png"...
pngtopnm: symbol lookup error: pngtopnm: undefined symbol: overflow2
Command failed: pngtopnm M57.png > /tmp/tmpx_LH2C.pnm
augment-xylist.c:588:backtick Failed to run command: /usr/local/astrometry/bin/image2pnm.py --sanitized-fits-outfile /tmp/tmp.sanitized.SH8tBY --fix-sdss --infile M57.png --uncompressed-outfile /tmp/tmp.uncompressed.ymJTTG --outfile /tmp/tmp.ppm.sLJjcp --ppm
 ioutils.c:566:run_command_get_outputs Command failed: return value 255
mike@tesla:~$ solve-field M57.png
Reading input file 1 of 1: "M57.png"...
pngtopnm: symbol lookup error: pngtopnm: undefined symbol: overflow2
Command failed: pngtopnm M57.png > /tmp/tmp5ZXYA8.pnm
augment-xylist.c:588:backtick Failed to run command: /usr/local/astrometry/bin/image2pnm.py --sanitized-fits-outfile /tmp/tmp.sanitized.cO48YC --fix-sdss --infile M57.png --uncompressed-outfile /tmp/tmp.uncompressed.aKcODs --outfile /tmp/tmp.ppm.kRItii --ppm
 ioutils.c:566:run_command_get_outputs Command failed: return value 255

I'm seriously thinking of shifting to another distro away from Debian.  What distro does the live service run on?  It's a problem my end rather than with the code, and this is starting to be a big block on my project!

[Dustin Lang]

Hmm!

Do you get that same error if you just run "pngtopnm" from the command line?

If your PATH points to /usr/local/netpbm/bin, then you should also set LD_LIBRARY_PATH -- you want the binaries and libraries to be in sync!

You can do "ldd $(which pngtopnm)" to see which dynamic libs that binary is linked to.

cheers,
--dstn

[Mike Bergenstjerna]

ldd reports

linux-vdso.so.1 (0x00007ffc7a3e9000)
    libnetpbm.so.10 => /usr/local/netpbm/lib/libnetpbm.so.10 (0x00007f3395123000)
    libpng12.so.0 => /lib/x86_64-linux-gnu/libpng12.so.0 (0x00007f3394efc000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f3394ce1000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f33949e0000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3394637000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f3395358000)

which I think is what I was expecting - using the libnetpbm.so I built from source.  I'm seriously thinking there's something kooky with my OS, this is just getting out of hand now and this is completely a problem with my box, rather than your code.

[Dustin Lang]

And "which pngtopnm" ?  If there is one in /usr/bin or whatever, but it sees your library in /usr/local/netpbm, bad things will happen.

[Mike Bergenstjerna]

Just the one, in /usr/local/netpbm/bin.  Same result with find / -name pngtopnm

[Dustin Lang]

Huh, that's strange.

And the netpbm / libnetpbm0 or whatever packages on debian didn't work?  I thought they worked just fine.

cheers,
--dstn

[Mike Bergenstjerna]

Yea I'm going to go from scratch tomorrow - I've got a cloud-free night and there's some hydrogen I want to have a look at ;)  I'm hoping Ole will send a link to his test package, I can't make any more of a mess of things with that than I have with this method!

[Mike Bergenstjerna]

Ole was kind enough to send me details of the debian package he's working on, and it went on first time with no problems at all.  Solving works 100% with plots, solve time is fast and the output is robust.  Now, the next question is how to get a web interface running, so I can plug my c# wrapper in and polish it without hogging the nova.astrometry.net server!

Many many thanks to Dustin and Ole for the kind help and patience.  There are lots of projects where you wait weeks for a reply to a support request, and I'm really grateful this isn't one of them!  Astrometry.net is an excellent program and the team should be really proud of it.

Best,
Mike

[Dustin Lang]

Glad to hear you're up and running!

As for a web service: The code for the nova.astrometry.net site is publicly available (https://github.com/dstndstn/astrometry.net/tree/master/net) but also largely undocumented... :(

It uses the Django framework and runs as a WSGI app.  I have only ever used it under apache.  We use postgres as the database.

For nova, the web server and solving server are two different machines; the web server ssh'es into the compute server, but that is just an implementation detail.

Anyway, may be way too heavy-weight for your own use, but just mentioning that it exists.

cheers,
--dustin

[Mike Bergenstjerna]

Eeep!   I'm a Microsoft .NET web developer by trade so this is a little outside of my comfort zone, but isn't that part of what being an astronomer is all about?  Find or make a solution to whatever challenges you face and all that ;)  In fact I shouldn't think there's too much stopping me writing something in ASP.NET to interface with the solver from the same or any other box, as you say it's just implementation details.