making a static binary

[shane...@gmail.com]

Hi, I distribute cwebp as part of the EWWW Image Optimizer for WordPress, and as such I generally try to provide a static binary for the widest support across varying linux distributions. I was doing some testing today with the binaries provided from the official downloads page, and noticed that you had managed to disable some dynamic links that my builds are still showing. In particular, libjpeg, libpng, and libz. So, I was wondering if anyone knew what flags were used to compile the official binaries for linux?

Thanks,

Shane

[James Zern]

Hi,

On Saturday, March 5, 2016 at 11:51:53 PM UTC-8, shane...@gmail.com wrote:

Hi, I distribute cwebp as part of the EWWW Image Optimizer for WordPress, and as such I generally try to provide a static binary for the widest support across varying linux distributions. I was doing some testing today with the binaries provided from the official downloads page, and noticed that you had managed to disable some dynamic links that my builds are still showing. In particular, libjpeg, libpng, and libz. So, I was wondering if anyone knew what flags were used to compile the official binaries for linux?

The binaries from the download page are built with custom builds of the libraries in the LIBRARY_PATH, mostly to enable some metadata features in libpng which are disabled by default in libpng12. You could set something similar up and do:

$ LIBRARY_PATH=<local_lib_path> ./configure ...

or if your distribution packages the static libraries:

$ ./configure --disable-shared --disable-tiff  LIBS='-Wl,-static,-lpng,-ljpeg,-lz,-Bdynamic'

I left off tiff for brevity as it may have a few dependencies. You can also try -static -static-libgcc, but that may require additional system libs. The above only targets the image libs.

[sbi...@dawson.edu]

Ok, so I got all that to work, I had to specify the LIBPNG_CONFIG var as well though, otherwise it tried to pull in system libs (dynamically).
I wondered about the '-static -static-libgcc' flags though. I've used those in the past for other builds and passed them as LDFLAGS, but when I do that with libwebp, it won't configure.
Here's my full command:
LDFLAGS="-static -static-libgcc" LIBRARY_PATH="/opt/libs" LIBPNG_CONFIG=/opt/bin/libpng-config CFLAGS="-Os" LIBS="-Wl,-static,-lpng,-ljpeg,-lz,-Bdynamic" ./configure --disable-shared --disable-tiff --includedir=/opt/bin/include

And the results (abbreviated):
configure: error: cannot run C compiled programs.

config.log ends like so:
Target: i386-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-libgcj-multifile --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --disable-plugin --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic --host=i386-redhat-linux
Thread model: posix
gcc version 4.1.2 20080704 (Red Hat 4.1.2-55)
configure:3380: $? = 0
configure:3369: gcc -V >&5
gcc: '-V' option must have argument
configure:3380: $? = 1
configure:3369: gcc -qversion >&5
gcc: unrecognized option '-qversion'
gcc: no input files
configure:3380: $? = 1
configure:3400: checking whether the C compiler works
configure:3422: gcc -Os  -static -static-libgcc conftest.c -Wl,-static,-lpng,-ljpeg,-lz,-Bdynamic >&5
configure:3426: $? = 0
configure:3474: result: yes
configure:3477: checking for C compiler default output file name
configure:3479: result: a.out
configure:3485: checking for suffix of executables
configure:3492: gcc -o conftest -Os  -static -static-libgcc conftest.c -Wl,-static,-lpng,-ljpeg,-lz,-Bdynamic >&5
configure:3496: $? = 0
configure:3518: result:
configure:3540: checking whether we are cross compiling
configure:3548: gcc -o conftest -Os  -static -static-libgcc conftest.c -Wl,-static,-lpng,-ljpeg,-lz,-Bdynamic >&5
configure:3552: $? = 0
configure:3559: ./conftest
./configure: ./conftest: /usr/lib/libc.so.1: bad ELF interpreter: No such file or directory
configure:3563: $? = 126
configure:3570: error: in `/root/libwebp-0.5.0':
configure:3572: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.

Any ideas?

[James Zern]

On Tuesday, March 8, 2016 at 1:24:36 PM UTC-8, sbi...@dawson.edu wrote:

On Monday, March 7, 2016 at 8:14:13 PM UTC-7, James Zern wrote:

Hi,

On Saturday, March 5, 2016 at 11:51:53 PM UTC-8, shane...@gmail.com wrote:

Hi, I distribute cwebp as part of the EWWW Image Optimizer for WordPress, and as such I generally try to provide a static binary for the widest support across varying linux distributions. I was doing some testing today with the binaries provided from the official downloads page, and noticed that you had managed to disable some dynamic links that my builds are still showing. In particular, libjpeg, libpng, and libz. So, I was wondering if anyone knew what flags were used to compile the official binaries for linux?

The binaries from the download page are built with custom builds of the libraries in the LIBRARY_PATH, mostly to enable some metadata features in libpng which are disabled by default in libpng12. You could set something similar up and do:

$ LIBRARY_PATH=<local_lib_path> ./configure ...

or if your distribution packages the static libraries:

$ ./configure --disable-shared --disable-tiff  LIBS='-Wl,-static,-lpng,-ljpeg,-lz,-Bdynamic'

I left off tiff for brevity as it may have a few dependencies. You can also try -static -static-libgcc, but that may require additional system libs. The above only targets the image libs.

Ok, so I got all that to work, I had to specify the LIBPNG_CONFIG var as well though, otherwise it tried to pull in system libs (dynamically).
I wondered about the '-static -static-libgcc' flags though. I've used those in the past for other builds and passed them as LDFLAGS, but when I do that with libwebp, it won't configure.
Here's my full command:
LDFLAGS="-static -static-libgcc" LIBRARY_PATH="/opt/libs" LIBPNG_CONFIG=/opt/bin/libpng-config CFLAGS="-Os" LIBS="-Wl,-static,-lpng,-ljpeg,-lz,-Bdynamic" ./configure --disable-shared --disable-tiff --includedir=/opt/bin/include

Note the variables can be supplied as configure arguments: './configure LIBS=...'. This allows the full command to be seen in config.log.

 

And the results (abbreviated):
configure: error: cannot run C compiled programs.

config.log ends like so:

[...]

configure:3548: gcc -o conftest -Os  -static -static-libgcc conftest.c -Wl,-static,-lpng,-ljpeg,-lz,-Bdynamic >&5
configure:3552: $? = 0
configure:3559: ./conftest
./configure: ./conftest: /usr/lib/libc.so.1: bad ELF interpreter: No such file or directory
configure:3563: $? = 126
configure:3570: error: in `/root/libwebp-0.5.0':
configure:3572: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.

Any ideas?

The remaining -Bdynamic is causing this. If you want a full static build then -static -static-libgcc should be enough. 

[shane...@gmail.com]

On Tuesday, March 8, 2016 at 6:38:47 PM UTC-7, James Zern wrote:

Here's my full command:
LDFLAGS="-static -static-libgcc" LIBRARY_PATH="/opt/libs" LIBPNG_CONFIG=/opt/bin/libpng-config CFLAGS="-Os" LIBS="-Wl,-static,-lpng,-ljpeg,-lz,-Bdynamic" ./configure --disable-shared --disable-tiff --includedir=/opt/bin/include

Note the variables can be supplied as configure arguments: './configure LIBS=...'. This allows the full command to be seen in config.log.

 

I think I had moved them to the front-end because I was trying to figure out why things were failing, and that had worked before on other platforms. But it didn't work either way for the LDFLAGS previously.

 

The remaining -Bdynamic is causing this. If you want a full static build then -static -static-libgcc should be enough. 

So I removed -Bdynamic and ran configure like so:

 ./configure --disable-shared --disable-tiff --includedir=/opt/bin/include LDFLAGS="-static -static-libgcc" LIBRARY_PATH="/opt/lib" LIBPNG_CONFIG=/opt/bin/libpng-config CFLAGS="-Os" LIBS="-Wl,-static,-lpng,-ljpeg,-lz"

That worked, but then make failed with this error:

make[1]: Entering directory `/root/libwebp-0.5.0/examples'

  CC       example_util.lo

  CCLD     libexampleutil.la

  CC       dwebp-dwebp.o

  CCLD     dwebp

/usr/lib/gcc/x86_64-redhat-linux/4.1.2/libgcc_eh.a(unwind-dw2-fde-glibc.o): In function `_Unwind_Find_FDE':

(.text+0x1522): undefined reference to `dl_iterate_phdr'

collect2: ld returned 1 exit status

make[1]: *** [dwebp] Error 1

make[1]: Leaving directory `/root/libwebp-0.5.0/examples'

make: *** [all-recursive] Error 1

I don't need dwebp, only cwebp, is there a way to bypass that so I can see if cwebp will build? Or is the compiler on CentOS 5 causing some compatibility problems?

[Shane Bishop]

I realized I wasn't quite following your instructions very well, so I started over a bit, this time on CentOS 6, just to see if it would resolve the glibc error.

I compiled zlib, libpng, and libjpeg statically, and installed in /usr/local/ which made things simpler, so I got to libwebp and ran this:

./configure --disable-shared --disable-tiff CFLAGS="-Os" LDFLAGS="-static -static-libgcc" LIBS="-Wl,-static,-lpng,-ljpeg,-lz"

That part worked, and it showed that JPG and PNG support were detected, but make failed with this, which seems similar to the issue I had on CentOS 5:

make[1]: Entering directory `/root/libwebp-0.5.0/examples'

  CC       example_util.lo

  CCLD     libexampleutil.la

  CC       dwebp-dwebp.o

  CCLD     dwebp

/usr/lib/gcc/i686-redhat-linux/4.4.7/../../../libpthread.a(libpthread.o): In function `sem_open':

(.text+0x708a): warning: the use of `mktemp' is dangerous, better use `mkstemp'

/usr/lib/gcc/i686-redhat-linux/4.4.7/libgcc_eh.a(unwind-dw2.o): In function `.L107':

(.text+0xbb9): undefined reference to `__stack_chk_fail_local'

/usr/lib/gcc/i686-redhat-linux/4.4.7/libgcc_eh.a(unwind-dw2.o): In function `.L364':

(.text+0x1ac5): undefined reference to `__stack_chk_fail_local'

/usr/lib/gcc/i686-redhat-linux/4.4.7/libgcc_eh.a(unwind-dw2.o): In function `_Unwind_Backtrace':

(.text+0x24ff): undefined reference to `__stack_chk_fail_local'

/usr/lib/gcc/i686-redhat-linux/4.4.7/libgcc_eh.a(unwind-dw2.o): In function `_Unwind_ForcedUnwind':

(.text+0x26ed): undefined reference to `__stack_chk_fail_local'

/usr/lib/gcc/i686-redhat-linux/4.4.7/libgcc_eh.a(unwind-dw2.o): In function `__frame_state_for':

(.text+0x2a28): undefined reference to `__stack_chk_fail_local'

/usr/lib/gcc/i686-redhat-linux/4.4.7/libgcc_eh.a(unwind-dw2.o):(.text+0x2bb8): more undefined references to `__stack_chk_fail_local' follow

/usr/lib/gcc/i686-redhat-linux/4.4.7/libgcc_eh.a(unwind-dw2-fde-glibc.o): In function `_Unwind_Find_FDE':

(.text+0x19cb): undefined reference to `dl_iterate_phdr'

/usr/bin/ld: dwebp: hidden symbol `__stack_chk_fail_local' isn't defined

/usr/bin/ld: final link failed: Nonrepresentable section on output

[James Zern]

On Tuesday, March 8, 2016 at 6:03:52 PM UTC-8, Shane Bishop wrote:

[...]

make[1]: *** [dwebp] Error 1

make[1]: Leaving directory `/root/libwebp-0.5.0/examples'

make: *** [all-recursive] Error 1

I don't need dwebp, only cwebp, is there a way to bypass that so I can see if cwebp will build? Or is the compiler on CentOS 5 causing some compatibility problems?

You can do 'make -k' (keep going), but you'll probably run into the same problem. 

[James Zern]

On Tuesday, March 8, 2016 at 8:31:19 PM UTC-8, Shane Bishop wrote:

I realized I wasn't quite following your instructions very well, so I started over a bit, this time on CentOS 6, just to see if it would resolve the glibc error.

I compiled zlib, libpng, and libjpeg statically, and installed in /usr/local/ which made things simpler, so I got to libwebp and ran this:

./configure --disable-shared --disable-tiff CFLAGS="-Os" LDFLAGS="-static -static-libgcc" LIBS="-Wl,-static,-lpng,-ljpeg,-lz"

Let's simplify this and use makefile.unix (make -f  makefile.unix) instead, autoconf/libtool may be complicating things by stripping the -static from the link step.

Just edit makefile.unix's EXTRA_FLAGS to include '-static -static-libgcc'; you may also need some combination of -fno-stack-protect / -D_FORTIFY_SOURCE=0 / <some other flag I'm forgetting>. EXTRA_FLAGS works for me if I remove -DWEBP_HAVE_TIFF as I didn't want to add in the rest of its dependencies.

[Shane Bishop]

On Wednesday, March 9, 2016 at 8:00:19 PM UTC-7, James Zern wrote:

On Tuesday, March 8, 2016 at 8:31:19 PM UTC-8, Shane Bishop wrote:

I realized I wasn't quite following your instructions very well, so I started over a bit, this time on CentOS 6, just to see if it would resolve the glibc error.

I compiled zlib, libpng, and libjpeg statically, and installed in /usr/local/ which made things simpler, so I got to libwebp and ran this:

./configure --disable-shared --disable-tiff CFLAGS="-Os" LDFLAGS="-static -static-libgcc" LIBS="-Wl,-static,-lpng,-ljpeg,-lz"

Let's simplify this and use makefile.unix (make -f  makefile.unix) instead, autoconf/libtool may be complicating things by stripping the -static from the link step.

Just edit makefile.unix's EXTRA_FLAGS to include '-static -static-libgcc'; you may also need some combination of -fno-stack-protect / -D_FORTIFY_SOURCE=0 / <some other flag I'm forgetting>. EXTRA_FLAGS works for me if I remove -DWEBP_HAVE_TIFF as I didn't want to add in the rest of its dependencies.

I think that did the trick, I made a couple other edits, and when I run 'ldd examples/cwebp' it says 'not a dynamic executable'. And the version string prints successfully. Now to test it out fully! 

[Shane Bishop]

Confirmed working on CentOS 5,6,7 32&64-bit, Debian 7&8, Ubuntu 12.04, 14.04 and 15.10

In all, these are the lines I changed:

EXTRA_FLAGS = -DWEBP_HAVE_PNG -DWEBP_HAVE_JPEG -static -static-libgcc

CWEBP_LIBS= $(DWEBP_LIBS) -ljpeg -Wl
CFLAGS = -Os -DNDEBUG $(EXTRA_FLAGS)

I modified the CFLAGS from -O3 to -Os so it is optimized for smallest size, old habit, but it makes the binary much smaller usually, about 1.2 MB in this case.

Also removed -DWEBP_HAVE_TIFF as you said, and added -static and -static-libgcc to the EXTRA_FLAGS and then added '-Wl' to the CWEBP_LIBS to match what you had originally suggested for LIBS.

Thank you so much for your help!

[James Zern]

On Wednesday, March 9, 2016 at 8:07:04 PM UTC-8, Shane Bishop wrote:

Confirmed working on CentOS 5,6,7 32&64-bit, Debian 7&8, Ubuntu 12.04, 14.04 and 15.10

In all, these are the lines I changed:

EXTRA_FLAGS = -DWEBP_HAVE_PNG -DWEBP_HAVE_JPEG -static -static-libgcc

CWEBP_LIBS= $(DWEBP_LIBS) -ljpeg -Wl
CFLAGS = -Os -DNDEBUG $(EXTRA_FLAGS)

I modified the CFLAGS from -O3 to -Os so it is optimized for smallest size, old habit, but it makes the binary much smaller usually, about 1.2 MB in this case.

You might try stripping the binary as well, in general webp performs better at -O2/-O3. I don't have numbers for encoding, but a change I made for the decoder on Android (ARM) boosted performance 11-22% depending on the image and device.

 

Also removed -DWEBP_HAVE_TIFF as you said, and added -static and -static-libgcc to the EXTRA_FLAGS and then added '-Wl' to the CWEBP_LIBS to match what you had originally suggested for LIBS.

TIFF is probably still possible, but you might need things like lzma, jbig.

 

Thank you so much for your help!

Glad it's working for you. Let us know if you have any further issues or questions.

[Shane Bishop]

Any idea how I can replicate this on FreeBSD? When using 'configure', I get similar errors as I had on linux, and when I try using makefile.unix, it just says "Error expanding embedded variable"

I tried using bash, but that didn't change anything...

[Shane Bishop]

On Thursday, March 10, 2016 at 9:20:03 PM UTC-7, Shane Bishop wrote:

Any idea how I can replicate this on FreeBSD? When using 'configure', I get similar errors as I had on linux, and when I try using makefile.unix, it just says "Error expanding embedded variable"

I tried using bash, but that didn't change anything...

I found that gmake takes care of that particular issue, but now it can't find jpeglib.h. I'll dig into that tomorrow, probably just need to change some prefixes here and there. 

[Shane Bishop]

One last post for tonight, added the include dir to EXTRA_FLAGS, and got this far:

 gcc -Os -DNDEBUG -DWEBP_HAVE_PNG -DWEBP_HAVE_JPEG -I/usr/local/include -DWEBP_USE_THREAD -fvisibility=hidden -Wextra -Wold-style-definition -Wmissing-prototypes -Wmissing-declarations -Wdeclaration-after-statement -Wshadow -Wformat-security -Wformat-nonliteral -Isrc/ -Wall -c src/dsp/alpha_processing_sse2.c -o src/dsp/alpha_processing_sse2.o

src/dsp/alpha_processing_sse2.c: In function 'MultARGBRow':

src/dsp/alpha_processing_sse2.c:237: note: use -flax-vector-conversions to permit conversions between vectors with differing element types or numbers of subparts

src/dsp/alpha_processing_sse2.c:237: error: incompatible type for argument 1 of '__builtin_ia32_psrlqi128'

makefile.unix:320: recipe for target 'src/dsp/alpha_processing_sse2.o' failed

gmake: *** [src/dsp/alpha_processing_sse2.o] Error 1

[Shane Bishop]

I tried one last thing last night, and that did the trick. In looking back through my notes, I realized I had issues building before on FreeBSD with the default gcc. So I installed gcc 4.9 and used that along with gmake, and it worked. So for posterity, here is what I had to modify in makefile.unix:

EXTRA_FLAGS= -DWEBP_HAVE_PNG -DWEBP_HAVE_JPEG -I/usr/local/include -static -static-libgcc

CC = gcc49

After that, running ldd against cwebp properly states: "not a dynamic ELF executable"

[James Zern]

On Friday, March 11, 2016 at 5:18:23 AM UTC-8, Shane Bishop wrote:

I tried one last thing last night, and that did the trick. In looking back through my notes, I realized I had issues building before on FreeBSD with the default gcc.

What version of freebsd / gcc out of curiosity? There was an earlier thread with a similar problem, but that seemed to be due to an incomplete upgrade. Using the binary packages for the release worked with an older 4.2 gcc if I'm remembering correctly.

 

So I installed gcc 4.9 and used that along with gmake, and it worked. So for posterity, here is what I had to modify in makefile.unix:

EXTRA_FLAGS= -DWEBP_HAVE_PNG -DWEBP_HAVE_JPEG -I/usr/local/include -static -static-libgcc

CC = gcc49

After that, running ldd against cwebp properly states: "not a dynamic ELF executable"

Thanks for keeping the thread updated so people can follow along.

[Shane Bishop]

On Friday, March 11, 2016 at 1:36:51 PM UTC-7, James Zern wrote:

On Friday, March 11, 2016 at 5:18:23 AM UTC-8, Shane Bishop wrote:

I tried one last thing last night, and that did the trick. In looking back through my notes, I realized I had issues building before on FreeBSD with the default gcc.

What version of freebsd / gcc out of curiosity? There was an earlier thread with a similar problem, but that seemed to be due to an incomplete upgrade. Using the binary packages for the release worked with an older 4.2 gcc if I'm remembering correctly.

It is FreeBSD 9.3 with gcc 4.2.something. I can check the exact version when I get back to my office if you want to know.

[James Zern]

Bad memory on my part. I was referring to the bug you filed a while ago [1]. That fix was in configure, so this kind of custom use of makefile.unix would need its own workaround or a newer compiler like you tried, which is probably better for performance.

[1] https://bugs.chromium.org/p/webp/issues/detail?id=274