[Haskell-cafe] Static executables in minimal Docker containers

[Michael Snoyman]

I'm trying to put together a minimal Docker container consisting of nothing but GHC-compiled static executables. Below you can see a full interaction I've had with GHC and Docker. The high level summary is that:

* When compiled statically, the executable runs just fine in both my host OS (Ubuntu 14.04) and an Ubuntu 14.04 Docker image

* That same executable run from a busybox (or a "scratch" image, not shown here since it's slightly longer to set up) hangs and then runs out of memory

I've watched the process in top, and it uses up a huge amount of CPU and memory. I get the same behavior whether I compiled with or without optimizations or the multithreaded runtime. I also get identical behavior with both GHC 7.8.4 and 7.10.1.

I'm not sure how best to proceed with trying to debug this, any suggestions?

vagrant@vagrant-ubuntu-trusty-64:~/Desktop$ cat > hello.hs <<EOF

main :: IO ()

main = putStrLn "Static Hello"

EOF

vagrant@vagrant-ubuntu-trusty-64:~/Desktop$ ghc -optl-static -optl-pthread hello.hs -static

[1 of 1] Compiling Main             ( hello.hs, hello.o )

Linking hello ...

/opt/ghc/7.8.4/lib/ghc-7.8.4/rts-1.0/libHSrts.a(Linker.o): In function `internal_dlopen':

(.text+0x5c): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

vagrant@vagrant-ubuntu-trusty-64:~/Desktop$ ./hello

Static Hello

vagrant@vagrant-ubuntu-trusty-64:~/Desktop$ docker run --rm -v $(pwd)/hello:/usr/local/bin/hello:ro ubuntu:14.04 /usr/local/bin/hello

Static Hello

vagrant@vagrant-ubuntu-trusty-64:~/Desktop$ docker run --rm -v $(pwd)/hello:/usr/local/bin/hello:ro busybox /usr/local/bin/hello

hello: out of memory (requested 1048576 bytes)

vagrant@vagrant-ubuntu-trusty-64:~/Desktop$ ghc --version

The Glorious Glasgow Haskell Compilation System, version 7.8.4

[Magnus Therning]

On 13 April 2015 at 10:54, Michael Snoyman <mic...@snoyman.com> wrote:
> I'm trying to put together a minimal Docker container consisting of nothing
> but GHC-compiled static executables. Below you can see a full interaction
> I've had with GHC and Docker. The high level summary is that:
>
> * When compiled statically, the executable runs just fine in both my host OS
> (Ubuntu 14.04) and an Ubuntu 14.04 Docker image
> * That same executable run from a busybox (or a "scratch" image, not shown
> here since it's slightly longer to set up) hangs and then runs out of memory

From what I remember busybox allows for quite a bit of
configurability, so "a busybox" might need a bit more details mabye.

Have you attempted running it under `strace` and/or `ltrace`?

/M

--
Magnus Therning OpenPGP: 0xAB4DFBA4
email: mag...@therning.org jabber: mag...@therning.org
twitter: magthe http://therning.org/magnus
_______________________________________________
Haskell-Cafe mailing list
Haskel...@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

[Michael Snoyman]

On Mon, Apr 13, 2015 at 12:07 PM Magnus Therning <mag...@therning.org> wrote:

On 13 April 2015 at 10:54, Michael Snoyman <mic...@snoyman.com> wrote:
> I'm trying to put together a minimal Docker container consisting of nothing
> but GHC-compiled static executables. Below you can see a full interaction
> I've had with GHC and Docker. The high level summary is that:
>
> * When compiled statically, the executable runs just fine in both my host OS
> (Ubuntu 14.04) and an Ubuntu 14.04 Docker image
> * That same executable run from a busybox (or a "scratch" image, not shown
> here since it's slightly longer to set up) hangs and then runs out of memory

From what I remember busybox allows for quite a bit of
configurability, so "a busybox" might need a bit more details mabye.

Have you attempted running it under `strace` and/or `ltrace`?

Sorry, I left off a word: I meant "a busybox image." There's a standard busybox Docker image which I'm testing with via the command I pasted below. There are certainly ways to configure busybox itself, but it should be trivial for others to reproduce this error simply by running my series of commands.

Unfortunately, strace and ltrace aren't available in that Docker image, but it's a good idea to see if I can get them running there somehow.

Michael 

[Greg Weber]

Haskell is not that great at producing statically linked libraries independent of the OS.

The issue you are running into would likely show up in another non-ubuntu image (or even possibly a different version of an ubuntu image), so you could probably use a Fedora image that has tracing.

How are you addressing the linker warning about needing a particular glibc version at runtime?

On Mon, Apr 13, 2015 at 3:28 AM, Sharif Olorin <sharif...@gmail.com> wrote:

Unfortunately, strace and ltrace aren't available in that Docker image, but it's a good idea to see if I can get them running there somehow.

Failing that, you might be able to get useful information of the same kind by running docker (the server, not the `docker run` command) under perf[0] and then running your busybox container. It should at least give you an idea of what it's doing when it explodes.

Sharif

[0]: https://perf.wiki.kernel.org/index.php/Tutorial

--
You received this message because you are subscribed to the Google Groups "Commercial Haskell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to commercialhask...@googlegroups.com.
To post to this group, send email to commerci...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/commercialhaskell/86ca2603-37f2-4645-9cd2-f09703f2be67%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jeremy]

Greg Weber wrote

> Haskell is not that great at producing statically linked libraries
> independent of the OS.
> The issue you are running into would likely show up in another non-ubuntu
> image (or even possibly a different version of an ubuntu image), so you
> could probably use a Fedora image that has tracing.

You could try compiling on Debian and running it in
https://registry.hub.docker.com/u/accursoft/micro-jessie/. Much bigger than
busybox, but 33Mb may be small enough for you.

(Remember that the OS layer will be shared between multiple images - I don't
know if that helps for your scenario.)



--
View this message in context: http://haskell.1045720.n5.nabble.com/Static-executables-in-minimal-Docker-containers-tp5768703p5768730.html
Sent from the Haskell - Haskell-Cafe mailing list archive at Nabble.com.

[Michael Snoyman]

I'm not sure if this issue would show up, but I can try it in Fedora tomorrow. I didn't address the linker warning at all right now, it seems to not have been triggered, though I suppose it is possible that it's the cause of this issue.

[Vo Minh Thu]

I missed this thread but I guess I tried something similar last month:
https://gist.github.com/noteed/4155ffad2b1d13ab17ee

[Aaron Levin]

FWIW: I experienced something similar to this (hanging) with the standard Debian image.

[Albert Y. C. Lai]

I wonder whether you already know the following, and whether it is
relevant to begin with. (Plus, my knowledge is fairly sketchy.)

Even though you statically link glibc, its code will, at run time,
dlopen a certain part of glibc.

Why: To provide a really uniform abstraction layer over user account
queries, e.g., man 3 getpwnam, regardless of whether the accounts are
from /etc/passwd, LDAP, or whatever.

Therefore, during run time, glibc first reads some config files of the
host to see what kind of user account database the host uses. If it's
/etc/passwd, then dlopen the implementation of getpwnam and friends for
/etc/passwd; else, if it's LDAP, then dlopen the implementation of
getpwnam and friends for LDAP; etc etc.

So that later when you call getpwnam, it will happen to "do the right
thing".

This demands the required *.so files to be accessible during run time.
Moreoever, if you statically link glibc, this also demands the required
*.so files to version-match the glibc you statically link.

(It is the main reason why most people give up on statically linking glibc.)

[Michael Snoyman]

I have a bit more information about this. In particular: I'm able to reproduce this using chroot (no Docker required), and it's reproducing with a dynamically linked executable too. Steps I used to reproduce:

1. Write a minimal "foo.hs" containing `main = putStrLn "Hello World"`

2. Compile that executable and put it in an empty directory

3. Run `ldd` on it and copy all necessary libraries inside that directory

4. Run `sudo strace -o log.txt . /foo`

I've uploaded the logs to:

https://gist.github.com/snoyberg/095efb17e36acc1d6360

Note that, due to size of the output, I killed the process just a few seconds after starting it, but when I let the output run much longer, I didn't see any difference in the results. I'll continue poking at this a bit, but most likely I'll open a GHC Trac ticket about it later today.

[Michael Snoyman]

Actually, I seem to have found the problem:

open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

I found that I needed to copy over the following files to make my program complete:

/usr/lib/x86_64-linux-gnu/gconv/gconv-modules
/usr/lib/x86_64-linux-gnu/gconv/UTF-32.so

Once I did that, I could get the executable to run in the chroot. However, even running the statically linked executable still required most of the shared libraries to be present inside the chroot. So it seems that:

* We can come up with a list of a few files that need to be present inside a Docker image to provide for minimal GHC-compiled executables

* There's a bug in the RTS that results in an infinite loop

I'm going to try to put together a semi-robust solution for the first problem, and I'll report the RTS issue on Trac.

[Magnus Therning]

On 14 April 2015 at 08:52, Michael Snoyman <mic...@snoyman.com> wrote:
> Actually, I seem to have found the problem:
>
> open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = -1
> ENOENT (No such file or directory)
> open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules", O_RDONLY|O_CLOEXEC) =
> -1 ENOENT (No such file or directory)
>
> I found that I needed to copy over the following files to make my program
> complete:
>
> /usr/lib/x86_64-linux-gnu/gconv/gconv-modules
> /usr/lib/x86_64-linux-gnu/gconv/UTF-32.so
>
> Once I did that, I could get the executable to run in the chroot. However,
> even running the statically linked executable still required most of the
> shared libraries to be present inside the chroot. So it seems that:
>
> * We can come up with a list of a few files that need to be present inside a
> Docker image to provide for minimal GHC-compiled executables
> * There's a bug in the RTS that results in an infinite loop
>
> I'm going to try to put together a semi-robust solution for the first
> problem, and I'll report the RTS issue on Trac.

Excellent that you found the issue.

Is there some way of controlling which libc ghc links with? There are
quite a few alternatives out there, and maybe it'd be easier to create
a *true* static binary by using another libc?

/M

--
Magnus Therning OpenPGP: 0xAB4DFBA4
email: mag...@therning.org jabber: mag...@therning.org
twitter: magthe http://therning.org/magnus

[Michael Snoyman]

Trac ticket created: https://ghc.haskell.org/trac/ghc/ticket/10298#ticket

I've also put together a Docker image called snoyberg/haskell-scratch (source at https://github.com/snoyberg/haskell-scratch), which seems to be working for me. Here's a minimal test I've put together which seems to be succeeding (note that I've also tried some real life programs):

#!/bin/bash

set -e
set -x

cat > tiny.hs <<EOF
main :: IO ()
main = putStrLn "Hello from a tiny Docker image"
EOF

ghc tiny.hs
strip tiny

cat > Dockerfile <<EOF
FROM snoyberg/haskell-scratch
ADD tiny /tiny
CMD ["/tiny"]
EOF

docker build -t tiny .
docker run --rm tiny

[Jon Schneider]

These ring a bell. I had pain with exactly this shared library (or lack
of) when playing with the output of cross-compiling ghc.

> /usr/lib/x86_64-linux-gnu/gconv/UTF-32.so

Jon

[Vo Minh Thu]

Michael, thanks for solving this !

I wonder how you came up with the need for /lib/ld-linux-x86-64.so.2.
I have needed only /lib64/ld-linux-x86-64.so.2 (which you have too),
so maybe you added it by mistake.

B.t.w, if you don't want to use a Dockerfile, you can do this:

> tar -C rootfs -c . | docker import - tiny

where rootfs is your repository.

[Michael Snoyman]

The /lib/ version of the file was necessary for statically linked executables I believe. It's arguably better to just remove it and tell people "don't do static executables" I suppose, given that they're clearly not *actually* static in reality ;)

[Simon Marlow]

Hi Michael,

This rang a bell for me. It might be the same as these:
https://ghc.haskell.org/trac/ghc/ticket/7695
https://ghc.haskell.org/trac/ghc/ticket/8928

I think the conclusion was that the IO library is failing to start
iconv, and printing the error messages causes it to retry loading iconv,
ad infinitum (or something like that). There's no fix yet, but it
probably isn't hard to fix, just that nobody got around to it yet.

Cheers,
Simon

> <mailto:commercialhask...@googlegroups.com>.

> To post to this group, send email to
> commerci...@googlegroups.com

> <mailto:commerci...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/commercialhaskell/86ca2603-37f2-4645-9cd2-f09703f2be67%40googlegroups.com

> <https://groups.google.com/d/msgid/commercialhaskell/86ca2603-37f2-4645-9cd2-f09703f2be67%40googlegroups.com?utm_medium=email&utm_source=footer>.

>
> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google
> Groups "Commercial Haskell" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to commercialhask...@googlegroups.com

> <mailto:commercialhask...@googlegroups.com>.

> To post to this group, send email to commerci...@googlegroups.com

> <mailto:commerci...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/commercialhaskell/CAKA2Jg%2B%3DzJiXmak2FU_5GWPO1Dcn%2BvwsiB_xWj%2B8GfHvMkoBjw%40mail.gmail.com
> <https://groups.google.com/d/msgid/commercialhaskell/CAKA2Jg%2B%3DzJiXmak2FU_5GWPO1Dcn%2BvwsiB_xWj%2B8GfHvMkoBjw%40mail.gmail.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.

[Michael Snoyman]

Thanks for the update Simon.

To unsubscribe from this group and stop receiving emails from it, send an email to commercialhask...@googlegroups.com.
To post to this group, send email to commerci...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/commercialhaskell/553407EF.1030303%40gmail.com.

[Sylvain Henry]

Hi,

I had the same bug with a single static binary in a initramfs. I fixed GHC.IO.Encoding to avoid using Iconv for the default (ASCII) charmap used with static binaries and now it works. Patch is in my comment on https://ghc.haskell.org/trac/ghc/ticket/10298

If the default LC_CTYPE value is the DEFAULT_CHARMAP from glibc/locale/programs/config.h, it is set to "ANSI_X3.4-1968" since 2000 and was set to "POSIX" before. Maybe we should match this one too?

Sylvain