Bug#796899: Acknowledgement (interesting segfault)

[Joey Hess]

reassign 796899 libc6
found 796899 2.19-19
thanks

This also happens with curl, not just ssh, so reassigning to libc6.

/lib64/ld-linux-x86-64.so.2 /usr/bin/curl
Segmentation fault

Since curl 7.44.0-1 works when run via ld.so, and curl 7.43.0-1
segfaults, I think this might have to do with the ongoing gcc
transition.

--
see shy jo

[Joey Hess]

Package: openssh-server
Version: 1:6.9p1-1
Severity: normal

/lib64/ld-linux-x86-64.so.2 /usr/bin/ssh
Segmentation fault

This is puzzling, AFAICS it should be the same as running ssh
directly. Happens on amd64, not on i386, and didn't used to happen
before probably version 1:6.9p1-1.

-- System Information:
Debian Release: stretch/sid
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.16.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.utf8, LC_CTYPE=en_US.utf8 (charmap=locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
ANSI_X3.4-1968)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages openssh-server depends on:
ii adduser 3.113+nmu3
ii debconf [debconf-2.0] 1.5.57
ii dpkg 1.18.1
ii init-system-helpers 1.23
ii libc6 2.19-19
ii libcomerr2 1.42.13-1
ii libgssapi-krb5-2 1.13.2+dfsg-2
ii libkrb5-3 1.13.2+dfsg-2
ii libpam-modules 1.1.8-3.1
ii libpam-runtime 1.1.8-3.1
ii libpam0g 1.1.8-3.1
ii libselinux1 2.3-2+b1
ii libssl1.0.0 1.0.2d-1
ii libwrap0 7.6.q-25
ii lsb-base 4.1+Debian13+nmu1
ii openssh-client 1:6.9p1-1
ii openssh-sftp-server 1:6.9p1-1
ii procps 2:3.3.10-2
ii zlib1g 1:1.2.8.dfsg-2+b1

Versions of packages openssh-server recommends:
ii ncurses-term 5.9+20150516-2
ii xauth 1:1.0.9-1

Versions of packages openssh-server suggests:
pn molly-guard <none>
pn monkeysphere <none>
pn rssh <none>
ii ssh-askpass 1:1.2.4.1-9
pn ufw <none>

-- debconf information excluded

--
see shy jo

[Joey Hess]

Colin Watson wrote:
> Here's LD_DEBUG=all output, which suggests it might relate to NSS.

> 22014: symbol=fclose; lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
> 22014: binding file /lib/x86_64-linux-gnu/libnss_compat.so.2 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `fclose' [GLIBC_2.2.5]

strace shows curl gets as far as reading ~/.curlrc before crashing, while
ssh seems to start running and reads /etc/passwd before crashing.

gdb shows ssh and curl crashing in fwrite and fputc, respectively.

Starting program: /lib64/ld-linux-x86-64.so.2 /usr/bin/ssh
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
__GI__IO_fwrite (buf=0x7ffff7db4a00, size=1, count=525, fp=0x0)
at iofwrite.c:41
41 iofwrite.c: No such file or directory.

Starting program: /lib64/ld-linux-x86-64.so.2 /usr/bin/curl
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
fputc (c=99, fp=0x0) at fputc.c:37
37 fputc.c: No such file or directory.

--
see shy jo

[Aurelien Jarno]

The fp pointer is NULL in both of the above functions. Could you please
try to get a backtrace to see which caller starts to pass a NULL
pointer?

--
Aurelien Jarno GPG: 4096R/1DDD8C9B
aure...@aurel32.net http://www.aurel32.net

[Joey Hess]

Aurelien Jarno wrote:
> The fp pointer is NULL in both of the above functions. Could you please
> try to get a backtrace to see which caller starts to pass a NULL
> pointer?

Tried building curl from source to get a useful backtrace, but that
build didn't have the problem.

Since that build was done using gcc 4.9.2-4, it may be another hint in
the direction of the recent gcc transitions.

--
see shy jo

[Joey Hess]

Joey Hess wrote:
> Tried building curl from source to get a useful backtrace, but that
> build didn't have the problem.
>
> Since that build was done using gcc 4.9.2-4, it may be another hint in
> the direction of the recent gcc transitions.

Indeed, I built curl with gcc 5.2.1-4 and it has the problem.
So, the gcc upgrade led to this problem.

Here is the backtrace:

joey@kite:~/tmp/curl-7.44.0/debian>LD_LIBRARY_PATH=./build/lib/.libs/ gdb /lib64/ld-linux-x86-64.so.2
GNU gdb (Debian 7.10-1) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.19.so...done.
done.
(gdb) run ./build/src/.libs/curl
Starting program: /lib64/ld-linux-x86-64.so.2 ./build/src/.libs/curl

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
fputc (c=99, fp=0x0) at fputc.c:37
37 fputc.c: No such file or directory.

(gdb) bt
#0 fputc (c=99, fp=0x0) at fputc.c:37
#1 0x00007ffff7b7cd9f in dprintf_formatf (data=<optimized out>,
stream=0x7ffff73d7470 <fputc>, format=<optimized out>,
ap_save=ap_save@entry=0x7fffffffe060) at mprintf.c:616
#2 0x00007ffff7b7e58e in curl_mfprintf (whereto=<optimized out>,
format=<optimized out>) at mprintf.c:1121
#3 0x00007ffff7ddae51 in ?? ()
#4 0x0000000000000000 in ?? ()

The code here is quite horrible, kind of looks like the FILE * has
been somehow optimised out, perhaps wrongly?

static int dprintf_formatf(
void *data, /* untouched by format(), just sent to the stream() function in
the second argument */
/* function pointer called for each output character */
int (*stream)(int, FILE *),

OUTCHAR(*f);

#define OUTCHAR(x) \
do{ \
if(stream((unsigned char)(x), (FILE *)data) != -1) \
done++; \
else \
return done; /* return immediately on failure */ \
} WHILE_FALSE

--
see shy jo

[Joey Hess]

By comparing stack traces under ld-linux.so and not, I was able to determine
that the NULL is coming from global->errors, which is supposed to get
initialized to STDERR but somehow isn't when ld-linux.so runs curl.

While playing with that, I noticed that trying to printf the address of global
causes a segfault, too. Here's a minimal test case for that, which
replaces src/tool_main.c in curl's source tree.

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[])
{
FILE *global=0;

printf("STARTED\n");
printf("GLOBAL %p\n", global);
}

joey@kite:~/tmp/curl-7.44.0/debian/build>./src/.libs/curl
STARTED
GLOBAL (nil)
joey@kite:~/tmp/curl-7.44.0/debian/build>/lib64/ld-linux-x86-64.so.2 ./src/.libs/curl
STARTED
Segmentation fault

(Building this same code manually, not in curl's source tree, I have not been
able to reproduce the problem. Something about how it's linked as part of
curl is contributing.)

Here's an even more minimal and strange test case!

joey@kite:~/tmp/curl-7.44.0/debian/build>cat src/tool_main.c
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[])
{
fprintf(stdout, "HELLO\n");
}
joey@kite:~/tmp/curl-7.44.0/debian/build>./src/.libs/curl
HELLO
joey@kite:~/tmp/curl-7.44.0/debian/build>/lib64/ld-linux-x86-64.so.2 ./src/.libs/curl
Segmentation fault

--
see shy jo

[Joey Hess]

> int main(int argc, char *argv[])
> {
> fprintf(stdout, "HELLO\n");
> }

Even fdopen(1, 'w') crashes the same way. Maybe whatever initialization
is needed for the stream functions to work isn't happening under
ld-linux.so.

--
see shy jo

[Joey Hess]

Joey Hess wrote:
> Even fdopen(1, 'w') crashes the same way.

Er, ignore that, it's obviously wrong.

But, stdout stderr etc are indeed looking very wrong..

joey@kite:~/tmp/curl-7.44.0/debian/build>cat src/tool_main.c
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[])
{

fprintf(stdout, "hello\n");
}
joey@kite:~/tmp/curl-7.44.0/debian/build>gdb /lib64/ld-linux-x86-64.so.2
(gdb) b __GI__IO_fwrite
Breakpoint 1 at 0x7ffff73d0a30: file iofwrite.c, line 35.
(gdb) r ./src/.libs/curl
Starting program: /lib64/ld-linux-x86-64.so.2 ./src/.libs/curl

[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, __GI__IO_fwrite (buf=0x7ffff7df362b, size=1, count=6,
fp=0x0) at iofwrite.c:35
^^^ stdout

--
see shy jo

[Aurelien Jarno]

control: forwarded -1 https://sourceware.org/bugzilla/show_bug.cgi?id=16381
control: tag -1 + fixed-upstream
control: fixed -1 2.21-0experimental0

On 2015-09-11 01:30, Joey Hess wrote:
> Fully minimal test case is in the attached shell script.
>
> This bug only occurs if the binary is linked with -pie.

Thanks a lot for this minimal test case, I have been able to reproduce
the bug and find the corresponding upstream bug. It's BZ#16381 and it
is fixed in upstream commit 798212a0.

It is fixed with the glibc version in experimental, I am therefore
marking this in the BTS. We'll fix this bug in the next glibc upload.
However so far it is unclear if the next upload to unstable will be from
the 2.19 branch or the 2.21 branch.