glibc, asan and libfuzzer

[Hanno Böck]

Hi,

I thought given the current issues with glibc I'd bring that up.

A while ago I had a conversation with Kostya about building glibc with
asan. I think it can be summed up as "it's possible, but requires lots
of manual work and is complicated".

The publicly available documentation is currently a wiki page listing
problems trying to build glibc with clang
https://sourceware.org/glibc/wiki/GlibcMeetsClang
and some reports about fuzzing done with libfuzzer
https://sourceware.org/glibc/wiki/FuzzingLibc

As far as I can see there is currently no public documentation how one
would compile glibc with asan (and/or libfuzzer).

I think it is a major drawback of security analysis of glibc that many
common tools don't work on it and it'd be great if this area could be
improved. So the question is: How realistic would it be to make this
stuff more easily accessible?

--
Hanno Böck
https://hboeck.de/

mail/jabber: ha...@hboeck.de
GPG: BBB51E42

[Konstantin Serebryany]

+Roland

The only good solution is to have the upstream glibc fixed and maintained in this state.

(We need to make it build with clang+asan and have the bots that verify it still works on every commit).

Roland wanted to try doing that; not sure what's the current state. 

Anyway, I think this should be discussed at libc-...@sourceware.org

On Wed, Feb 17, 2016 at 3:21 AM, Hanno Böck <ha...@hboeck.de> wrote:

Hi,

I thought given the current issues with glibc I'd bring that up.

A while ago I had a conversation with Kostya about building glibc with
asan. I think it can be summed up as "it's possible, but requires lots
of manual work and is complicated".

The publicly available documentation is currently a wiki page listing
problems trying to build glibc with clang
https://sourceware.org/glibc/wiki/GlibcMeetsClang
and some reports about fuzzing done with libfuzzer
https://sourceware.org/glibc/wiki/FuzzingLibc

As far as I can see there is currently no public documentation how one
would compile glibc with asan (and/or libfuzzer).

True. My current instructions are pretty involved. 

Let me dump them here FTR (last checked 1 month ago). 

First, build glibc in a usual way: 

wget http://ftp.gnu.org/gnu/glibc/glibc-2.22.tar.bz2

tar xf glibc-2.22.tar.bz2

(

mkdir glibc_build_plain

cd glibc_build_plain/

../glibc-2.22/configure --prefix=`pwd`/../glibc_inst_plain && make -j && make install

)

Grab fresh clang.

Revert clang r255371, or apply this patch. Then rebuild clang.

(This is a long and sad story... Hopefully Roland will fix it)

--- llvm/tools/clang/lib/Sema/SemaDecl.cpp   (revision 257672)

+++ llvm/tools/clang/lib/Sema/SemaDecl.cpp   (working copy)

@@ -2381,7 +2381,7 @@

   // Attributes declared post-definition are currently ignored.

   checkNewAttributesAfterDef(*this, New, Old);

 

-  if (AsmLabelAttr *NewA = New->getAttr<AsmLabelAttr>()) {

+  if (0)  if (AsmLabelAttr *NewA = New->getAttr<AsmLabelAttr>()) {

     if (AsmLabelAttr *OldA = Old->getAttr<AsmLabelAttr>()) {

       if (OldA->getLabel() != NewA->getLabel()) {

         // This redeclaration changes __asm__ label.

Now, download the attached clang-gcc-wrapper.py and put it into ./

Build using clang-gcc-wrapper.py instead of gcc:

(

mkdir glibc_build_clang # name is important for clang-gcc-wrapper.py

cd glibc_build_clang

CC=`pwd`/../clang-gcc-wrapper.py ../glibc-2.22/configure --prefix=`pwd`/../glibc_inst_clang

make -k  # -j

)

The build will fail to complete (thus -k), but it will produce all needed .so

files.

Now, copy .so files to glibc_build_plain/:

for f in librt libdl libresolv libpthread libcrypt libm; do cp -v glibc_build_clang/*/$f.so glibc_inst_plain/lib/$f-2.22.so; done

cp glibc_build_clang/libc.so glibc_inst_plain/lib/libc-2.22.so

Note: if you are building version other than 22, change the names.

Now, verify that you have proper instrumentation.

% cat use-after-free-on-gethostbyname.c

#include <netdb.h>

#include <stdlib.h>

int main() {

  char *x = (char*)malloc(10);

  free(x);

  gethostbyname(x);

}

% export SYSROOT=`pwd`/glibc_inst_plain/

% clang -g -fsanitize=address use-after-free-on-gethostbyname.c \

 -Wl,-rpath=$SYSROOT/lib -Wl,-dynamic-linker=$SYSROOT/lib/ld-2.22.so

% ASAN_OPTIONS=replace_str=0 ./a.out

You should get 

==25426==ERROR: AddressSanitizer: heap-use-after-free on address 0x60200000eff0

at pc 0x7f65db399e2d bp 0x7ffe902a0c50 sp 0x7ffe902a0c48

READ of size 1 at 0x60200000eff0 thread T0

    #0 0x7f65db399e2c in __GI___libc_res_nsearch  glibc-2.22/resolv/res_query.c:356:18

NOTE: you should get the first frame inside the glibc code, not inside asan

interceptor. Compare to running w/o ASAN_OPTIONS=replace_str=0:

#0 0x49a490 in index llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:470:5

Finally, have a look at clang-gcc-wrapper.py to make sure you instrument all the

code you need:

asan_whitelist = [#"posix",

                  "string", "wcsmbs", "wctype", "stdio-common", "time", "resolv"]

Some parts of glibc still don't build with clang, so we are using a whitelist. 

 

--kcc

 

I think it is a major drawback of security analysis of glibc that many
common tools don't work on it and it'd be great if this area could be
improved. So the question is: How realistic would it be to make this
stuff more easily accessible?

--
Hanno Böck
https://hboeck.de/

mail/jabber: ha...@hboeck.de
GPG: BBB51E42

--
You received this message because you are subscribed to the Google Groups "address-sanitizer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to address-saniti...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Yuri Gribov]

On Wed, Feb 17, 2016 at 5:45 PM, Konstantin Serebryany
<konstantin....@gmail.com> wrote:
> +Roland
>
> The only good solution is to have the upstream glibc fixed and maintained in
> this state.
> (We need to make it build with clang+asan and have the bots that verify it
> still works on every commit).

Why not sanitize it with GCC though?

[Konstantin Serebryany]

On Wed, Feb 17, 2016 at 11:50 AM, Yuri Gribov <tetr...@gmail.com> wrote:

On Wed, Feb 17, 2016 at 5:45 PM, Konstantin Serebryany
<konstantin....@gmail.com> wrote:
> +Roland
>
> The only good solution is to have the upstream glibc fixed and maintained in
> this state.
> (We need to make it build with clang+asan and have the bots that verify it
> still works on every commit).

Why not sanitize it with GCC though?

You can, it's not much simpler. I've tried. 

You will avoid the clang-specific problems, but you still can not build the entire thing with asan. 

But if you want to fuzz glibc, you'll need the coverage instrumentation. 

GCC has the one that we use in the kernel, but not the one we use with libFuzzer. 

[Yuri Gribov]

On Wed, Feb 17, 2016 at 10:52 PM, Konstantin Serebryany

<konstantin....@gmail.com> wrote:
>
>
> On Wed, Feb 17, 2016 at 11:50 AM, Yuri Gribov <tetr...@gmail.com> wrote:
>>
>> On Wed, Feb 17, 2016 at 5:45 PM, Konstantin Serebryany
>> <konstantin....@gmail.com> wrote:
>> > +Roland
>> >
>> > The only good solution is to have the upstream glibc fixed and
>> > maintained in
>> > this state.
>> > (We need to make it build with clang+asan and have the bots that verify
>> > it
>> > still works on every commit).
>>
>> Why not sanitize it with GCC though?
>
>
> You can, it's not much simpler. I've tried.
> You will avoid the clang-specific problems, but you still can not build the
> entire thing with asan.
> But if you want to fuzz glibc, you'll need the coverage instrumentation.
> GCC has the one that we use in the kernel, but not the one we use with
> libFuzzer.

I guess it would be much easier to port coverage to GCC than to build
Glibc and convince maintainers to take Clang-specific patches ;)

[Konstantin Serebryany]

On Wed, Feb 17, 2016 at 12:21 PM, Yuri Gribov <tetr...@gmail.com> wrote:

On Wed, Feb 17, 2016 at 10:52 PM, Konstantin Serebryany
<konstantin....@gmail.com> wrote:
>
>
> On Wed, Feb 17, 2016 at 11:50 AM, Yuri Gribov <tetr...@gmail.com> wrote:
>>
>> On Wed, Feb 17, 2016 at 5:45 PM, Konstantin Serebryany
>> <konstantin....@gmail.com> wrote:
>> > +Roland
>> >
>> > The only good solution is to have the upstream glibc fixed and
>> > maintained in
>> > this state.
>> > (We need to make it build with clang+asan and have the bots that verify
>> > it
>> > still works on every commit).
>>
>> Why not sanitize it with GCC though?
>
>
> You can, it's not much simpler. I've tried.
> You will avoid the clang-specific problems, but you still can not build the
> entire thing with asan.
> But if you want to fuzz glibc, you'll need the coverage instrumentation.
> GCC has the one that we use in the kernel, but not the one we use with
> libFuzzer.

I guess it would be much easier to port coverage to GCC than to build
Glibc and convince maintainers to take Clang-specific patches ;)

The glibc maintainers are already convinced to port glibc to clang, 

and most of the work is done. (But not all, sadly). 

[Maxim Ostapenko]

FYI, this bug can be found by patched AddressSanitizer:

==18133==ERROR: AddressSanitizer: dynamic-stack-buffer-overflow on address 0x7ffc15acc820 at pc 0x00000048fea6 bp 0x7ffc15aca950 sp 0x7ffc15aca0f0

info: UDP server 1: sending response: 2076 bytes, RCODE 0 (for www123.example/1/28)

READ of size 65536 at 0x7ffc15acc820 thread T0

    #0 0x48fea5 in recvfrom /home/max/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:223:3

    #1 0x7febc202028d in send_dg /home/max/src/CVE-2015-7547/glibc-2.22/resolv/res_send.c:1265:19

    #2 0x7febc202028d in __libc_res_nsend /home/max/src/CVE-2015-7547/glibc-2.22/resolv/res_send.c:527

    #3 0x7febc2015c95 in __GI___libc_res_nquery /home/max/src/CVE-2015-7547/glibc-2.22/resolv/res_query.c:227:6

    #4 0x7febc2017a1b in __libc_res_nquerydomain /home/max/src/CVE-2015-7547/glibc-2.22/resolv/res_query.c:594:10

    #5 0x7febc2017a1b in __GI___libc_res_nsearch /home/max/src/CVE-2015-7547/glibc-2.22/resolv/res_query.c:381

    #6 0x7fea334eddd8 in _nss_dns_gethostbyname4_r /home/max/src/CVE-2015-7547/glibc-2.22/resolv/nss_dns/dns-host.c:316:11

    #7 0x7febc139e51a in gaih_inet /home/max/src/CVE-2015-7547/glibc-2.22/posix/../sysdeps/posix/getaddrinfo.c:862

    #8 0x7febc13a06ec in __GI_getaddrinfo /home/max/src/CVE-2015-7547/glibc-2.22/posix/../sysdeps/posix/getaddrinfo.c:2417

    #9 0x43bf6c in __interceptor_getaddrinfo /home/max/src/llvm/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:1967:13

    #10 0x4d860c in test_different_sizes /home/max/src/CVE-2015-7547/bug18665.c:114:9

    #11 0x4d7dd4 in main /home/max/src/CVE-2015-7547/bug18665.c:139:3

    #12 0x7febc11ed514 in __libc_start_main /home/max/src/CVE-2015-7547/glibc-2.22/csu/libc-start.c:289

    #13 0x418fa5 in _start (/home/max/src/CVE-2015-7547/bug18665+0x418fa5)

Address 0x7ffc15acc820 is located in stack of thread T0 at offset 4768 in frame

    #0 0x7febc201733f in __GI___libc_res_nsearch /home/max/src/CVE-2015-7547/glibc-2.22/resolv/res_query.c:342

  This frame has 2 object(s):

    [32, 1057) 'nbuf.i'

    [1200, 2225) 'tmp' <== Memory access at offset 4768 overflows this variable

HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext

      (longjmp and C++ exceptions *are* supported)

SUMMARY: AddressSanitizer: dynamic-stack-buffer-overflow /home/max/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:223:3 in recvfrom

Shadow bytes around the buggy address:

  0x100002b518b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

  0x100002b518c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

  0x100002b518d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

  0x100002b518e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

  0x100002b518f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

=>0x100002b51900: 00 00 00 00[cb]cb cb cb f1 f1 f1 f1 00 f2 f2 f2

  0x100002b51910: 00 f2 f2 f2 00 f2 f2 f2 04 f2 00 f2 f2 f2 00 f2

  0x100002b51920: f2 f2 04 f2 04 f2 04 f3 00 00 00 00 00 00 00 00

  0x100002b51930: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

  0x100002b51940: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

  0x100002b51950: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Shadow byte legend (one shadow byte represents 8 application bytes):

  Addressable:           00

  Partially addressable: 01 02 03 04 05 06 07 

  Heap left redzone:       fa

  Heap right redzone:      fb

  Freed heap region:       fd

  Stack left redzone:      f1

  Stack mid redzone:       f2

  Stack right redzone:     f3

  Stack partial redzone:   f4

  Stack after return:      f5

  Stack use after scope:   f8

  Global redzone:          f9

  Global init order:       f6

  Poisoned by user:        f7

  Container overflow:      fc

  Array cookie:            ac

  Intra object redzone:    bb

  ASan internal:           fe

  Left alloca redzone:     ca

  Right alloca redzone:    cb

==18133==ABORTING

diff --git a/lib/asan/asan_interceptors.cc b/lib/asan/asan_interceptors.cc

index faac15b..d41a665 100644

--- a/lib/asan/asan_interceptors.cc

+++ b/lib/asan/asan_interceptors.cc

@@ -214,6 +214,17 @@ DECLARE_REAL_AND_INTERCEPTOR(void, free, void *)

   } while (false)

 #include "sanitizer_common/sanitizer_common_syscalls.inc"

 

+

+INTERCEPTOR(SSIZE_T, recvfrom, int fd, void *buf, SIZE_T len, int flags,

+            void *srcaddr, int *addrlen) {

+  ENSURE_ASAN_INITED();

+  SIZE_T srcaddr_sz;

+  if (srcaddr) srcaddr_sz = *addrlen;

+  ASAN_READ_RANGE(nullptr, buf, len);

+  SSIZE_T res = REAL(recvfrom)(fd, buf, len, flags, srcaddr, addrlen);

+  return res;

+}

+

 struct ThreadStartParam {

   atomic_uintptr_t t;

   atomic_uintptr_t is_registered;

@@ -759,6 +770,8 @@ void InitializeAsanInterceptors() {

     ASAN_INTERCEPT_FUNC(memcpy);

   }

 

+  INTERCEPT_FUNCTION(recvfrom);

+

   // Intercept str* functions.

   ASAN_INTERCEPT_FUNC(strcat);  // NOLINT

   ASAN_INTERCEPT_FUNC(strchr);

Kostya, perhaps it makes sense to add recvfrom interceptor to ASan? MSan has it.

 

[Maxim Ostapenko]

Alloca instrumentation is really cool :)

пятница, 19 февраля 2016 г., 21:31:46 UTC+3 пользователь Maxim Ostapenko написал:

[Maxim Ostapenko]

Oh,I mean CVE-2015-7547 of course.

[Konstantin Serebryany]

On Fri, Feb 19, 2016 at 10:28 AM, Maxim Ostapenko <chef...@gmail.com> wrote:

FYI, this bug can be found by patched AddressSanitizer:

By patched, you mean the recvfrom interceptor below, right? 

Would you like to contribute it? 

 

--

[Maxim Ostapenko]

Of course. Will post the patch shortly.

[Konstantin Serebryany]

Thanks! 

On Fri, Feb 19, 2016 at 12:21 PM, Maxim Ostapenko <chef...@gmail.com> wrote:

Of course. Will post the patch shortly.