Static linking and thread-safe allocation

[Peter Waller]

I would like to make cgo calls to an API which expects malloc to be thread safe.

I get the impression that the allocator I get from libc if I do
nothing special is not thread safe. It crashes with double-free when I
try to use C++ functions (via a C interface) which are otherwise
thread-safe (but try to allocate/free).

What is the easiest way to achieve this?

Here is what I have so far, but I'm happy to discard it all if there
is a better way.

https://gist.github.com/pwaller/b805ddc57de281bc1659

(In summary, a Dockerfile which builds a binary with `-tags netgo` and
`-ldflags '-linkmode external -extldflags "-static -Wl,-v
-Wl,--eh-frame-hdr"'`, a main.go which uses cgo to try and do an alloc
and free)

I'm trying to use jemalloc (I picked an allocator at random which
looked easy to build). It crashes on program start with a stack trace
I cannot decipher, reproduced below.

Any advice?

Thanks,

- Peter

Program received signal SIGSEGV, Segmentation fault.
0x000000000051065c in get_nprocs ()
(gdb) bt
#0 0x000000000051065c in get_nprocs ()
#1 0x000000000050d7b5 in sysconf ()
#2 0x000000000049d9b6 in malloc_ncpus () at ../src/jemalloc.c:256
#3 malloc_init_hard () at ../src/jemalloc.c:776
#4 0x000000000049e32d in malloc_init () at ../src/jemalloc.c:292
#5 je_malloc (size=<optimised out>) at ../src/jemalloc.c:929
#6 0x0000000000541753 in _dl_get_origin ()
#7 0x0000000000511f0f in _dl_non_dynamic_init ()
#8 0x0000000000512e08 in __libc_init_first ()
#9 0x00000000004d5d82 in __libc_start_main ()
#10 0x00000000004019a7 in _start ()

[Kevin Malachowski]

From knowing nothing about your problem except from what I learned from looking at the stacktrace it would appear that your program failed to get the number of processors on your machine once it left the jemalloc library. Maybe trying printf'ing information at ../src/jemalloc.c:256?

Also, I just checked using this simple program and I can't get it to crash by using the built-in malloc/free. Can you try running it to see if it fails for you in the same way?

What system are you on? Did you check the docs for your malloc and find that it isn't threadsafe? I've personally used malloc with pthreads, so I know at least that works for a normal gcc/unix-y setup.

// main.go

package main

import (

        "fmt"

        "./cgo"

        "runtime"

        "sync"

)

func main() {

        ch := make(chan []byte, 1000)

        var waiter sync.WaitGroup

        waiter.Add(100)

        for i := 0; i < 100; i++ {

                go allocater(waiter.Done, ch)

        }

        go func() {

                waiter.Wait()

                close(ch)

        }()

        for m := range ch {

                m[2] = 'c'

                fmt.Println(string(m[:3]))

                cgo.Free(m)

        }

}

func allocater(done func(), dst chan<-[]byte) {

        defer done()

        // force a unique thread

        runtime.LockOSThread()

        for i := 0; i < 10; i++ {

                m := cgo.Malloc(1024)

                m[0] = 'a'

                m[1] = 'b'

                dst<-m

        }

}

// cgo/cgo.go

package cgo

// #include <stdlib.h>

import "C"

import (

        "reflect"

        "unsafe"

)

func Malloc(sz int) []byte {

        cbuf := C.malloc(C.size_t(sz))

        head := reflect.SliceHeader{uintptr(cbuf),sz,sz}

        return *(*[]byte)(unsafe.Pointer(&head))

}

func Free(data []byte) {

        C.free(unsafe.Pointer(&data[0]))

[Peter Waller]

Thanks very much for your reply.

I must have misattributed this crash. I ran your example in my
environment and ensured that I was calling the same allocator used by
the C++ code and the example works. Thanks for helping me rule this
out.

Now I will have to try and figure out what the real problem is. I hope
valgrind works.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Kevin Malachowski]

Not a problem. cgo takes a bit of getting used to but it generally works out alright once you learn about the sharp edges. Let me know if you want any more pointers.

[Peter Waller]

I cracked my problem. The library I was calling isn't remotely
thread-safe. I was just mistaken.

I was under the impression gdb didn't work on go binaries, but it
seems it works fine on cgo binaries. So I just ran under gdb and saw
much more detail about the crashes, seeing deeper into the stack than
the "entersyscall()" in the runtime.

Thanks again for the pointers.

[Kevin Malachowski]

Not a problem. As a general rule I assume all C libraries are single-threaded unless it specifically says so. I usually have one goroutine per C library (with the thread locked) running some sort of loop that serializes access to the library somehow (using channels, using a callback provided during init, etc).