[ANN] lz4 - LZ4 compression/decompression in pure go

568 views
Skip to first unread message

pierre...@gmail.com

unread,
Mar 14, 2015, 1:39:20 PM3/14/15
to golan...@googlegroups.com
Hello gophers,


I am please to announce the LZ4 package (github.com/pierrec/lz4) implementing the LZ4 compression/decompression algorithm in 100% Go.

The API is similar to the ones in the standard compress packages but it is such that concurrency is supported.

Any feedback most welcomed,


Pierre

Aliaksandr Valialkin

unread,
Mar 14, 2015, 4:51:32 PM3/14/15
to golan...@googlegroups.com, pierre...@gmail.com
Hi Pierre,

I tried substituting compress/flate with your library - see https://github.com/valyala/gorpc/tree/pierrec-lz4 .

Good news: all tests passed.

Bad news: benchmarks with enabled compression became much slower.

compress/flate:

GOMAXPROCS=4 go test -test.bench="RealApp1000"
BenchmarkRealApp1000Workers-4   50000     32988 ns/op
BenchmarkRealApp10000Workers-4   30000     36921 ns/op
BenchmarkRealApp100000Workers-4   50000     40005 ns/op

pierrec/lz4:

GOMAXPROCS=4 go test -test.bench="RealApp1000"
PASS
BenchmarkRealApp1000Workers-4   10000    207998 ns/op
BenchmarkRealApp10000Workers-4    5000    223354 ns/op
BenchmarkRealApp100000Workers-4   10000    181333 ns/op

Can you look into these performance problems? As I understand, optimized lz4 implementation should be much faster comparing to compress/flate .

pierre...@gmail.com

unread,
Mar 15, 2015, 4:34:48 AM3/15/15
to golan...@googlegroups.com, pierre...@gmail.com
Hi Aliaksandr,


That's bad results :(.

I will look into it but I am running into a nil pointer dereference when trying to run tests:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x1 pc=0x80cb35c]

goroutine 22 [running]:
sync/atomic.AddUint64(0x18601014, 0x1, 0x0, 0x1, 0x83bba20)
/usr/local/go/src/sync/atomic/asm_386.s:118 +0xc
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:178 +0x2ae
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:136 +0x404

goroutine 1 [chan receive]:
testing.RunTests(0x82fbb18, 0x83ba0a0, 0x19, 0x19, 0x18634301)
/usr/local/go/src/testing/testing.go:556 +0x89f
testing.(*M).Run(0x18634000, 0x83c19e0)
/usr/local/go/src/testing/testing.go:485 +0x5e
main.main()

goroutine 21 [sleep]:
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/rpc_test.go:69 +0x1fe
testing.tRunner(0x18642180, 0x83ba0b8)
/usr/local/go/src/testing/testing.go:447 +0xb1
created by testing.RunTests
/usr/local/go/src/testing/testing.go:555 +0x863

goroutine 12 [sleep]:
github.com/valyala/gorpc.func·006()
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:168 +0x1d0
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:171 +0x1b8

goroutine 9 [sleep]:
github.com/valyala/gorpc.func·006()
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:168 +0x1d0
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:171 +0x1b8

goroutine 14 [sleep]:
github.com/valyala/gorpc.func·006()
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:168 +0x1d0
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:171 +0x1b8

goroutine 16 [sleep]:
github.com/valyala/gorpc.func·006()
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:168 +0x1d0
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:171 +0x1b8

goroutine 18 [sleep]:
github.com/valyala/gorpc.func·006()
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:168 +0x1d0
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:171 +0x1b8

goroutine 20 [sleep]:
github.com/valyala/gorpc.func·006()
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:168 +0x1d0
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:171 +0x1b8

goroutine 23 [select]:
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:321 +0x2b8
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:136 +0x1c7

goroutine 24 [select]:
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:321 +0x2b8
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:136 +0x1c7

goroutine 25 [select]:
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:321 +0x2b8
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:136 +0x1c7

goroutine 27 [IO wait]:
net.(*pollDesc).Wait(0x18648278, 0x77, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x42
net.(*pollDesc).WaitWrite(0x18648278, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:93 +0x40
net.(*netFD).connect(0x18648240, 0x0, 0x0, 0xb7517090, 0x18637400, 0xcc973afb, 0xe, 0x594127f, 0x83bba20, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:114 +0x21c
net.(*netFD).dial(0x18648240, 0xb7516020, 0x0, 0xb7516020, 0x18636f40, 0xcc973afb, 0xe, 0x594127f, 0x83bba20, 0x0, ...)
/usr/local/go/src/net/sock_posix.go:139 +0x2c6
net.socket(0x8285980, 0x3, 0x2, 0x1, 0x0, 0x18636f00, 0xb7516020, 0x0, 0xb7516020, 0x18636f40, ...)
/usr/local/go/src/net/sock_posix.go:91 +0x346
net.internetSocket(0x8285980, 0x3, 0xb7516020, 0x0, 0xb7516020, 0x18636f40, 0xcc973afb, 0xe, 0x594127f, 0x83bba20, ...)
/usr/local/go/src/net/ipsock_posix.go:137 +0xed
net.dialTCP(0x8285980, 0x3, 0x0, 0x18636f40, 0xcc973afb, 0xe, 0x594127f, 0x83bba20, 0x8063d1d, 0x0, ...)
/usr/local/go/src/net/tcpsock_posix.go:156 +0xfc
net.dialSingle(0x8285980, 0x3, 0x1862e534, 0x6, 0x0, 0x0, 0xb7515fe0, 0x18636f40, 0xcc973afb, 0xe, ...)
/usr/local/go/src/net/dial.go:235 +0x35a
net.func·016(0xcc973afb, 0xe, 0x594127f, 0x83bba20, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:163 +0xd0
net.dial(0x8285980, 0x3, 0xb7515fe0, 0x18636f40, 0x18690f50, 0xcc973afb, 0xe, 0x594127f, 0x83bba20, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:40 +0x63
net.(*Dialer).Dial(0x83b9880, 0x8285980, 0x3, 0x1862e534, 0x6, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:170 +0x3d8
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/transport.go:51 +0x67
github.com/valyala/gorpc.func·003()
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:314 +0x52
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:319 +0x17b

goroutine 28 [IO wait]:
net.(*pollDesc).Wait(0x18648378, 0x77, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x42
net.(*pollDesc).WaitWrite(0x18648378, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:93 +0x40
net.(*netFD).connect(0x18648340, 0x0, 0x0, 0xb7517090, 0x18637440, 0xcc973afb, 0xe, 0x598d446, 0x83bba20, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:114 +0x21c
net.(*netFD).dial(0x18648340, 0xb7516020, 0x0, 0xb7516020, 0x18637420, 0xcc973afb, 0xe, 0x598d446, 0x83bba20, 0x0, ...)
/usr/local/go/src/net/sock_posix.go:139 +0x2c6
net.socket(0x8285980, 0x3, 0x2, 0x1, 0x0, 0x18637400, 0xb7516020, 0x0, 0xb7516020, 0x18637420, ...)
/usr/local/go/src/net/sock_posix.go:91 +0x346
net.internetSocket(0x8285980, 0x3, 0xb7516020, 0x0, 0xb7516020, 0x18637420, 0xcc973afb, 0xe, 0x598d446, 0x83bba20, ...)
/usr/local/go/src/net/ipsock_posix.go:137 +0xed
net.dialTCP(0x8285980, 0x3, 0x0, 0x18637420, 0xcc973afb, 0xe, 0x598d446, 0x83bba20, 0x8063d1d, 0x0, ...)
/usr/local/go/src/net/tcpsock_posix.go:156 +0xfc
net.dialSingle(0x8285980, 0x3, 0x1862e534, 0x6, 0x0, 0x0, 0xb7515fe0, 0x18637420, 0xcc973afb, 0xe, ...)
/usr/local/go/src/net/dial.go:235 +0x35a
net.func·016(0xcc973afb, 0xe, 0x598d446, 0x83bba20, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:163 +0xd0
net.dial(0x8285980, 0x3, 0xb7515fe0, 0x18637420, 0x18691f50, 0xcc973afb, 0xe, 0x598d446, 0x83bba20, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:40 +0x63
net.(*Dialer).Dial(0x83b9880, 0x8285980, 0x3, 0x1862e534, 0x6, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:170 +0x3d8
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/transport.go:51 +0x67
github.com/valyala/gorpc.func·003()
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:314 +0x52
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:319 +0x17b

goroutine 29 [runnable]:
net.(*pollDesc).Wait(0x18648438, 0x77, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x42
net.(*pollDesc).WaitWrite(0x18648438, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:93 +0x40
net.(*netFD).connect(0x18648400, 0x0, 0x0, 0xb7517090, 0x18637480, 0xcc973afb, 0xe, 0x59ac8ff, 0x83bba20, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:114 +0x21c
net.(*netFD).dial(0x18648400, 0xb7516020, 0x0, 0xb7516020, 0x18637460, 0xcc973afb, 0xe, 0x59ac8ff, 0x83bba20, 0x0, ...)
/usr/local/go/src/net/sock_posix.go:139 +0x2c6
net.socket(0x8285980, 0x3, 0x2, 0x1, 0x0, 0x18637400, 0xb7516020, 0x0, 0xb7516020, 0x18637460, ...)
/usr/local/go/src/net/sock_posix.go:91 +0x346
net.internetSocket(0x8285980, 0x3, 0xb7516020, 0x0, 0xb7516020, 0x18637460, 0xcc973afb, 0xe, 0x59ac8ff, 0x83bba20, ...)
/usr/local/go/src/net/ipsock_posix.go:137 +0xed
net.dialTCP(0x8285980, 0x3, 0x0, 0x18637460, 0xcc973afb, 0xe, 0x59ac8ff, 0x83bba20, 0x8063d1d, 0x0, ...)
/usr/local/go/src/net/tcpsock_posix.go:156 +0xfc
net.dialSingle(0x8285980, 0x3, 0x1862e534, 0x6, 0x0, 0x0, 0xb7515fe0, 0x18637460, 0xcc973afb, 0xe, ...)
/usr/local/go/src/net/dial.go:235 +0x35a
net.func·016(0xcc973afb, 0xe, 0x59ac8ff, 0x83bba20, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:163 +0xd0
net.dial(0x8285980, 0x3, 0xb7515fe0, 0x18637460, 0x18692f50, 0xcc973afb, 0xe, 0x59ac8ff, 0x83bba20, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:40 +0x63
net.(*Dialer).Dial(0x83b9880, 0x8285980, 0x3, 0x1862e534, 0x6, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:170 +0x3d8
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/transport.go:51 +0x67
github.com/valyala/gorpc.func·003()
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:314 +0x52
/home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:319 +0x17b
exit status 2

Tamás Gulácsi

unread,
Mar 15, 2015, 8:30:27 AM3/15/15
to golan...@googlegroups.com
On 386, atomic nerds special care for alignment in structs!

Aliaksandr Valialkin

unread,
Mar 15, 2015, 8:52:58 PM3/15/15
to Tamás Gulácsi, golang-nuts
On Sun, Mar 15, 2015 at 2:30 PM, Tamás Gulácsi <tgula...@gmail.com> wrote:
On 386, atomic nerds special care for alignment in structs!

Shouldn't golang automatically align struct members in order to prevent unaligned memory access? As for 'atomic nerds' I just use atomic.AddUint64() for simple counters, which may be incremented from arbitrary number of concurrently running goroutines.

Caleb Spare

unread,
Mar 15, 2015, 8:57:41 PM3/15/15
to Aliaksandr Valialkin, Tamás Gulácsi, golang-nuts
Shouldn't golang automatically align struct members in order to prevent unaligned memory access? As for 'atomic nerds' I just use atomic.AddUint64() for simple counters, which may be incremented from arbitrary number of concurrently running goroutines.

​See the Bugs section at the bottom of http://golang.org/pkg/sync/atomic/:

"​
On x86-32, the 64-bit functions use instructions unavailable before the Pentium MMX. On non-Linux ARM, the 64-bit functions use instructions unavailable before the ARMv6k core. On both ARM and x86-32, it is the caller's responsibility to arrange for 64-bit alignment of 64-bit words accessed atomically. The first word in a global variable or in an allocated struct or slice can be relied upon to be 64-bit aligned.
​"​

minux

unread,
Mar 15, 2015, 9:51:58 PM3/15/15
to Aliaksandr Valialkin, Tamás Gulácsi, golang-nuts
The problem is that on 32-bit architectures, the alignment requirement for uint64 is just 4-byte,
however, 64-bit atomic instructions require 8-byte alignment.

Unfortunately, there is no easy way to get 8-byte alignment except placing the fields at the
beginning of a struct or manually adjust the position on a 32-bit platform, which is very tedious
and error-prone.
Reply all
Reply to author
Forward
0 new messages