Backrunner fatal error: invalid heap pointer

[Алексей Плотник]

I'm testing buckrunner daemon under the load. Backrunner proxies requests to 2 back-end nodes, 2 replicas. I'm using SSD.

What I have tried is to ran 6 threads (terminals), every send 10,000 inserts of different objects of the same size. Simply I used curl to do that:

    curl -X POST http://localhost:9090/upload/b1/object-w1-1 --data "20KB of data"

"w1" is for worker 1, "w2" is for worker 2 and so on. One-by-one, I tried to upload 10K objects (x6 workers) = 60K objects

Eventually backrunner process failed. Error is:

    runtime: garbage collector found invalid heap pointer *(0xb4cd60+0x240)=0x1 s=nil
    fatal error: invalid heap pointer

Full log (gzipped) https://mega.nz/#!dwZ1TJZY!5IVKNMv2BilaIEfq50f93VSlYEnddAzN0s7LzEIv4X4

Backrunner config: http://pastebin.com/C2PRA6KB

Node 1 config: http://pastebin.com/cZ9Ksvn9

Node 2 config: http://pastebin.com/xrAk5RJv

Is that a bug or should I tune something?

Thanks!

[Алексей Плотник]

bench.sh: http://pastebin.com/7YxYyPsA

[Evgeniy Polyakov]

Hi Alexey

21.07.2016, 09:13, "Алексей Плотник" <odis...@gmail.com>:

> I'm testing buckrunner daemon under the load. Backrunner proxies requests to 2 back-end nodes, 2 replicas. I'm using SSD.
>
> What I have tried is to ran 6 threads (terminals), every send 10,000 inserts of different objects of the same size. Simply I used curl to do that:
>
>     curl -X POST http://localhost:9090/upload/b1/object-w1-1 --data "20KB of data"
>
> "w1" is for worker 1, "w2" is for worker 2 and so on. One-by-one, I tried to upload 10K objects (x6 workers) = 60K objects
>
> Eventually backrunner process failed. Error is:
>
>     runtime: garbage collector found invalid heap pointer *(0xb4cd60+0x240)=0x1 s=nil
>     fatal error: invalid heap pointer

We will take a look at it.
Please tell us what were backrunner and elliptics-go commits used to build a server?
This info is provided in backrunner's /proxy_stat/ handler, please show it here.

Also what were the elliptics client version?
It is not allowed to change the library without recompiling backrunner and elliptics-go, otherwise really weird bugs occur.

[Алексей Плотник]

Hi, Eugeny.

GET /proxy_stats/ is located here: http://pastebin.com/LqbrtSUj

Elliptics components were installed from yum repo few days ago:

[root@dev ~]# yum list installed | grep elliptics
elliptics.x86_64                 2.26.11.0-1.el7                @reverbrain
elliptics-client.x86_64          2.26.11.0-1.el7                @reverbrain
elliptics-client-devel.x86_64    2.26.11.0-1.el7                @reverbrain

Backrunner was installed from git master few days ago:

[root@dev backrunner]# git show | head
commit cbf56105b42a4e894d03a81bb69c17ec221ce3b8
Author: Evgeniy Polyakov <zbr@ioremap.net>
Date:   Sun Jul 10 18:12:42 2016 +0300

Is that information enough?

Many thanks!

[Алексей Плотник]

go version go1.4.2 linux/amd64

[Evgeniy Polyakov]

Hi Alexey

As we discussed at @github, the solution is to upgrade to go 1.6, because of the following bugs
https://github.com/golang/go/issues/11907
https://github.com/golang/go/issues/12238

[Алексей Плотник]

Yes. Evgeniy, that was the best and fastest customer expirience I've ever seen. Thanks a lot. Now backrunner serves data like a boss!