Running logstash-forwarder causing a (apparently) go crash

[Daryl Erwin]

I am using the logstash-forwarder code and when I access stdin it causes the following SIGSEGV. Not sure if any of this is useful.

2014/05/28 13:37:43.163688 Loading registrar data

2014/05/28 13:37:43.163839 Loading client ssl certificate: ./logstash-forwarder.crt and /home/derwin/software/logstash-forwarder-master/logstash-forwarder.key

2014/05/28 13:37:43.175184 Starting harvester: -

2014/05/28 13:37:43.175212 Current file offset: 0

SIGSEGV: segmentation violation

PC=0x8059b14

scanblock(0xf71fe000, 0xf71fe024, 0x2, 0xf67fc100)

        /usr/local/go/src/pkg/runtime/mgc0.c:878 +0x754

markroot(0x1870c480, 0x7)

        /usr/local/go/src/pkg/runtime/mgc0.c:1210 +0x93

runtime.parfordo(0x1870c480)

        /usr/local/go/src/pkg/runtime/parfor.c:105 +0x98

gc(0xf7e13f84)

        /usr/local/go/src/pkg/runtime/mgc0.c:2113 +0x255

mgc(0x18701c80)

        /usr/local/go/src/pkg/runtime/mgc0.c:2053 +0x2e

runtime.mcall(0x8192e83)

        /usr/local/go/src/pkg/runtime/asm_386.s:190 +0x40

goroutine 8 [garbage collection]:

runtime.gc(0x1)

        /usr/local/go/src/pkg/runtime/mgc0.c:2024 +0x19b fp=0xf7e13f90

runfinq()

        /usr/local/go/src/pkg/runtime/mgc0.c:2325 +0x1ba fp=0xf7e13fcc

runtime.goexit()

        /usr/local/go/src/pkg/runtime/proc.c:1394 fp=0xf7e13fd0

created by runtime.gc

        /usr/local/go/src/pkg/runtime/mgc0.c:1966

goroutine 1 [chan receive]:

main.Registrar(0x1872f580)

        /home/derwin/software/logstash-forwarder-master/registrar.go:9 +0x5f

main.main()

        /home/derwin/software/logstash-forwarder-master/logstash-forwarder.go:72 +0x306

goroutine 3 [sleep]:

time.Sleep(0x540be400, 0x2)

        /usr/local/go/src/pkg/runtime/time.goc:31 +0x3c

main.Prospect(0x1870a8c0, 0x0, 0x4, 0x1870a8e0, 0x18731a80)

        /home/derwin/software/logstash-forwarder-master/prospector.go:34 +0x310

created by main.main

        /home/derwin/software/logstash-forwarder-master/logstash-forwarder.go:63 +0x27f

goroutine 4 [runnable]:

main.Spool(0x18731a80, 0x1872f540, 0x400, 0x0, 0x2a05f200, ...)

        /home/derwin/software/logstash-forwarder-master/spooler.go:26 +0x4e0

created by main.main

        /home/derwin/software/logstash-forwarder-master/logstash-forwarder.go:67 +0x2d3

goroutine 5 [runnable]:

math/big.nat.divLarge(0x187556c0, 0x20, 0x44, 0x18755900, 0x20, ...)

        /usr/local/go/src/pkg/math/big/nat.go:541 +0x2ce

math/big.nat.div(0x187556c0, 0x20, 0x44, 0x18755900, 0x20, ...)

        /usr/local/go/src/pkg/math/big/nat.go:519 +0x271

math/big.nat.expNNWindowed(0x187557e0, 0x40, 0x44, 0x1872d870, 0x20, ...)

        /usr/local/go/src/pkg/math/big/nat.go:1346 +0x694

math/big.nat.expNN(0x1872d900, 0x20, 0x24, 0x1872d870, 0x20, ...)

        /usr/local/go/src/pkg/math/big/nat.go:1255 +0x3b2

math/big.nat.probablyPrime(0x18701a00, 0x20, 0x25, 0x14, 0x80a2567)

        /usr/local/go/src/pkg/math/big/nat.go:1440 +0x588

math/big.(*Int).ProbablyPrime(0x18739340, 0x14, 0x0)

        /usr/local/go/src/pkg/math/big/int.go:721 +0x44

crypto/rsa.(*PrivateKey).Validate(0x1871c390, 0x2, 0x0)

        /usr/local/go/src/pkg/crypto/rsa/rsa.go:91 +0x8f

crypto/x509.ParsePKCS1PrivateKey(0x18754000, 0x4a7, 0x4a7, 0x1871c390, 0x0, ...)

        /usr/local/go/src/pkg/crypto/x509/pkcs1.go:77 +0x3f2

crypto/tls.parsePrivateKey(0x18754000, 0x4a7, 0x4a7, 0xc, 0x1874e601, ...)

        /usr/local/go/src/pkg/crypto/tls/tls.go:209 +0x3b

crypto/tls.X509KeyPair(0x1874c55f, 0x0, 0x200, 0x1874e68b, 0x0, ...)

        /usr/local/go/src/pkg/crypto/tls/tls.go:163 +0x255

crypto/tls.LoadX509KeyPair(0x1870a820, 0x18, 0x1873a1e0, 0x46, 0x0, ...)

        /usr/local/go/src/pkg/crypto/tls/tls.go:129 +0x107

main.connect(0x1872f480, 0x187433c0)

        /home/derwin/software/logstash-forwarder-master/publisher1.go:138 +0x16e

main.Publishv1(0x1872f540, 0x1872f580, 0x1872f480)

        /home/derwin/software/logstash-forwarder-master/publisher1.go:43 +0x59

created by main.main

        /home/derwin/software/logstash-forwarder-master/logstash-forwarder.go:69 +0x2f8

goroutine 6 [chan send]:

main.(*Harvester).Harvest(0x1870a940, 0x18731a80)

        /home/derwin/software/logstash-forwarder-master/harvester.go:83 +0x8c0

created by main.Prospect

        /home/derwin/software/logstash-forwarder-master/prospector.go:18 +0x12f

eax     0x1d

ebx     0x81a465c

ecx     0x10

edx     0x1870ac50

edi     0x0

esi     0x10001

ebp     0x0

esp     0xf67fc038

eip     0x8059b14

eflags  0x10287

cs      0x23

fs      0x0

gs      0x63

[Dmitry Vyukov]

Run it under race detector.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Daryl Erwin]

go build -race

go build: -race is only supported on linux/amd64, darwin/amd64 and windows/amd64

(I am not linux guru - not sure why its complaining - its linux)

uname -a

Linux r4m2 2.6.18-238.el5 #1 SMP Sun Dec 19 14:22:44 EST 2010 x86_64 x86_64 x86_64 GNU/Linux

[Russ Cox]

You are on a 64-bit Linux system but you are running 32-bit Go. You built with GOARCH=386 or copied a toolchain that did. Check your $GOARCH and check the output of 'go env'. You may need to rebuild the toolchain.

Once you move to 64-bit builds I think you will not see the problem anymore. However, before you do that, could you send us the output of 'go version'? We've seen problems like this before but we thought we had fixed them. I'd like to know the version so I can check on that.

Thanks.

Russ

[Daryl Erwin]

It seems to work fine on my rhel box. This other box i am not certain of the O/S or hw.

[Daryl Erwin]

from here: 

go1.2.2.linux-386.tar.gz

go version go1.2.2 linux/386

[Russ Cox]

Try using go1.2.2.linux-amd64.tgz instead.

Russ

[Daryl Erwin]

Now with  go1.2.2.linux-amd64.tar.gz

I simply get :-(

go version

Segmentation fault

[Russ Cox]

I am not sure why it is crashing, but the fact that your kernel is from 2010 may have something to do with it. Perhaps Go needs something not present in that kernel. I guess you should stick with the 386 version. :-(

If you have a box where amd64 works you could run the race detector there.

Russ

[Daryl Erwin]

Yes a bit surprising that this "jump" box is so old .. its a box that allows me access to our hadoop cluster. I will see if the others are any better. (Find it hard to believe its imaged that old)

[Daryl Erwin]

My infrastructure guys tell me all the hadoop access boxes are the same .. *but* in October they will be CentOS 6.5 -- good grief.

[Daryl Erwin]

Tried the newer version.. no segfault (yet?) but its giving me this error.

2014/05/28 16:03:27.457167 Connecting to 127.0.0.1:7000 (localhost)

2014/05/28 16:03:27.457361 Failed to tls handshake with 127.0.0.1 tls: either ServerName or InsecureSkipVerify must be specified in the tls.Config

[Brad Fitzpatrick]

That's documented here: http://tip.golang.org/doc/go1.3#major_library_changes

--

[Daryl Erwin]

Its telling me I need a tls.Config file and to set the values.

Now where that goes is another question..

[Andrew Lytvynov]

It's telling you that logstash-forwarder source needs to set either ServerName or InsecureSkipVerify in tls.Config struct it uses.

Nothing to do with files, it's an issue that they (logstash-forwarder authors) have to fix.

[mois...@gmail.com]

This workaround works for me!

You have to rebuild your package editing publisher1.go.


Here:
func connect(config *NetworkConfig) (socket *tls.Conn) {
  var tlsconfig tls.Config
.....

Yo have to add
  tlsconfig.InsecureSkipVerify = true
.....

So finally results:
func connect(config *NetworkConfig) (socket *tls.Conn) {
  var tlsconfig tls.Config
  tlsconfig.InsecureSkipVerify = true
.....

Rebuild your package. Take care because you are using Insecure verify with this method!;)

[nguyenki...@gmail.com]

Could you show me how to do it by your workaround solution?

[Moisés Rodríguez]

Hello!

----> You has to look for in the code:

func connect(config *NetworkConfig) (socket *tls.Conn) {
  var tlsconfig tls.Config

-----> Yo have to add
  tlsconfig.InsecureSkipVerify = true
.....

-----> So finally results:

func connect(config *NetworkConfig) (socket *tls.Conn) {
  var tlsconfig tls.Config
  tlsconfig.InsecureSkipVerify = true

2015-04-21 4:55 GMT+01:00 <nguyenki...@gmail.com>:

Could you show me how to do it by your workaround solution?

--

Atentamente, Moisés Rodríguez Barrera

------------------
mois...@gmail.com

[nguyenki...@gmail.com]

Could you please tell me the name of class that contains those code? So I can find it in google source code

[Dave Cheney]

Please take this discussion to golang-nuts.

On Tue, 21 Apr 2015 09:41  <nguyenki...@gmail.com> wrote:

Could you please tell me the name of class that contains those code? So I can find it in google source code

--

[Moisés Rodríguez]

You can grep it in the whole source code.

[Nguyen Ryan]

Thank you very much man :)