too many open files issue
525 views
Skip to first unread message
靳禹
Mar 22, 2015, 11:10:01 PM3/22/15
to consu...@googlegroups.com
Hi all,
I recently ran into this "too many open files" issue when using watch handler. I have a handler written in Go to watch for a keyprefix like `nodename/`, under that there a layout like `nodename/services/`, `nodename/checks/` etc. and several value nodes under each secondary paths. There is another handler written in bash for all the checks, which is just a dummy one just out put the input to a local file.
Now I am seeing this error each time I call a `reload` for consul server
[ERR] agent: Failed to invoke watch handler '/usr/local/l7nxg/bin/consul-seneschal/seneschal -v 3 -log_dir /data/log/seneschal -rpc-addr 172.31.16.147:8400': pipe2: too many open files
Any ideas to fix this? What have I been doing wrong about the watch handler?
Thanks in advance.
靳禹
Mar 22, 2015, 11:35:06 PM3/22/15
to consu...@googlegroups.com
Actually, this issue has been escalated ....
在 2015年3月23日星期一 UTC+8上午11:10:01，靳禹写道：
2015/03/23 03:30:52 [ERR] agent.rpc: Failed to accept client: accept tcp 172.31.16.147:8400: too many open files
2015/03/23 03:30:52 [ERR] agent.rpc: Failed to accept client: accept tcp 172.31.16.147:8400: too many open files
2015/03/23 03:30:52 [ERR] agent.rpc: Failed to accept client: accept tcp 172.31.16.147:8400: too many open files
2015/03/23 03:30:52 [ERR] agent.rpc: Failed to accept client: accept tcp 172.31.16.147:8400: too many open files
2015/03/23 03:30:52 [ERR] agent.rpc: Failed to accept client: accept tcp 172.31.16.147:8400: too many open files
2015/03/23 03:30:52 [ERR] agent.rpc: Failed to accept client: accept tcp 172.31.16.147:8400: too many open files
2015/03/23 03:30:52 [ERR] agent.rpc: Failed to accept client: accept tcp 172.31.16.147:8400: too many open files
2015/03/23 03:30:52 [ERR] agent.rpc: Failed to accept client: accept tcp 172.31.16.147:8400: too many open files
2015/03/23 03:30:52 [ERR] agent.rpc: Failed to accept client: accept tcp 172.31.16.147:8400: too many open files
2015/03/23 03:30:52 [ERR] agent.rpc: Failed to accept client: accept tcp 172.31.16.147:8400: too many open files
2015/03/23 03:30:52 [ERR] agent.rpc: Failed to accept client: accept tcp 172.31.16.147:8400: too many open files
2015/03/23 03:31:26 [ERR] http: Request /v1/kv/mongo-master/?index=73503&recurse=, error: rpc error: unexpected EOF
2015/03/23 03:31:26 [ERR] http: Request /v1/kv/mongo-master/?index=73503&recurse=, error: rpc error: unexpected EOF
2015/03/23 03:31:26 [ERR] http: Request /v1/kv/mongo-master/?index=73503&recurse=, error: rpc error: unexpected EOF
2015/03/23 03:31:26 [ERR] http: Request /v1/kv/mongo-master/?index=73503&recurse=, error: rpc error: unexpected EOF
2015/03/23 03:29:09 [ERR] agent: failed to invoke '/usr/local/nagios/libexec/check_disk -w 80 -c 90 -p /opt/disk2': fork/exec /bin/bash: too many open files
2015/03/23 03:29:15 [ERR] agent: failed to invoke '/usr/local/nagios/libexec/check_memory.sh -w 70 -c 90 -i': fork/exec /bin/bash: too many open files
2015/03/23 03:29:28 [ERR] agent: failed to invoke '/usr/local/nagios/libexec/check_load -r -w 3,2,2 -c 6,5,5': fork/exec /bin/bash: too many open files
2015/03/23 03:29:30 [ERR] agent: failed to invoke '/usr/local/nagios/libexec/check_load -r -w 3,3,3 -c 6,6,6': fork/exec /bin/bash: too many open files
I am seeing enormous error logs as above... help is bloody needed :(
I did not even know what's causing this ....
在 2015年3月23日星期一 UTC+8上午11:10:01，靳禹写道：
靳禹
Mar 22, 2015, 11:41:56 PM3/22/15
to consu...@googlegroups.com
The error log from Consul might be helpful for debugging
goroutine 453553 [IO wait]:
net.(*pollDesc).Wait(0xc20a8a5100, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20a8a5100, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20a8a50a0, 0xc208be5000, 0x1000, 0x1000, 0x0, 0x7fd078e59ef8, 0xc20a9d9cd0)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20a6122d0, 0xc208be5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7fd078e5cf30, 0xc20a6122d0, 0xc20aa65238, 0xc208be5000, 0x1000, 0x1000, 0xa6c0a0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc20a3021e0, 0xc208be5000, 0x1000, 0x1000, 0xc208012000, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc20b109e60)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc20b109e60, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc20aa651e0)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f
goroutine 420649 [IO wait, 1063 minutes]:
net.(*pollDesc).Wait(0xc209e36d10, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc209e36d10, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc209e36cb0, 0xc209b8c000, 0x1000, 0x1000, 0x0, 0x7fd078e59ef8, 0xc20884ec18)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20aae95d0, 0xc209b8c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7fd078e5cf30, 0xc20aae95d0, 0xc208a90688, 0xc209b8c000, 0x1000, 0x1000, 0x7fd078e48010, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc2096cb520, 0xc209b8c000, 0x1000, 0x1000, 0xc207d516a2, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc208c67920)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc208c67920, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc208a90630)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f
goroutine 434063 [select, 1062 minutes]:
net/http.(*persistConn).roundTrip(0xc209744a50, 0xc2098170e0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1082 +0x7ad
net/http.(*Transport).RoundTrip(0xc208092090, 0xc20acef6c0, 0xc209f17b80, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:235 +0x558
net/http.send(0xc20acef6c0, 0x7fd078e5a0a0, 0xc208092090, 0x42, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:219 +0x4fc
net/http.(*Client).send(0x119ed40, 0xc20acef6c0, 0x42, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:142 +0x15b
net/http.(*Client).doFollowingRedirects(0x119ed40, 0xc20acef6c0, 0xd6d580, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:367 +0xb25
net/http.(*Client).Do(0x119ed40, 0xc20acef6c0, 0x119efc0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:174 +0xa4
github.com/hashicorp/consul/api.(*Client).doRequest(0xc20a8aa3c0, 0xc209f17b30, 0xc20ad21ae0, 0xc20ad21b08, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/api.go:287 +0xde
github.com/hashicorp/consul/api.(*KV).getInternal(0xc20adf1f18, 0xc20884e150, 0xd, 0xc20a88dc20, 0xc20a8efc80, 0xc20a8efc80, 0xc20a8efc80, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/kv.go:104 +0x289
github.com/hashicorp/consul/api.(*KV).List(0xc20adf1f18, 0xc20884e150, 0xd, 0xc20a8efc80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/kv.go:59 +0x136
github.com/hashicorp/consul/watch.func·002(0xc209679c20, 0x9bdd00, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/funcs.go:66 +0x136
github.com/hashicorp/consul/watch.(*WatchPlan).Run(0xc209679c20, 0xc20a253280, 0x12, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/plan.go:48 +0x3fb
github.com/hashicorp/consul/command/agent.func·013(0xc209679c20)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:822 +0x190
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:825 +0xc13
goroutine 296407 [select, 4052 minutes]:
net/http.(*persistConn).writeLoop(0xc208a911e0)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
goroutine 315968 [IO wait, 3861 minutes]:
net.(*pollDesc).Wait(0xc2083526f0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2083526f0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208352690, 0xc20863e000, 0x1000, 0x1000, 0x0, 0x7fd078e59ef8, 0xc2086477e8)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20803bb00, 0xc20863e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7fd078e5cf30, 0xc20803bb00, 0xc208d2cf78, 0xc20863e000, 0x1000, 0x1000, 0x7fd078e48010, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc20a1d8980, 0xc20863e000, 0x1000, 0x1000, 0xc207ffc44f, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc20a5989c0)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc20a5989c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc208d2cf20)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f
goroutine 484527 [select]:
github.com/hashicorp/consul/watch.(*WatchPlan).Run(0xc2086919a0, 0xc209a825c0, 0x12, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/plan.go:66 +0x763
github.com/hashicorp/consul/command/agent.func·013(0xc2086919a0)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:822 +0x190
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:825 +0xc13
goroutine 453454 [select, 855 minutes]:
net/http.(*persistConn).writeLoop(0xc20a051340)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
root@ip-172-31-16-147:/data/supervisor/consul# vi consul.err
root@ip-172-31-16-147:/data/supervisor/consul#
root@ip-172-31-16-147:/data/supervisor/consul# cat *err
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x495361]
goroutine 1 [running]:
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:807 +0xc81
github.com/hashicorp/consul/command/agent.(*Command).handleSignals(0xc2080320b0, 0x0, 0xc208286660, 0xc2082866c0, 0x1)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:716 +0x4d9
github.com/hashicorp/consul/command/agent.(*Command).Run(0xc2080320b0, 0xc20800a020, 0x13, 0x13, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:686 +0x23dd
github.com/mitchellh/cli.(*CLI).Run(0xc20806e280, 0xc20802b7a0, 0x0, 0x0)
/opt/gopath/src/github.com/mitchellh/cli/cli.go:100 +0x3ce
main.realMain(0x412f8a)
/opt/gopath/src/github.com/hashicorp/consul/main.go:37 +0x36c
main.main()
/opt/gopath/src/github.com/hashicorp/consul/main.go:12 +0x1f
goroutine 5 [syscall, 4053 minutes]:
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
/usr/local/go/src/os/signal/signal_unix.go:27 +0x35
goroutine 10 [select, 14069 minutes]:
/opt/gopath/src/github.com/armon/go-metrics/inmem_signal.go:63 +0xf1
created by github.com/armon/go-metrics.NewInmemSignal
/opt/gopath/src/github.com/armon/go-metrics/inmem_signal.go:37 +0x258
goroutine 11 [sleep]:
/opt/gopath/src/github.com/armon/go-metrics/metrics.go:67 +0x2c
created by github.com/armon/go-metrics.New
/opt/gopath/src/github.com/armon/go-metrics/start.go:61 +0xe8
goroutine 13 [runnable]:
/opt/gopath/src/github.com/hashicorp/consul/consul/pool.go:363 +0x4bb
created by github.com/hashicorp/consul/consul.NewPool
/opt/gopath/src/github.com/hashicorp/consul/consul/pool.go:156 +0x165
goroutine 17 [syscall, 14069 minutes, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1
goroutine 50 [runnable]:
syscall.Syscall(0x3, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/syscall/asm_linux_amd64.s:21 +0x5
syscall.Close(0x3, 0x0, 0x0)
/usr/local/go/src/syscall/zsyscall_linux_amd64.go:280 +0x55
net.closesocket(0x3, 0x0, 0x0)
/usr/local/go/src/net/fd_unix.go:510 +0x3a
net.(*netFD).destroy(0xc2080f1e30)
/usr/local/go/src/net/fd_unix.go:135 +0x46
net.(*netFD).readUnlock(0xc2080f1e30)
/usr/local/go/src/net/fd_unix.go:169 +0x4c
net.(*netFD).accept(0xc2080f1e30, 0x0, 0x7fd078e6d3c0, 0xc208d93e40)
/usr/local/go/src/net/fd_unix.go:427 +0x579
net.(*TCPListener).AcceptTCP(0xc20803a200, 0x689ac4, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
net.(*TCPListener).Accept(0xc20803a200, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:244 +0x4c
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:48 +0x56
created by github.com/hashicorp/consul/consul.NewServer
/opt/gopath/src/github.com/hashicorp/consul/consul/server.go:278 +0x1364
goroutine 55 [runnable]:
/opt/gopath/src/github.com/hashicorp/consul/command/agent/user_event.go:114 +0x2cc
/opt/gopath/src/github.com/hashicorp/consul/command/agent/agent.go:176 +0x9fa
goroutine 485093 [IO wait]:
net.(*pollDesc).Wait(0xc20a025a30, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20a025a30, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20a0259d0, 0xc208ec2000, 0x2, 0x10000, 0x0, 0x7fd078e59ef8, 0xc20ac065c0)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20adf02e0, 0xc208ec2000, 0x2, 0x10000, 0x41fc01, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
github.com/miekg/dns.(*Conn).Read(0xc209963bd0, 0xc208ec2000, 0x10000, 0x10000, 0x10000, 0x0, 0x0)
/opt/gopath/src/github.com/miekg/dns/client.go:203 +0x150
github.com/miekg/dns.(*Conn).ReadMsg(0xc209963bd0, 0xc20931b170, 0x0, 0x0)
/opt/gopath/src/github.com/miekg/dns/client.go:175 +0x128
github.com/miekg/dns.(*Client).exchange(0xc209963b80, 0xc20931b170, 0xc208163300, 0x12, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/miekg/dns/client.go:156 +0x38c
github.com/miekg/dns.(*Client).Exchange(0xc209963b80, 0xc20931b170, 0xc208163300, 0x12, 0xffffffffffffff01, 0x425b29, 0x0, 0x0)
/opt/gopath/src/github.com/miekg/dns/client.go:95 +0x83
created by github.com/miekg/dns.(*Server).Shutdown
/opt/gopath/src/github.com/miekg/dns/server.go:355 +0x4cd
goroutine 64 [select, 1470 minutes]:
github.com/hashicorp/go-checkpoint.func·001()
/opt/gopath/src/github.com/hashicorp/go-checkpoint/checkpoint.go:188 +0x154
/opt/gopath/src/github.com/hashicorp/go-checkpoint/checkpoint.go:196 +0x1d5
goroutine 67 [runnable]:
/opt/gopath/src/github.com/hashicorp/consul/command/agent/local.go:280 +0x33a
/opt/gopath/src/github.com/hashicorp/consul/command/agent/agent.go:506 +0x4a
goroutine 169333 [select, 8054 minutes]:
github.com/hashicorp/yamux.(*Stream).Read(0xc2083b8c80, 0xc208298000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/yamux/stream.go:125 +0x3ab
bufio.(*Reader).fill(0xc2083cc840)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadByte(0xc2083cc840, 0x10, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:196 +0x86
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/decode.go:90 +0x45
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/msgpack.go:540 +0x48
github.com/hashicorp/go-msgpack/codec.(*Decoder).decode(0xc2083cc8a0, 0x993b20, 0xc2082bed80)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/decode.go:635 +0x59
github.com/hashicorp/go-msgpack/codec.(*Decoder).Decode(0xc2083cc8a0, 0x993b20, 0xc2082bed80, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/decode.go:630 +0x7f
github.com/hashicorp/go-msgpack/codec.(*rpcCodec).read(0xc20856bd80, 0x993b20, 0xc2082bed80, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/rpc.go:88 +0xd9
github.com/hashicorp/go-msgpack/codec.(*goRpcCodec).ReadRequestHeader(0xc20856bd80, 0xc2082bed80, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/rpc.go:127 +0x61
net/rpc.(*Server).readRequestHeader(0xc20808ca80, 0x7fd078e6c6a8, 0xc20856bd80, 0x0, 0x0, 0xc2082bed80, 0xc20803a100, 0x0, 0x0)
/usr/local/go/src/net/rpc/server.go:575 +0x97
net/rpc.(*Server).readRequest(0xc20808ca80, 0x7fd078e6c6a8, 0xc20856bd80, 0xc200000000, 0xc2082bed80, 0xb3e6c0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/rpc/server.go:542 +0xa2
net/rpc.(*Server).ServeRequest(0xc20808ca80, 0x7fd078e6c6a8, 0xc20856bd80, 0x0, 0x0)
/usr/local/go/src/net/rpc/server.go:485 +0x7f
github.com/hashicorp/consul/consul.(*Server).handleConsulConn(0xc20800a420, 0x7fd078e5f2c0, 0xc2083b8c80)
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:160 +0x160
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:145 +0x31d
goroutine 4978 [IO wait, 13916 minutes]:
net.(*pollDesc).Wait(0xc208256ca0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208256ca0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208256c40, 0xc20831d000, 0x1000, 0x1000, 0x0, 0x7fd078e59ef8, 0xc2083eaaa0)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc2082a2090, 0xc20831d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
bufio.(*Reader).fill(0xc208148c00)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadByte(0xc208148c00, 0x300000002, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:196 +0x86
github.com/hashicorp/raft.(*NetworkTransport).handleCommand(0xc2080f0000, 0xc208148c00, 0xc208148c60, 0xc208256d20, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/raft/net_transport.go:391 +0x4e
github.com/hashicorp/raft.(*NetworkTransport).handleConn(0xc2080f0000, 0x7fd078e5cdc8, 0xc2082a2090)
/opt/gopath/src/github.com/hashicorp/raft/net_transport.go:375 +0x278
/opt/gopath/src/github.com/hashicorp/raft/net_transport.go:362 +0x34c
goroutine 90 [IO wait, 13916 minutes]:
net.(*pollDesc).Wait(0xc20814eca0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20814eca0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20814ec40, 0xc208170000, 0x1000, 0x1000, 0x0, 0x7fd078e59ef8, 0xc2083e0570)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc2082a2810, 0xc208170000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
bufio.(*Reader).fill(0xc2082d6780)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadByte(0xc2082d6780, 0x300000002, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:196 +0x86
github.com/hashicorp/raft.(*NetworkTransport).handleCommand(0xc2080f0000, 0xc2082d6780, 0xc2082d67e0, 0xc2081cf3b0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/raft/net_transport.go:391 +0x4e
github.com/hashicorp/raft.(*NetworkTransport).handleConn(0xc2080f0000, 0x7fd078e5cdc8, 0xc2082a2810)
/opt/gopath/src/github.com/hashicorp/raft/net_transport.go:375 +0x278
/opt/gopath/src/github.com/hashicorp/raft/net_transport.go:362 +0x34c
goroutine 4993 [select, 8054 minutes]:
github.com/hashicorp/yamux.(*Session).AcceptStream(0xc2083bc6e0, 0xc2081d2e08, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/yamux/session.go:162 +0x173
github.com/hashicorp/yamux.(*Session).Accept(0xc2083bc6e0, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/yamux/session.go:156 +0x4c
github.com/hashicorp/consul/consul.(*Server).handleMultiplexV2(0xc20800a420, 0x7fd078e5cdc8, 0xc2082a2120)
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:138 +0x188
github.com/hashicorp/consul/consul.(*Server).handleConn(0xc20800a420, 0x7fd078e5cdc8, 0xc2082a2120, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:104 +0x6b5
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:57 +0x1bb
goroutine 4996 [select]:
/opt/gopath/src/github.com/hashicorp/yamux/session.go:254 +0x106
created by github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/yamux/session.go:96 +0x435
goroutine 5033 [select]:
github.com/hashicorp/yamux.(*Stream).Read(0xc20814af00, 0xc2084d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/yamux/stream.go:125 +0x3ab
bufio.(*Reader).fill(0xc2083ccf60)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadByte(0xc2083ccf60, 0x3, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:196 +0x86
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/decode.go:90 +0x45
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/msgpack.go:540 +0x48
github.com/hashicorp/go-msgpack/codec.(*Decoder).decode(0xc2083ccfc0, 0x993b20, 0xc2083e63e0)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/decode.go:635 +0x59
github.com/hashicorp/go-msgpack/codec.(*Decoder).Decode(0xc2083ccfc0, 0x993b20, 0xc2083e63e0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/decode.go:630 +0x7f
github.com/hashicorp/go-msgpack/codec.(*rpcCodec).read(0xc208596dc0, 0x993b20, 0xc2083e63e0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/rpc.go:88 +0xd9
github.com/hashicorp/go-msgpack/codec.(*goRpcCodec).ReadRequestHeader(0xc208596dc0, 0xc2083e63e0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/rpc.go:127 +0x61
net/rpc.(*Server).readRequestHeader(0xc20808ca80, 0x7fd078e6c6a8, 0xc208596dc0, 0x0, 0x0, 0xc2083e63e0, 0xc20803a100, 0x0, 0x0)
/usr/local/go/src/net/rpc/server.go:575 +0x97
net/rpc.(*Server).readRequest(0xc20808ca80, 0x7fd078e6c6a8, 0xc208596dc0, 0xc200000000, 0xc2083e63e0, 0xb3e6c0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/rpc/server.go:542 +0xa2
net/rpc.(*Server).ServeRequest(0xc20808ca80, 0x7fd078e6c6a8, 0xc208596dc0, 0x0, 0x0)
/usr/local/go/src/net/rpc/server.go:485 +0x7f
github.com/hashicorp/consul/consul.(*Server).handleConsulConn(0xc20800a420, 0x7fd078e5f2c0, 0xc20814af00)
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:160 +0x160
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:145 +0x31d
goroutine 5032 [select]:
/opt/gopath/src/github.com/hashicorp/yamux/session.go:254 +0x106
created by github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/yamux/session.go:96 +0x435
goroutine 5030 [IO wait]:
net.(*pollDesc).Wait(0xc2081cf020, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2081cf020, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc2081cefc0, 0xc2084c9000, 0x1000, 0x1000, 0x0, 0x7fd078e59ef8, 0xc20ba048b0)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc2082a2030, 0xc2084c9000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
bufio.(*Reader).fill(0xc2083ccae0)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Read(0xc2083ccae0, 0xc208397530, 0xc, 0xc, 0xc2081ce488, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:174 +0x26c
io.ReadAtLeast(0x7fd078e5c3b8, 0xc2083ccae0, 0xc208397530, 0xc, 0xc, 0xc, 0x0, 0x0, 0x0)
/usr/local/go/src/io/io.go:298 +0xf1
io.ReadFull(0x7fd078e5c3b8, 0xc2083ccae0, 0xc208397530, 0xc, 0xc, 0xc, 0x0, 0x0)
/usr/local/go/src/io/io.go:316 +0x6d
/opt/gopath/src/github.com/hashicorp/yamux/session.go:340 +0xcb
created by github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/yamux/session.go:93 +0x3f0
goroutine 95413 [select, 9505 minutes]:
github.com/hashicorp/yamux.(*Stream).Read(0xc2085f8b40, 0xc2083c5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/yamux/stream.go:125 +0x3ab
bufio.(*Reader).fill(0xc208362840)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadByte(0xc208362840, 0x3, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:196 +0x86
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/decode.go:90 +0x45
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/msgpack.go:540 +0x48
github.com/hashicorp/go-msgpack/codec.(*Decoder).decode(0xc2083628a0, 0x993b20, 0xc2083ff680)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/decode.go:635 +0x59
github.com/hashicorp/go-msgpack/codec.(*Decoder).Decode(0xc2083628a0, 0x993b20, 0xc2083ff680, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/decode.go:630 +0x7f
github.com/hashicorp/go-msgpack/codec.(*rpcCodec).read(0xc20822bac0, 0x993b20, 0xc2083ff680, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/rpc.go:88 +0xd9
github.com/hashicorp/go-msgpack/codec.(*goRpcCodec).ReadRequestHeader(0xc20822bac0, 0xc2083ff680, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/rpc.go:127 +0x61
net/rpc.(*Server).readRequestHeader(0xc20808ca80, 0x7fd078e6c6a8, 0xc20822bac0, 0x0, 0x0, 0xc2083ff680, 0xc20803a100, 0x0, 0x0)
/usr/local/go/src/net/rpc/server.go:575 +0x97
net/rpc.(*Server).readRequest(0xc20808ca80, 0x7fd078e6c6a8, 0xc20822bac0, 0xc200000000, 0xc2083ff680, 0xb3e800, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/rpc/server.go:542 +0xa2
net/rpc.(*Server).ServeRequest(0xc20808ca80, 0x7fd078e6c6a8, 0xc20822bac0, 0x0, 0x0)
/usr/local/go/src/net/rpc/server.go:485 +0x7f
github.com/hashicorp/consul/consul.(*Server).handleConsulConn(0xc20800a420, 0x7fd078e5f2c0, 0xc2085f8b40)
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:160 +0x160
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:145 +0x31d
goroutine 4995 [select]:
/opt/gopath/src/github.com/hashicorp/yamux/session.go:298 +0x554
created by github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/yamux/session.go:94 +0x40a
goroutine 5029 [select, 10664 minutes]:
github.com/hashicorp/yamux.(*Session).AcceptStream(0xc20836e640, 0xc20840fe08, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/yamux/session.go:162 +0x173
github.com/hashicorp/yamux.(*Session).Accept(0xc20836e640, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/yamux/session.go:156 +0x4c
github.com/hashicorp/consul/consul.(*Server).handleMultiplexV2(0xc20800a420, 0x7fd078e5cdc8, 0xc2082a2030)
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:138 +0x188
github.com/hashicorp/consul/consul.(*Server).handleConn(0xc20800a420, 0x7fd078e5cdc8, 0xc2082a2030, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:104 +0x6b5
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:57 +0x1bb
goroutine 4997 [select]:
github.com/hashicorp/yamux.(*Stream).Read(0xc20814ba40, 0xc2084cf000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/yamux/stream.go:125 +0x3ab
bufio.(*Reader).fill(0xc2083dbf80)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadByte(0xc2083dbf80, 0x3, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:196 +0x86
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/decode.go:90 +0x45
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/msgpack.go:540 +0x48
github.com/hashicorp/go-msgpack/codec.(*Decoder).decode(0xc208326000, 0x993b20, 0xc2083909a0)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/decode.go:635 +0x59
github.com/hashicorp/go-msgpack/codec.(*Decoder).Decode(0xc208326000, 0x993b20, 0xc2083909a0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/decode.go:630 +0x7f
github.com/hashicorp/go-msgpack/codec.(*rpcCodec).read(0xc208157f00, 0x993b20, 0xc2083909a0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/rpc.go:88 +0xd9
github.com/hashicorp/go-msgpack/codec.(*goRpcCodec).ReadRequestHeader(0xc208157f00, 0xc2083909a0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/rpc.go:127 +0x61
net/rpc.(*Server).readRequestHeader(0xc20808ca80, 0x7fd078e6c6a8, 0xc208157f00, 0x0, 0x0, 0xc2083909a0, 0xc20803a100, 0x0, 0x0)
/usr/local/go/src/net/rpc/server.go:575 +0x97
net/rpc.(*Server).readRequest(0xc20808ca80, 0x7fd078e6c6a8, 0xc208157f00, 0xc200000000, 0xc2083909a0, 0xb3e800, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/rpc/server.go:542 +0xa2
net/rpc.(*Server).ServeRequest(0xc20808ca80, 0x7fd078e6c6a8, 0xc208157f00, 0x0, 0x0)
/usr/local/go/src/net/rpc/server.go:485 +0x7f
github.com/hashicorp/consul/consul.(*Server).handleConsulConn(0xc20800a420, 0x7fd078e5f2c0, 0xc20814ba40)
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:160 +0x160
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:145 +0x31d
goroutine 5031 [select]:
/opt/gopath/src/github.com/hashicorp/yamux/session.go:298 +0x554
created by github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/yamux/session.go:94 +0x40a
goroutine 4994 [IO wait]:
net.(*pollDesc).Wait(0xc208305cd0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208305cd0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208305c70, 0xc2085f5000, 0x1000, 0x1000, 0x0, 0x7fd078e59ef8, 0xc208df7970)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc2082a2120, 0xc2085f5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
bufio.(*Reader).fill(0xc2083db6e0)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Read(0xc2083db6e0, 0xc208387824, 0xc, 0xc, 0xc, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:174 +0x26c
io.ReadAtLeast(0x7fd078e5c3b8, 0xc2083db6e0, 0xc208387824, 0xc, 0xc, 0xc, 0x0, 0x0, 0x0)
/usr/local/go/src/io/io.go:298 +0xf1
io.ReadFull(0x7fd078e5c3b8, 0xc2083db6e0, 0xc208387824, 0xc, 0xc, 0xc, 0x0, 0x0)
/usr/local/go/src/io/io.go:316 +0x6d
/opt/gopath/src/github.com/hashicorp/yamux/session.go:340 +0xcb
created by github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/yamux/session.go:93 +0x3f0
goroutine 470958 [select]:
github.com/hashicorp/consul/consul.(*Server).blockingRPCOpt(0xc20800a420, 0xc20a3cf6d0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:370 +0x2b6
github.com/hashicorp/consul/consul.(*Server).blockingRPC(0xc20800a420, 0xc20a3cf6a0, 0xc209b1b5a8, 0xc20803a458, 0x1, 0x1, 0xc209f99fe0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:298 +0xdd
github.com/hashicorp/consul/consul.(*Health).ChecksInState(0xc20803a1d8, 0xc20a3cf680, 0xc209b1b590, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/consul/health_endpoint.go:29 +0x38f
reflect.Value.call(0xa8db60, 0xb3d5e0, 0x13, 0xbd0100, 0x4, 0xc20a455830, 0x3, 0x3, 0x0, 0x0, ...)
/usr/local/go/src/reflect/value.go:419 +0x10e5
reflect.Value.Call(0xa8db60, 0xb3d5e0, 0x13, 0xc20a455830, 0x3, 0x3, 0x0, 0x0, 0x0)
/usr/local/go/src/reflect/value.go:296 +0xbc
net/rpc.(*service).call(0xc20808cb40, 0xc20808ca80, 0xc20ac06370, 0xc2080ace80, 0xc20a6350e0, 0xb3dcc0, 0xc20a3cf680, 0x16, 0x986860, 0xc209b1b590, ...)
/usr/local/go/src/net/rpc/server.go:382 +0x1f7
net/rpc.(*Server).ServeRequest(0xc20808ca80, 0x7fd078e5e8d0, 0xc20ae7e300, 0x0, 0x0)
/usr/local/go/src/net/rpc/server.go:497 +0x2a0
github.com/hashicorp/consul/consul.(*Server).RPC(0xc20800a420, 0xc3e390, 0x14, 0xb3dcc0, 0xc20a3cf630, 0x986860, 0xc209b1b530, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/consul/server.go:708 +0x13a
github.com/hashicorp/consul/command/agent.(*Agent).RPC(0xc208070500, 0xc3e390, 0x14, 0xb3dcc0, 0xc20a3cf630, 0x986860, 0xc209b1b530, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/agent.go:374 +0x8e
github.com/hashicorp/consul/command/agent.(*HTTPServer).HealthChecksInState(0xc208190550, 0x7fd078e5eb70, 0xc20a1f26e0, 0xc20964b450, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/health_endpoint.go:27 +0x2e7
github.com/hashicorp/consul/command/agent.*HTTPServer.HealthChecksInState·fm(0x7fd078e5eb70, 0xc20a1f26e0, 0xc20964b450, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/http.go:209 +0x69
github.com/hashicorp/consul/command/agent.func·022(0x7fd078e5eb70, 0xc20a1f26e0, 0xc20964b450)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/http.go:290 +0x160
net/http.HandlerFunc.ServeHTTP(0xc208162c60, 0x7fd078e5eb70, 0xc20a1f26e0, 0xc20964b450)
/usr/local/go/src/net/http/server.go:1265 +0x41
net/http.(*ServeMux).ServeHTTP(0xc208172db0, 0x7fd078e5eb70, 0xc20a1f26e0, 0xc20964b450)
/usr/local/go/src/net/http/server.go:1541 +0x17d
net/http.serverHandler.ServeHTTP(0xc208094b40, 0x7fd078e5eb70, 0xc20a1f26e0, 0xc20964b450)
/usr/local/go/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc20a685720)
/usr/local/go/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:1751 +0x35e
goroutine 406871 [select, 1075 minutes]:
net/http.(*persistConn).roundTrip(0xc208539080, 0xc208d17b00, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1082 +0x7ad
net/http.(*Transport).RoundTrip(0xc208092090, 0xc209aaa8f0, 0xc20a9b8000, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:235 +0x558
net/http.send(0xc209aaa8f0, 0x7fd078e5a0a0, 0xc208092090, 0x42, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:219 +0x4fc
net/http.(*Client).send(0x119ed40, 0xc209aaa8f0, 0x42, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:142 +0x15b
net/http.(*Client).doFollowingRedirects(0x119ed40, 0xc209aaa8f0, 0xd6d580, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:367 +0xb25
net/http.(*Client).Do(0x119ed40, 0xc209aaa8f0, 0x119efc0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:174 +0xa4
github.com/hashicorp/consul/api.(*Client).doRequest(0xc20aeed080, 0xc208fd9f90, 0xc209fc9ae0, 0xc209fc9b08, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/api.go:287 +0xde
github.com/hashicorp/consul/api.(*KV).getInternal(0xc208aeebd8, 0xc208387e20, 0xd, 0xc20882edb0, 0xc20a9f1980, 0xc20a9f1980, 0xc20a9f1980, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/kv.go:104 +0x289
github.com/hashicorp/consul/api.(*KV).List(0xc208aeebd8, 0xc208387e20, 0xd, 0xc20a9f1980, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/kv.go:59 +0x136
github.com/hashicorp/consul/watch.func·002(0xc20823c5a0, 0x9bdd00, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/funcs.go:66 +0x136
github.com/hashicorp/consul/watch.(*WatchPlan).Run(0xc20823c5a0, 0xc20a692660, 0x12, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/plan.go:48 +0x3fb
github.com/hashicorp/consul/command/agent.func·013(0xc20823c5a0)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:822 +0x190
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:825 +0xc13
goroutine 485081 [select]:
/opt/gopath/src/github.com/hashicorp/yamux/session.go:254 +0x106
created by github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/yamux/session.go:96 +0x435
goroutine 453554 [select]:
net/http.(*persistConn).writeLoop(0xc20aa651e0)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
goroutine 315969 [select, 3861 minutes]:
net/http.(*persistConn).writeLoop(0xc208d2cf20)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
goroutine 484528 [select]:
net/http.(*persistConn).roundTrip(0xc20aa651e0, 0xc20a9d9c80, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1082 +0x7ad
net/http.(*Transport).RoundTrip(0xc208092090, 0xc20964b520, 0xc208d93fc0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:235 +0x558
net/http.send(0xc20964b520, 0x7fd078e5a0a0, 0xc208092090, 0x36, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:219 +0x4fc
net/http.(*Client).send(0x119ed40, 0xc20964b520, 0x36, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:142 +0x15b
net/http.(*Client).doFollowingRedirects(0x119ed40, 0xc20964b520, 0xd6d580, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:367 +0xb25
net/http.(*Client).Do(0x119ed40, 0xc20964b520, 0x119efc0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:174 +0xa4
github.com/hashicorp/consul/api.(*Client).doRequest(0xc20aede9c0, 0xc20a3cf360, 0x3, 0xc3a550, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/api.go:287 +0xde
github.com/hashicorp/consul/api.(*Catalog).Nodes(0xc20adf0238, 0xc208d93f80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/catalog.go:100 +0xe2
github.com/hashicorp/consul/watch.func·004(0xc208691a40, 0xc209a83b00, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/funcs.go:94 +0x11c
github.com/hashicorp/consul/watch.(*WatchPlan).Run(0xc208691a40, 0xc209a826c0, 0x12, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/plan.go:48 +0x3fb
github.com/hashicorp/consul/command/agent.func·013(0xc208691a40)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:822 +0x190
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:825 +0xc13
goroutine 434086 [IO wait, 1062 minutes]:
net.(*pollDesc).Wait(0xc20b050680, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20b050680, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20b050620, 0xc20961d000, 0x1000, 0x1000, 0x0, 0x7fd078e59ef8, 0xc2098171e8)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20adf1f48, 0xc20961d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7fd078e5cf30, 0xc20adf1f48, 0xc209744aa8, 0xc20961d000, 0x1000, 0x1000, 0x7fd078e48010, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc208a97e00, 0xc20961d000, 0x1000, 0x1000, 0xc207d20e0a, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc20ab58f00)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc20ab58f00, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc209744a50)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f
goroutine 484526 [select]:
net/http.(*persistConn).roundTrip(0xc2097451e0, 0xc20ac06280, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1082 +0x7ad
net/http.(*Transport).RoundTrip(0xc208092090, 0xc20964b790, 0xc20ae7e2c0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:235 +0x558
net/http.send(0xc20964b790, 0x7fd078e5a0a0, 0xc208092090, 0x39, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:219 +0x4fc
net/http.(*Client).send(0x119ed40, 0xc20964b790, 0x39, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:142 +0x15b
net/http.(*Client).doFollowingRedirects(0x119ed40, 0xc20964b790, 0xd6d580, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:367 +0xb25
net/http.(*Client).Do(0x119ed40, 0xc20964b790, 0x119efc0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:174 +0xa4
github.com/hashicorp/consul/api.(*Client).doRequest(0xc20b0fb860, 0xc20a3cf5e0, 0x3, 0xc209f99ec0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/api.go:287 +0xde
github.com/hashicorp/consul/api.(*Health).State(0xc20adf0270, 0xbdd6c0, 0x3, 0xc20ae7e280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/health.go:121 +0x1c3
github.com/hashicorp/consul/watch.func·006(0xc208691900, 0x99d100, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/funcs.go:157 +0x18c
github.com/hashicorp/consul/watch.(*WatchPlan).Run(0xc208691900, 0xc209a824e0, 0x12, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/plan.go:48 +0x3fb
github.com/hashicorp/consul/command/agent.func·013(0xc208691900)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:822 +0x190
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:825 +0xc13
goroutine 485078 [select]:
github.com/hashicorp/yamux.(*Session).AcceptStream(0xc209ed3cc0, 0xc209629e08, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/yamux/session.go:162 +0x173
github.com/hashicorp/yamux.(*Session).Accept(0xc209ed3cc0, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/yamux/session.go:156 +0x4c
github.com/hashicorp/consul/consul.(*Server).handleMultiplexV2(0xc20800a420, 0x7fd078e5cdc8, 0xc20adf01b8)
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:138 +0x188
github.com/hashicorp/consul/consul.(*Server).handleConn(0xc20800a420, 0x7fd078e5cdc8, 0xc20adf01b8, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:104 +0x6b5
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:57 +0x1bb
goroutine 453433 [IO wait, 855 minutes]:
net.(*pollDesc).Wait(0xc20a69e300, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20a69e300, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20a69e2a0, 0xc2092f7000, 0x1000, 0x1000, 0x0, 0x7fd078e59ef8, 0xc20a558b30)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc2082a38e0, 0xc2092f7000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7fd078e5cf30, 0xc2082a38e0, 0xc208b55de8, 0xc2092f7000, 0x1000, 0x1000, 0x7fd078e48010, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc208925280, 0xc2092f7000, 0x1000, 0x1000, 0xc207fd5c6e, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc20abfd860)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc20abfd860, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc208b55d90)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f
goroutine 453453 [IO wait, 855 minutes]:
net.(*pollDesc).Wait(0xc20b133720, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20b133720, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20b1336c0, 0xc208b62000, 0x1000, 0x1000, 0x0, 0x7fd078e59ef8, 0xc20afa1d90)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20adf1678, 0xc208b62000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7fd078e5cf30, 0xc20adf1678, 0xc20a051398, 0xc208b62000, 0x1000, 0x1000, 0x7fd078e48010, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc20a7a9420, 0xc208b62000, 0x1000, 0x1000, 0xc207d20e80, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc2093adaa0)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc2093adaa0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc20a051340)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f
goroutine 453561 [select]:
github.com/hashicorp/consul/consul.(*Server).blockingRPCOpt(0xc20800a420, 0xc20a3cf3b0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:370 +0x2b6
github.com/hashicorp/consul/consul.(*Server).blockingRPC(0xc20800a420, 0xc20ae7e090, 0xc209b1aa68, 0xc20803a448, 0x1, 0x1, 0xc209f99aa0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:298 +0xdd
github.com/hashicorp/consul/consul.(*Catalog).ListNodes(0xc20803a1d0, 0xc20ae7e080, 0xc209b1aa50, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/consul/catalog_endpoint.go:130 +0x333
reflect.Value.call(0xa8d520, 0xb75380, 0x13, 0xbd0100, 0x4, 0xc208cc3840, 0x3, 0x3, 0x0, 0x0, ...)
/usr/local/go/src/reflect/value.go:419 +0x10e5
reflect.Value.Call(0xa8d520, 0xb75380, 0x13, 0xc208cc3840, 0x3, 0x3, 0x0, 0x0, 0x0)
/usr/local/go/src/reflect/value.go:296 +0xbc
net/rpc.(*service).call(0xc20808cb00, 0xc20808ca80, 0xc20a9d9d28, 0xc2080acc00, 0xc208459440, 0xb3de00, 0xc20ae7e080, 0x16, 0x9869e0, 0xc209b1aa50, ...)
/usr/local/go/src/net/rpc/server.go:382 +0x1f7
net/rpc.(*Server).ServeRequest(0xc20808ca80, 0x7fd078e5e8d0, 0xc20ae7e040, 0x0, 0x0)
/usr/local/go/src/net/rpc/server.go:497 +0x2a0
github.com/hashicorp/consul/consul.(*Server).RPC(0xc20800a420, 0xc3bdd0, 0x11, 0xb3de00, 0xc20ae7e000, 0x9869e0, 0xc209b1aa20, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/consul/server.go:708 +0x13a
github.com/hashicorp/consul/command/agent.(*Agent).RPC(0xc208070500, 0xc3bdd0, 0x11, 0xb3de00, 0xc20ae7e000, 0x9869e0, 0xc209b1aa20, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/agent.go:374 +0x8e
github.com/hashicorp/consul/command/agent.(*HTTPServer).CatalogNodes(0xc208190550, 0x7fd078e5eb70, 0xc20a1f2460, 0xc20964ad00, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/catalog_endpoint.go:69 +0x1ad
github.com/hashicorp/consul/command/agent.*HTTPServer.CatalogNodes·fm(0x7fd078e5eb70, 0xc20a1f2460, 0xc20964ad00, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/http.go:202 +0x69
github.com/hashicorp/consul/command/agent.func·022(0x7fd078e5eb70, 0xc20a1f2460, 0xc20964ad00)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/http.go:290 +0x160
net/http.HandlerFunc.ServeHTTP(0xc208162a20, 0x7fd078e5eb70, 0xc20a1f2460, 0xc20964ad00)
/usr/local/go/src/net/http/server.go:1265 +0x41
net/http.(*ServeMux).ServeHTTP(0xc208172db0, 0x7fd078e5eb70, 0xc20a1f2460, 0xc20964ad00)
/usr/local/go/src/net/http/server.go:1541 +0x17d
net/http.serverHandler.ServeHTTP(0xc208094b40, 0x7fd078e5eb70, 0xc20a1f2460, 0xc20964ad00)
/usr/local/go/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc2091ef220)
/usr/local/go/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:1751 +0x35e
goroutine 485079 [IO wait]:
net.(*pollDesc).Wait(0xc208bc3f70, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208bc3f70, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208bc3f10, 0xc20b1cf000, 0x1000, 0x1000, 0x0, 0x7fd078e59ef8, 0xc20a6dba70)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20adf01b8, 0xc20b1cf000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
bufio.(*Reader).fill(0xc20aedf0e0)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Read(0xc20aedf0e0, 0xc20a6db754, 0xc, 0xc, 0xc20a6dba60, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:174 +0x26c
io.ReadAtLeast(0x7fd078e5c3b8, 0xc20aedf0e0, 0xc20a6db754, 0xc, 0xc, 0xc, 0x0, 0x0, 0x0)
/usr/local/go/src/io/io.go:298 +0xf1
io.ReadFull(0x7fd078e5c3b8, 0xc20aedf0e0, 0xc20a6db754, 0xc, 0xc, 0xc, 0x0, 0x0)
/usr/local/go/src/io/io.go:316 +0x6d
/opt/gopath/src/github.com/hashicorp/yamux/session.go:340 +0xcb
created by github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/yamux/session.go:93 +0x3f0
goroutine 453434 [select, 855 minutes]:
net/http.(*persistConn).writeLoop(0xc208b55d90)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
goroutine 485091 [IO wait]:
net.(*pollDesc).Wait(0xc20a0259c0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20a0259c0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20a025960, 0xc20a255800, 0x200, 0x200, 0x0, 0x7fd078e59ef8, 0xc20ac064a8)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20adf02a0, 0xc20a255800, 0x200, 0x200, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
github.com/miekg/dns.(*Conn).Read(0xc209963b30, 0xc20a255800, 0x200, 0x200, 0x200, 0x0, 0x0)
/opt/gopath/src/github.com/miekg/dns/client.go:230 +0x4a4
github.com/miekg/dns.(*Conn).ReadMsg(0xc209963b30, 0xc20931aea0, 0x0, 0x0)
/opt/gopath/src/github.com/miekg/dns/client.go:175 +0x128
github.com/miekg/dns.(*Client).exchange(0xc209963a90, 0xc20931aea0, 0xc208163300, 0x12, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/miekg/dns/client.go:156 +0x38c
github.com/miekg/dns.(*Client).Exchange(0xc209963a90, 0xc20931aea0, 0xc208163300, 0x12, 0x1, 0x425b29, 0x0, 0x0)
/opt/gopath/src/github.com/miekg/dns/client.go:95 +0x83
created by github.com/miekg/dns.(*Server).Shutdown
/opt/gopath/src/github.com/miekg/dns/server.go:355 +0x4cd
goroutine 453443 [select, 855 minutes]:
net/http.(*persistConn).roundTrip(0xc20a051340, 0xc20afa1930, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1082 +0x7ad
net/http.(*Transport).RoundTrip(0xc208092090, 0xc209d7dee0, 0xc20b1cd350, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:235 +0x558
net/http.send(0xc209d7dee0, 0x7fd078e5a0a0, 0xc208092090, 0x2d, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:219 +0x4fc
net/http.(*Client).send(0x119ed40, 0xc209d7dee0, 0x2d, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:142 +0x15b
net/http.(*Client).doFollowingRedirects(0x119ed40, 0xc209d7dee0, 0xd6d580, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:367 +0xb25
net/http.(*Client).Do(0x119ed40, 0xc209d7dee0, 0x119efc0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:174 +0xa4
github.com/hashicorp/consul/api.(*Client).doRequest(0xc208626360, 0xc209bdd0e0, 0x3, 0xc20a7a8e60, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/api.go:287 +0xde
github.com/hashicorp/consul/api.(*Health).State(0xc20adf10f8, 0xbdd6c0, 0x3, 0xc208e48f00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/health.go:121 +0x1c3
github.com/hashicorp/consul/watch.func·006(0xc208759400, 0x0, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/funcs.go:157 +0x18c
github.com/hashicorp/consul/watch.(*WatchPlan).Run(0xc208759400, 0xc20a7a8e00, 0x12, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/plan.go:48 +0x3fb
github.com/hashicorp/consul/command/agent.func·013(0xc208759400)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:822 +0x190
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:825 +0xc13
goroutine 439951 [IO wait, 856 minutes]:
net.(*pollDesc).Wait(0xc209d94760, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc209d94760, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc209d94700, 0xc2087f0000, 0x1000, 0x1000, 0x0, 0x7fd078e59ef8, 0xc208d50608)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20adf0438, 0xc2087f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7fd078e5cf30, 0xc20adf0438, 0xc20b1d14f8, 0xc2087f0000, 0x1000, 0x1000, 0xa6c0a0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc20905c920, 0xc2087f0000, 0x1000, 0x1000, 0xc208012000, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc208710300)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc208710300, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc20b1d14a0)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f
goroutine 453360 [select, 855 minutes]:
net/http.(*persistConn).roundTrip(0xc209177130, 0xc209d1f9b0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1082 +0x7ad
net/http.(*Transport).RoundTrip(0xc208092090, 0xc20965bad0, 0xc20856b8c0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:235 +0x558
net/http.send(0xc20965bad0, 0x7fd078e5a0a0, 0xc208092090, 0x39, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:219 +0x4fc
net/http.(*Client).send(0x119ed40, 0xc20965bad0, 0x39, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:142 +0x15b
net/http.(*Client).doFollowingRedirects(0x119ed40, 0xc20965bad0, 0xd6d580, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:367 +0xb25
net/http.(*Client).Do(0x119ed40, 0xc20965bad0, 0x119efc0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:174 +0xa4
github.com/hashicorp/consul/api.(*Client).doRequest(0xc209f7e900, 0xc2083ed680, 0x3, 0xc209d1af20, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/api.go:287 +0xde
github.com/hashicorp/consul/api.(*Health).State(0xc2082a3390, 0xbdd6c0, 0x3, 0xc20856b840, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/health.go:121 +0x1c3
github.com/hashicorp/consul/watch.func·006(0xc20990c140, 0x99d100, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/funcs.go:157 +0x18c
github.com/hashicorp/consul/watch.(*WatchPlan).Run(0xc20990c140, 0xc209d03d40, 0x12, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/plan.go:48 +0x3fb
github.com/hashicorp/consul/command/agent.func·013(0xc20990c140)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:822 +0x190
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:825 +0xc13
goroutine 453423 [select, 855 minutes]:
net/http.(*persistConn).roundTrip(0xc208b55d90, 0xc20a5585e0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1082 +0x7ad
net/http.(*Transport).RoundTrip(0xc208092090, 0xc209d6cf70, 0xc208509080, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:235 +0x558
net/http.send(0xc209d6cf70, 0x7fd078e5a0a0, 0xc208092090, 0x2d, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:219 +0x4fc
net/http.(*Client).send(0x119ed40, 0xc209d6cf70, 0x2d, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:142 +0x15b
net/http.(*Client).doFollowingRedirects(0x119ed40, 0xc209d6cf70, 0xd6d580, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:367 +0xb25
net/http.(*Client).Do(0x119ed40, 0xc209d6cf70, 0x119efc0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:174 +0xa4
github.com/hashicorp/consul/api.(*Client).doRequest(0xc20abfc9c0, 0xc20a8a16d0, 0x3, 0xc208924bc0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/api.go:287 +0xde
github.com/hashicorp/consul/api.(*Health).State(0xc2082a3820, 0xbdd6c0, 0x3, 0xc208cefd80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/health.go:121 +0x1c3
github.com/hashicorp/consul/watch.func·006(0xc209ed2a00, 0x0, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/funcs.go:157 +0x18c
github.com/hashicorp/consul/watch.(*WatchPlan).Run(0xc209ed2a00, 0xc208924b00, 0x12, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/plan.go:48 +0x3fb
github.com/hashicorp/consul/command/agent.func·013(0xc209ed2a00)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:822 +0x190
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:825 +0xc13
goroutine 453382 [select, 855 minutes]:
net/http.(*persistConn).writeLoop(0xc209177130)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
goroutine 485082 [select]:
github.com/hashicorp/yamux.(*Stream).Read(0xc209802000, 0xc20a185000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/yamux/stream.go:125 +0x3ab
bufio.(*Reader).fill(0xc20aedf560)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadByte(0xc20aedf560, 0x10, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:196 +0x86
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/decode.go:90 +0x45
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/msgpack.go:540 +0x48
github.com/hashicorp/go-msgpack/codec.(*Decoder).decode(0xc20aedf5c0, 0x993b20, 0xc20a303800)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/decode.go:635 +0x59
github.com/hashicorp/go-msgpack/codec.(*Decoder).Decode(0xc20aedf5c0, 0x993b20, 0xc20a303800, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/decode.go:630 +0x7f
github.com/hashicorp/go-msgpack/codec.(*rpcCodec).read(0xc209b10180, 0x993b20, 0xc20a303800, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/rpc.go:88 +0xd9
github.com/hashicorp/go-msgpack/codec.(*goRpcCodec).ReadRequestHeader(0xc209b10180, 0xc20a303800, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/go-msgpack/codec/rpc.go:127 +0x61
net/rpc.(*Server).readRequestHeader(0xc20808ca80, 0x7fd078e6c6a8, 0xc209b10180, 0x0, 0x0, 0xc20a303800, 0xc20803a100, 0x0, 0x0)
/usr/local/go/src/net/rpc/server.go:575 +0x97
net/rpc.(*Server).readRequest(0xc20808ca80, 0x7fd078e6c6a8, 0xc209b10180, 0xc200000000, 0xc20a303800, 0xb3e6c0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/rpc/server.go:542 +0xa2
net/rpc.(*Server).ServeRequest(0xc20808ca80, 0x7fd078e6c6a8, 0xc209b10180, 0x0, 0x0)
/usr/local/go/src/net/rpc/server.go:485 +0x7f
github.com/hashicorp/consul/consul.(*Server).handleConsulConn(0xc20800a420, 0x7fd078e5f2c0, 0xc209802000)
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:160 +0x160
/opt/gopath/src/github.com/hashicorp/consul/consul/rpc.go:145 +0x31d
goroutine 439960 [select, 856 minutes]:
net/http.(*persistConn).roundTrip(0xc20b1d14a0, 0xc20a206c90, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1082 +0x7ad
net/http.(*Transport).RoundTrip(0xc208092090, 0xc2080c31e0, 0xc208fd8c30, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:235 +0x558
net/http.send(0xc2080c31e0, 0x7fd078e5a0a0, 0xc208092090, 0x42, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:219 +0x4fc
net/http.(*Client).send(0x119ed40, 0xc2080c31e0, 0x42, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:142 +0x15b
net/http.(*Client).doFollowingRedirects(0x119ed40, 0xc2080c31e0, 0xd6d580, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:367 +0xb25
net/http.(*Client).Do(0x119ed40, 0xc2080c31e0, 0x119efc0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:174 +0xa4
github.com/hashicorp/consul/api.(*Client).doRequest(0xc2087d6d80, 0xc208fd8be0, 0xc2082c7ae0, 0xc2082c7b08, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/api.go:287 +0xde
github.com/hashicorp/consul/api.(*KV).getInternal(0xc20adf0ce0, 0xc20a6da730, 0xd, 0xc209707b60, 0xc20ae7fd80, 0xc20ae7fd80, 0xc20ae7fd80, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/kv.go:104 +0x289
github.com/hashicorp/consul/api.(*KV).List(0xc20adf0ce0, 0xc20a6da730, 0xd, 0xc20ae7fd80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/kv.go:59 +0x136
github.com/hashicorp/consul/watch.func·002(0xc209678dc0, 0x9bdd00, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/funcs.go:66 +0x136
github.com/hashicorp/consul/watch.(*WatchPlan).Run(0xc209678dc0, 0xc208b75bc0, 0x12, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/plan.go:48 +0x3fb
github.com/hashicorp/consul/command/agent.func·013(0xc209678dc0)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:822 +0x190
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:825 +0xc13
goroutine 296384 [select, 4052 minutes]:
net/http.(*persistConn).roundTrip(0xc208a911e0, 0xc208db3ed0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1082 +0x7ad
net/http.(*Transport).RoundTrip(0xc208092090, 0xc20a1560d0, 0xc2092f53b0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:235 +0x558
net/http.send(0xc20a1560d0, 0x7fd078e5a0a0, 0xc208092090, 0x42, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:219 +0x4fc
net/http.(*Client).send(0x119ed40, 0xc20a1560d0, 0x42, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:142 +0x15b
net/http.(*Client).doFollowingRedirects(0x119ed40, 0xc20a1560d0, 0xd6d580, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:367 +0xb25
net/http.(*Client).Do(0x119ed40, 0xc20a1560d0, 0x119efc0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:174 +0xa4
github.com/hashicorp/consul/api.(*Client).doRequest(0xc20a33a120, 0xc2092f5360, 0xc209651ae0, 0xc209651b08, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/api.go:287 +0xde
github.com/hashicorp/consul/api.(*KV).getInternal(0xc208ac3fd0, 0xc20918aff0, 0xd, 0xc20882e030, 0xc20a225f80, 0xc20a225f80, 0xc20a225f80, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/kv.go:104 +0x289
github.com/hashicorp/consul/api.(*KV).List(0xc208ac3fd0, 0xc20918aff0, 0xd, 0xc20a225f80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/kv.go:59 +0x136
github.com/hashicorp/consul/watch.func·002(0xc209421360, 0x9bdd00, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/funcs.go:66 +0x136
github.com/hashicorp/consul/watch.(*WatchPlan).Run(0xc209421360, 0xc208e43140, 0x12, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/plan.go:48 +0x3fb
github.com/hashicorp/consul/command/agent.func·013(0xc209421360)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:822 +0x190
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:825 +0xc13
goroutine 470954 [select]:
net/http.(*persistConn).writeLoop(0xc2097451e0)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
goroutine 420626 [select, 1063 minutes]:
net/http.(*persistConn).roundTrip(0xc208a90630, 0xc20884eb20, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1082 +0x7ad
net/http.(*Transport).RoundTrip(0xc208092090, 0xc20815ea90, 0xc2094f4b90, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:235 +0x558
net/http.send(0xc20815ea90, 0x7fd078e5a0a0, 0xc208092090, 0x42, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:219 +0x4fc
net/http.(*Client).send(0x119ed40, 0xc20815ea90, 0x42, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:142 +0x15b
net/http.(*Client).doFollowingRedirects(0x119ed40, 0xc20815ea90, 0xd6d580, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:367 +0xb25
net/http.(*Client).Do(0x119ed40, 0xc20815ea90, 0x119efc0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:174 +0xa4
github.com/hashicorp/consul/api.(*Client).doRequest(0xc20aedea20, 0xc2094f4b40, 0xc20aab7ae0, 0xc20aab7b08, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/api.go:287 +0xde
github.com/hashicorp/consul/api.(*KV).getInternal(0xc20aae9568, 0xc208d51580, 0xd, 0xc208a21c50, 0xc209b11c80, 0xc209b11c80, 0xc209b11c80, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/kv.go:104 +0x289
github.com/hashicorp/consul/api.(*KV).List(0xc20aae9568, 0xc208d51580, 0xd, 0xc209b11c80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/kv.go:59 +0x136
github.com/hashicorp/consul/watch.func·002(0xc20ae2ee60, 0x9bdd00, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/funcs.go:66 +0x136
github.com/hashicorp/consul/watch.(*WatchPlan).Run(0xc20ae2ee60, 0xc208b75640, 0x12, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/plan.go:48 +0x3fb
github.com/hashicorp/consul/command/agent.func·013(0xc20ae2ee60)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:822 +0x190
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:825 +0xc13
goroutine 296406 [IO wait, 4052 minutes]:
net.(*pollDesc).Wait(0xc20a151480, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20a151480, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20a151420, 0xc209de5000, 0x1000, 0x1000, 0x0, 0x7fd078e59ef8, 0xc208db3f78)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc2082a2228, 0xc209de5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7fd078e5cf30, 0xc2082a2228, 0xc208a91238, 0xc209de5000, 0x1000, 0x1000, 0x7fd078e48010, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc208bcf240, 0xc209de5000, 0x1000, 0x1000, 0xc207fd5dc5, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc209de2cc0)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc209de2cc0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc208a911e0)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f
goroutine 420650 [select, 1063 minutes]:
net/http.(*persistConn).writeLoop(0xc208a90630)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
goroutine 406894 [IO wait, 1075 minutes]:
net.(*pollDesc).Wait(0xc20814fc60, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20814fc60, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20814fc00, 0xc2083ba000, 0x1000, 0x1000, 0x0, 0x7fd078e59ef8, 0xc208d17bf8)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc208aeec08, 0xc2083ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7fd078e5cf30, 0xc208aeec08, 0xc2085390d8, 0xc2083ba000, 0x1000, 0x1000, 0x7fd078e48010, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc20a2cc600, 0xc2083ba000, 0x1000, 0x1000, 0xc207f5113e, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc20b0fccc0)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc20b0fccc0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc208539080)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f
goroutine 453381 [IO wait, 855 minutes]:
net.(*pollDesc).Wait(0xc2092a1aa0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2092a1aa0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc2092a1a40, 0xc20a762000, 0x1000, 0x1000, 0x0, 0x7fd078e59ef8, 0xc209d1f140)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc2082a2628, 0xc20a762000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7fd078e5cf30, 0xc2082a2628, 0xc209177188, 0xc20a762000, 0x1000, 0x1000, 0xa6c0a0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc20889a6e0, 0xc20a762000, 0x1000, 0x1000, 0xc208012000, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc209550f00)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc209550f00, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc209177130)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f
goroutine 439952 [select, 856 minutes]:
net/http.(*persistConn).writeLoop(0xc20b1d14a0)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
goroutine 434087 [select, 1062 minutes]:
net/http.(*persistConn).writeLoop(0xc209744a50)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
goroutine 406895 [select, 1075 minutes]:
net/http.(*persistConn).writeLoop(0xc208539080)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
goroutine 470953 [IO wait]:
net.(*pollDesc).Wait(0xc20b133f70, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20b133f70, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20b133f10, 0xc2082bb000, 0x1000, 0x1000, 0x0, 0x7fd078e59ef8, 0xc20a9d98f0)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20a612570, 0xc2082bb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7fd078e5cf30, 0xc20a612570, 0xc209745238, 0xc2082bb000, 0x1000, 0x1000, 0xa6c0a0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc2096cbea0, 0xc2082bb000, 0x1000, 0x1000, 0xc208012000, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc2092c2d20)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc2092c2d20, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc2097451e0)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f
goroutine 485080 [select]:
/opt/gopath/src/github.com/hashicorp/yamux/session.go:298 +0x554
created by github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/yamux/session.go:94 +0x40a
goroutine 315950 [select, 3861 minutes]:
net/http.(*persistConn).roundTrip(0xc208d2cf20, 0xc208647780, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1082 +0x7ad
net/http.(*Transport).RoundTrip(0xc208092090, 0xc209aaaa90, 0xc2092f4c30, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:235 +0x558
net/http.send(0xc209aaaa90, 0x7fd078e5a0a0, 0xc208092090, 0x42, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:219 +0x4fc
net/http.(*Client).send(0x119ed40, 0xc209aaaa90, 0x42, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:142 +0x15b
net/http.(*Client).doFollowingRedirects(0x119ed40, 0xc209aaaa90, 0xd6d580, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:367 +0xb25
net/http.(*Client).Do(0x119ed40, 0xc209aaaa90, 0x119efc0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:174 +0xa4
github.com/hashicorp/consul/api.(*Client).doRequest(0xc20a8623c0, 0xc2092f4b90, 0xc2082c9ae0, 0xc2082c9b08, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/api.go:287 +0xde
github.com/hashicorp/consul/api.(*KV).getInternal(0xc20803bad8, 0xc20a9f6a70, 0xd, 0xc2080c8960, 0xc20a9f1b80, 0xc20a9f1b80, 0xc20a9f1b80, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/kv.go:104 +0x289
github.com/hashicorp/consul/api.(*KV).List(0xc20803bad8, 0xc20a9f6a70, 0xd, 0xc20a9f1b80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/kv.go:59 +0x136
github.com/hashicorp/consul/watch.func·002(0xc20a522aa0, 0x9bdd00, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/funcs.go:66 +0x136
github.com/hashicorp/consul/watch.(*WatchPlan).Run(0xc20a522aa0, 0xc20a693d60, 0x12, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/plan.go:48 +0x3fb
github.com/hashicorp/consul/command/agent.func·013(0xc20a522aa0)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:822 +0x190
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:825 +0xc13
goroutine 453553 [IO wait]:
net.(*pollDesc).Wait(0xc20a8a5100, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20a8a5100, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20a8a50a0, 0xc208be5000, 0x1000, 0x1000, 0x0, 0x7fd078e59ef8, 0xc20a9d9cd0)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20a6122d0, 0xc208be5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7fd078e5cf30, 0xc20a6122d0, 0xc20aa65238, 0xc208be5000, 0x1000, 0x1000, 0xa6c0a0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc20a3021e0, 0xc208be5000, 0x1000, 0x1000, 0xc208012000, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc20b109e60)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc20b109e60, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc20aa651e0)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f
goroutine 420649 [IO wait, 1063 minutes]:
net.(*pollDesc).Wait(0xc209e36d10, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc209e36d10, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc209e36cb0, 0xc209b8c000, 0x1000, 0x1000, 0x0, 0x7fd078e59ef8, 0xc20884ec18)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20aae95d0, 0xc209b8c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7fd078e5cf30, 0xc20aae95d0, 0xc208a90688, 0xc209b8c000, 0x1000, 0x1000, 0x7fd078e48010, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc2096cb520, 0xc209b8c000, 0x1000, 0x1000, 0xc207d516a2, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc208c67920)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc208c67920, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc208a90630)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f
goroutine 434063 [select, 1062 minutes]:
net/http.(*persistConn).roundTrip(0xc209744a50, 0xc2098170e0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1082 +0x7ad
net/http.(*Transport).RoundTrip(0xc208092090, 0xc20acef6c0, 0xc209f17b80, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:235 +0x558
net/http.send(0xc20acef6c0, 0x7fd078e5a0a0, 0xc208092090, 0x42, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:219 +0x4fc
net/http.(*Client).send(0x119ed40, 0xc20acef6c0, 0x42, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:142 +0x15b
net/http.(*Client).doFollowingRedirects(0x119ed40, 0xc20acef6c0, 0xd6d580, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:367 +0xb25
net/http.(*Client).Do(0x119ed40, 0xc20acef6c0, 0x119efc0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:174 +0xa4
github.com/hashicorp/consul/api.(*Client).doRequest(0xc20a8aa3c0, 0xc209f17b30, 0xc20ad21ae0, 0xc20ad21b08, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/api.go:287 +0xde
github.com/hashicorp/consul/api.(*KV).getInternal(0xc20adf1f18, 0xc20884e150, 0xd, 0xc20a88dc20, 0xc20a8efc80, 0xc20a8efc80, 0xc20a8efc80, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/kv.go:104 +0x289
github.com/hashicorp/consul/api.(*KV).List(0xc20adf1f18, 0xc20884e150, 0xd, 0xc20a8efc80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/api/kv.go:59 +0x136
github.com/hashicorp/consul/watch.func·002(0xc209679c20, 0x9bdd00, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/funcs.go:66 +0x136
github.com/hashicorp/consul/watch.(*WatchPlan).Run(0xc209679c20, 0xc20a253280, 0x12, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/plan.go:48 +0x3fb
github.com/hashicorp/consul/command/agent.func·013(0xc209679c20)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:822 +0x190
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:825 +0xc13
goroutine 296407 [select, 4052 minutes]:
net/http.(*persistConn).writeLoop(0xc208a911e0)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
goroutine 315968 [IO wait, 3861 minutes]:
net.(*pollDesc).Wait(0xc2083526f0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2083526f0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208352690, 0xc20863e000, 0x1000, 0x1000, 0x0, 0x7fd078e59ef8, 0xc2086477e8)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20803bb00, 0xc20863e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7fd078e5cf30, 0xc20803bb00, 0xc208d2cf78, 0xc20863e000, 0x1000, 0x1000, 0x7fd078e48010, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc20a1d8980, 0xc20863e000, 0x1000, 0x1000, 0xc207ffc44f, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc20a5989c0)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc20a5989c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc208d2cf20)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f
goroutine 484527 [select]:
github.com/hashicorp/consul/watch.(*WatchPlan).Run(0xc2086919a0, 0xc209a825c0, 0x12, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/consul/watch/plan.go:66 +0x763
github.com/hashicorp/consul/command/agent.func·013(0xc2086919a0)
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:822 +0x190
/opt/gopath/src/github.com/hashicorp/consul/command/agent/command.go:825 +0xc13
goroutine 453454 [select, 855 minutes]:
net/http.(*persistConn).writeLoop(0xc20a051340)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
在 2015年3月23日星期一 UTC+8上午11:10:01，靳禹写道：
Hi all,
靳禹
Mar 23, 2015, 1:24:15 AM3/23/15
to consu...@googlegroups.com
UPDATE!
here the config json I am using
{
"watches": [
{
"type": "checks",
"handler": "/root/health-check-handler.sh"
},
{
"type": "keyprefix",
"prefix": "mongo-master/services/",
"handler": "/usr/local/l7nxg/bin/consul-seneschal/seneschal -v 3 -log_dir /data/log/seneschal/services -rpc-addr 172.31.16.147:8400"
}
]
}
if I use only the checks watch, it works perfectly. As long as I add the later watch, it will mess up everything. With the above config, whenever I use consul reload to trigger the reload, the consul cli keeps sending hangup sig and finally causes `too many open files` error.
here is the log:
==> Caught signal: hangup
==> Reloading configuration...
==> WARNING: Expect Mode enabled, expecting 3 servers
2015/03/23 04:21:45 [INFO] agent.rpc: Accepted client: 172.31.16.147:60979
==> Caught signal: hangup
==> Reloading configuration...
==> WARNING: Expect Mode enabled, expecting 3 servers
2015/03/23 04:21:45 [INFO] agent.rpc: Accepted client: 172.31.16.147:60981
==> Caught signal: hangup
==> Reloading configuration...
==> WARNING: Expect Mode enabled, expecting 3 servers
2015/03/23 04:21:45 [INFO] agent.rpc: Accepted client: 172.31.16.147:60983
==> Caught signal: hangup
==> Reloading configuration...
==> WARNING: Expect Mode enabled, expecting 3 servers
2015/03/23 04:21:45 [INFO] agent.rpc: Accepted client: 172.31.16.147:60985
==> Caught signal: hangup
==> Reloading configuration...
==> WARNING: Expect Mode enabled, expecting 3 servers
2015/03/23 04:21:45 [INFO] agent.rpc: Accepted client: 172.31.16.147:60987
==> Caught signal: hangup
==> Reloading configuration...
==> WARNING: Expect Mode enabled, expecting 3 servers
2015/03/23 04:21:45 [INFO] agent.rpc: Accepted client: 172.31.16.147:60989
==> Caught signal: hangup
==> Reloading configuration...
==> WARNING: Expect Mode enabled, expecting 3 servers
2015/03/23 04:21:45 [INFO] agent.rpc: Accepted client: 172.31.16.147:60991
I think it is not related to the handler, since it has not been called since the consul repeatedly reloading the configurations
在 2015年3月23日星期一 UTC+8上午11:10:01，靳禹写道：
Hi all,
靳禹
Mar 23, 2015, 1:47:49 AM3/23/15
to consu...@googlegroups.com
problem solved... probably it is not a good idea to trigger a reload from a watch handler? What would you recommend to dynamically configure all nodes in a given DC?
在 2015年3月23日星期一 UTC+8上午11:10:01，靳禹写道：
Hi all,
Michael Fischer
Mar 23, 2015, 2:18:15 AM3/23/15
to 靳禹, consu...@googlegroups.com
I'm chuckling a bit inside at your predicament. It seems like a brilliant idea to use Consul's KV store as a very lightweight configuration management tool, until you configure Consul itself with it.
Best regards,
--Michael
Armon Dadgar
Mar 23, 2015, 1:42:19 PM3/23/15
to 靳禹, consu...@googlegroups.com
Hey Jinyu,
I’ve opened a ticket regarding the panic you reported here:
Any information about version, environment, etc would be useful to attach to that ticket.
It should be okay to reload Consul from a handler, but there may be some edge cases that you
are triggering by doing so. Please attach that information to the ticket as well, so that we can
get those resolved.
Thanks!
Best Regards,
Armon Dadgar
--
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Roman Rusakov
Mar 22, 2016, 7:11:09 AM3/22/16
to Consul
Hi all,
This is very interesting problem - how to manage consul configuration from consul.
For example due to Health HTTP endpoint has no method to query more than one service health at once, in order to dynamically generate nginx config we need to watch on new services has been added or removed (with proper Tag) and add or remove /v1/health/service watchers accordingly.
But due to https://github.com/hashicorp/consul/issues/571 - this will fall into infinite loop as well.
I wonder how consul-template is dealing with it in his nested ranges?
Roman Rusakov
Mar 24, 2016, 11:38:16 AM3/24/16
to Consul
>This is very interesting problem - how to manage consul configuration from consul.
If someone is interested in it. It looks like external watcher e.g. watcher that is created from cmdline does not trigger when main consul reloads - so this potentially may be solution to manage consul config from consul.
Thanks,
Roman
Reply all
Reply to author
Forward
0 new messages