ganeti-2.7.0_rc1: gnt-instance and gnt-node exit with error: socket.error: [Errno 104] Connection reset by peer

[George Diamantopoulos]

Hello all,

I've been getting the following error with the latest ganeti rc (ganeti daemons seem to start fine):

death ~ # gnt-instance list
death ~ # gnt-instance list
Traceback (most recent call last):
  File "/usr/sbin/gnt-instance", line 21, in <module>
    sys.exit(main.Main())
  File "/usr/lib64/python2.7/site-packages/ganeti/client/gnt_instance.py", line 1605, in Main
    env_override=_ENV_OVERRIDE)
  File "/usr/lib64/python2.7/site-packages/ganeti/cli.py", line 2399, in GenericMain
    result = func(options, args)
  File "/usr/lib64/python2.7/site-packages/ganeti/client/gnt_instance.py", line 230, in ListInstances
    force_filter=opts.force_filter)
  File "/usr/lib64/python2.7/site-packages/ganeti/cli.py", line 3065, in GenericList
    response = cl.Query(resource, fields, qfilter)
  File "/usr/lib64/python2.7/site-packages/ganeti/luxi.py", line 544, in Query
    result = self.CallMethod(REQ_QUERY, (what, fields, qfilter))
  File "/usr/lib64/python2.7/site-packages/ganeti/luxi.py", line 471, in CallMethod
    version=constants.LUXI_VERSION)
  File "/usr/lib64/python2.7/site-packages/ganeti/luxi.py", line 383, in CallLuxiMethod
    response_msg = transport_cb(request_msg)
  File "/usr/lib64/python2.7/site-packages/ganeti/luxi.py", line 452, in _SendMethodCall
    return self.transport.Call(data)
  File "/usr/lib64/python2.7/site-packages/ganeti/luxi.py", line 282, in Call
    return self.Recv()
  File "/usr/lib64/python2.7/site-packages/ganeti/luxi.py", line 260, in Recv
    data = self.socket.recv(4096)
socket.error: [Errno 104] Connection reset by peer

Note that the first time I ran the tool, I got no output, whereas the second time I got this traceback. Anyone knows where to start with this?

[Guido Trotter]

On Tue, May 21, 2013 at 3:05 PM, George Diamantopoulos <georg...@gmail.com> wrote:

Hello all,

I've been getting the following error with the latest ganeti rc (ganeti daemons seem to start fine):

Do you have split queries enabled? If so could you check your conf-daemon.log file for any hint/information you may find there?

Thanks,

Guido

[George Diamantopoulos]

Hello Guido,

I haven't explicitly set --enable-split-query, so unless it is on by default for RC1, it shouldn't be used.

conf-daemon.log doesn't seem to contain anything useful either:

2013-05-21 17:02:48,876492000000 EEST: ganeti-confd pid=23258 NOTICE ganeti-confd daemon startup
2013-05-21 17:02:48,884862000000 EEST: ganeti-confd pid=23258 INFO Loaded new config, serial 137
2013-05-21 17:02:48,885142000000 EEST: ganeti-confd pid=23258 INFO Starting up in inotify mode
2013-05-21 17:13:52,444992000000 EEST: ganeti-confd pid=26961 NOTICE ganeti-confd daemon startup
2013-05-21 17:13:52,452924000000 EEST: ganeti-confd pid=26961 INFO Loaded new config, serial 137
2013-05-21 17:13:52,453205000000 EEST: ganeti-confd pid=26961 INFO Starting up in inotify mode

the following is reported in master-daemon.log though, which seems relevant:

2013-05-21 17:18:28,748: ganeti-masterd pid=27134/MainThread INFO ganeti-masterd daemon startup
2013-05-21 17:18:28,750: ganeti-masterd pid=27134/MainThread INFO Using PycURL libcurl/7.30.0 OpenSSL/1.0.1e zlib/1.2.8
2013-05-21 17:18:28,768: ganeti-masterd pid=27134/MainThread INFO Inspecting job queue
2013-05-21 17:18:28,768: ganeti-masterd pid=27134/MainThread INFO Job queue inspection: 0/1 (50.0 %)
2013-05-21 17:18:28,769: ganeti-masterd pid=27134/MainThread INFO Job queue inspection: 1/1 (100.0 %)
2013-05-21 17:18:28,769: ganeti-masterd pid=27134/MainThread INFO Job queue inspection finished
2013-05-21 17:18:38,295: ganeti-masterd pid=27134/MainThread ERROR Error while handling asyncore request

Traceback (most recent call last):

  File "/usr/lib64/python2.7/asyncore.py", line 108, in readwrite
    obj.handle_read_event()
  File "/usr/lib64/python2.7/asyncore.py", line 443, in handle_read_event
    self.handle_accept()
  File "/usr/lib64/python2.7/site-packages/ganeti/daemon.py", line 176, in handle_accept
    client_address = netutils.GetSocketCredentials(connected_socket)
  File "/usr/lib64/python2.7/site-packages/ganeti/netutils.py", line 96, in GetSocketCredentials
    peercred = sock.getsockopt(socket.SOL_SOCKET, IN.SO_PEERCRED,
AttributeError: 'module' object has no attribute 'SO_PEERCRED'


Thanks,

George

[Guido Trotter]

On Tue, May 21, 2013 at 4:20 PM, George Diamantopoulos
<georg...@gmail.com> wrote:
> Hello Guido,
>
> I haven't explicitly set --enable-split-query, so unless it is on by default
> for RC1, it shouldn't be used.
>

This is used by default, but only if the required libraries are
present when ./configure is run.

Indeed, this is the problem. Which distribution do you run?
See https://code.google.com/p/ganeti/issues/detail?id=191

Thanks,

Guido

[George Diamantopoulos]

I'm on Gentoo. Any workarounds for this? I'm running linux-3.9.2, python 2.7.4...

Replacing "SO_PEERCRED" with "17" didn't work for me...

[Guido Trotter]

On Tue, May 21, 2013 at 4:45 PM, George Diamantopoulos <georg...@gmail.com> wrote:

I'm on Gentoo. Any workarounds for this? I'm running linux-3.9.2, python 2.7.4...

Replacing "SO_PEERCRED" with "17" didn't work for me...

What happened for you in this case? Do you have a different stack trace?

(did you restart Ganeti daemons after making the change?)

Thanks,

Guido