Failed to connect to metadata daemon
549 views
Skip to first unread message
Lucas, Sascha
Jul 2, 2015, 7:49:46 AM7/2/15
to gan...@googlegroups.com
Hi,
it is the second time, that I'm seeing following error message (Could not update metadata ...):
# gnt-instance add ... hc2.XXXXXX.de
Thu Jul 2 13:06:37 2015 - INFO: Selected nodes for instance hc2.XXXXXX.de via iallocator hail: node852.XXXXXX.de, node851.XXXXXX.de
...
Thu Jul 2 13:06:43 2015 * running the instance OS create scripts...
Thu Jul 2 13:08:29 2015 * resuming disk sync
Thu Jul 2 13:08:30 2015 Could not update metadata for instance 'hc2.XXXXXX.de': Error while executing backend function: Failed to connect to metadata daemon: /var/run/ganeti/socket/ganeti-metad
Thu Jul 2 13:08:30 2015 * starting instance...
It seems I'm unable to find what meta-daemon is and what it's good for. There seems not much value in the logs: /var/log/ganeti/meta-daemon* contains some entries, but not from today. The socket /var/run/ganeti/socket/ganeti-metad does not exist. There seems no ganeti-metad running, nothing is listening on port 80...
I'm using 2.14 with split-user setup. Do I hit issue 949? Then why I don't get this error every time?
Thanks, Sascha.
Aufsichtsratsvorsitzender: Herbert Vogel
Geschäftsführung: Michael Krüger
Sitz der Gesellschaft: Halle/Saale
Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414
UST-ID-Nr. DE 158253683
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.
it is the second time, that I'm seeing following error message (Could not update metadata ...):
# gnt-instance add ... hc2.XXXXXX.de
Thu Jul 2 13:06:37 2015 - INFO: Selected nodes for instance hc2.XXXXXX.de via iallocator hail: node852.XXXXXX.de, node851.XXXXXX.de
...
Thu Jul 2 13:06:43 2015 * running the instance OS create scripts...
Thu Jul 2 13:08:29 2015 * resuming disk sync
Thu Jul 2 13:08:30 2015 Could not update metadata for instance 'hc2.XXXXXX.de': Error while executing backend function: Failed to connect to metadata daemon: /var/run/ganeti/socket/ganeti-metad
Thu Jul 2 13:08:30 2015 * starting instance...
It seems I'm unable to find what meta-daemon is and what it's good for. There seems not much value in the logs: /var/log/ganeti/meta-daemon* contains some entries, but not from today. The socket /var/run/ganeti/socket/ganeti-metad does not exist. There seems no ganeti-metad running, nothing is listening on port 80...
I'm using 2.14 with split-user setup. Do I hit issue 949? Then why I don't get this error every time?
Thanks, Sascha.
Aufsichtsratsvorsitzender: Herbert Vogel
Geschäftsführung: Michael Krüger
Sitz der Gesellschaft: Halle/Saale
Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414
UST-ID-Nr. DE 158253683
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.
Petr Pudlák
Jul 2, 2015, 8:00:46 AM7/2/15
to gan...@googlegroups.com
Hi Sascha,
the metadata daemon allows instances to retrieve OS parameters from the cluster using a HTTP interface. See this (partly implemented) design:
What you're seeing seems like a bug. If you haven't asked for any of the metadata features, Ganeti shouldn't be trying to use the daemon (and if so, it should start the daemon first). Could you share (perhaps censored and/or just to my email address) the arguments you used when creating the instance? And I'll examine the code to see when such a situation might happen.
Thanks,
Petr
--
Petr Pudlák
Software Engineer

Google Germany GmbH
Dienerstraße 12
80331 München
Geschäftsführer: Graham Law, Christine Elizabeth Flores
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Diese E-Mail ist vertraulich. Wenn Sie nicht der richtige Adressat sind, leiten Sie diese bitte nicht weiter, informieren Sie den Absender und löschen Sie die E-Mail und alle Anhänge. Vielen Dank.
This e-mail is confidential. If you are not the right addressee please do not forward it, please inform the sender, and please erase this e-mail including any attachments. Thanks.
Lucas, Sascha
Jul 2, 2015, 8:23:36 AM7/2/15
to gan...@googlegroups.com
Hi Petr,
> the metadata daemon allows instances to retrieve OS parameters from the cluster using a HTTP interface
Ah, thanks. I heard about this ...
> Could you share (perhaps censored and/or just to my email address) the arguments you used when creating the instance? And I'll examine the code to see when such a situation might happen.
Here is the full command:
# gnt-instance add -t drbd -H lxc -B minmem=128G,maxmem=128G,vcpus=8 -o clone-image+sles11-sp3 --net 0:network=gruen103,ip=X.X.X.X --disk 0:size=15G,name='/:ext3:noatime' --disk 1:size=100G,name='/data:xfs:noatime' --no-name-check --no-ip-check --no-wait-for-sync hc2.XXXXXX.de
The OS interface is a self written shell script (only utilizing libguestfs, nothing more).
HTH. Thanks, Sascha.
> the metadata daemon allows instances to retrieve OS parameters from the cluster using a HTTP interface
> Could you share (perhaps censored and/or just to my email address) the arguments you used when creating the instance? And I'll examine the code to see when such a situation might happen.
# gnt-instance add -t drbd -H lxc -B minmem=128G,maxmem=128G,vcpus=8 -o clone-image+sles11-sp3 --net 0:network=gruen103,ip=X.X.X.X --disk 0:size=15G,name='/:ext3:noatime' --disk 1:size=100G,name='/data:xfs:noatime' --no-name-check --no-ip-check --no-wait-for-sync hc2.XXXXXX.de
The OS interface is a self written shell script (only utilizing libguestfs, nothing more).
HTH. Thanks, Sascha.
Petr Pudlák
Jul 2, 2015, 9:09:24 AM7/2/15
to gan...@googlegroups.com
Hi Sascha,
you wrote that there is no ganeti-metad socket and no ganeti-metad daemon running. Was this the master node, or the node where the instance is running? There should be one on the node where the instance has been started.
Thanks
Petr
Lucas, Sascha
Jul 3, 2015, 2:55:53 AM7/3/15
to gan...@googlegroups.com
Hi Petr,
> you wrote that there is no ganeti-metad socket and no ganeti-metad daemon running. Was this the master node, or the node where the instance is running?
I'm sorry. This was before I realized, for what Ganeti-metad is good for. I thought it is one of this daemons running only on master...
> There should be one on the node where the instance has been started.
Yes, there is one running. The metad-logs indicate activity for the relevant time, but no errors:
/var/log/ganeti/meta-daemon.log:
2015-07-02 13:08:30,421735000000 CEST: ganeti-metad pid=1681/ThreadId 3 NOTICE ganeti-metad daemon startup
2015-07-02 13:08:30,431853000000 CEST: ganeti-metad pid=1681/ThreadId 3 INFO Changing permissions of /var/run/ganeti/socket/ganeti-metad to 600
/var/log/ganeti/meta-daemon-error.log:
[02/Jul/2015:13:08:30 +0200] Server.httpServe: START, binding to [http://0.0.0.0:80/]
/var/log/ganeti/meta-daemon-access.log is empty.
From node daemon I found the relevant logs:
2015-07-02 13:08:30,143: ganeti-noded pid=1667 INFO RunCmd /usr/lib64/ganeti/daemon-util start ganeti-metad
2015-07-02 13:08:30,426: ganeti-noded pid=1667 ERROR Error in RPC call
Traceback (most recent call last):
File "/usr/share/ganeti/2.14.0/ganeti/server/noded.py", line 191, in HandleRequest
result = (True, method(serializer.LoadJson(req.request_body)))
File "/usr/share/ganeti/2.14.0/ganeti/server/noded.py", line 678, in perspective_instance_metadata_modify
return backend.ModifyInstanceMetadata(instance)
File "/usr/share/ganeti/2.14.0/ganeti/backend.py", line 3043, in ModifyInstanceMetadata
raise TimeoutError("Failed to connect to metadata daemon: %s" % err)
TimeoutError: Failed to connect to metadata daemon: /var/run/ganeti/socket/ganeti-metad
2015-07-02 13:08:30,442: ganeti-noded pid=1667 INFO x.x.x.x:50717 POST /instance_metadata_modify HTTP/1.1 200
> you wrote that there is no ganeti-metad socket and no ganeti-metad daemon running. Was this the master node, or the node where the instance is running?
> There should be one on the node where the instance has been started.
/var/log/ganeti/meta-daemon.log:
2015-07-02 13:08:30,421735000000 CEST: ganeti-metad pid=1681/ThreadId 3 NOTICE ganeti-metad daemon startup
2015-07-02 13:08:30,431853000000 CEST: ganeti-metad pid=1681/ThreadId 3 INFO Changing permissions of /var/run/ganeti/socket/ganeti-metad to 600
/var/log/ganeti/meta-daemon-error.log:
[02/Jul/2015:13:08:30 +0200] Server.httpServe: START, binding to [http://0.0.0.0:80/]
/var/log/ganeti/meta-daemon-access.log is empty.
From node daemon I found the relevant logs:
2015-07-02 13:08:30,143: ganeti-noded pid=1667 INFO RunCmd /usr/lib64/ganeti/daemon-util start ganeti-metad
2015-07-02 13:08:30,426: ganeti-noded pid=1667 ERROR Error in RPC call
Traceback (most recent call last):
File "/usr/share/ganeti/2.14.0/ganeti/server/noded.py", line 191, in HandleRequest
result = (True, method(serializer.LoadJson(req.request_body)))
File "/usr/share/ganeti/2.14.0/ganeti/server/noded.py", line 678, in perspective_instance_metadata_modify
return backend.ModifyInstanceMetadata(instance)
File "/usr/share/ganeti/2.14.0/ganeti/backend.py", line 3043, in ModifyInstanceMetadata
raise TimeoutError("Failed to connect to metadata daemon: %s" % err)
TimeoutError: Failed to connect to metadata daemon: /var/run/ganeti/socket/ganeti-metad
2015-07-02 13:08:30,442: ganeti-noded pid=1667 INFO x.x.x.x:50717 POST /instance_metadata_modify HTTP/1.1 200
Petr Pudlák
Jul 3, 2015, 3:54:43 AM7/3/15
to gan...@googlegroups.com
Hi Sascha,
that's actually good news, this problem is a race condition similar to Issue #1115 with WConfd, and I'm already working in a fix.
Thanks for the report and helping to investigate!
Petr
Daniel Howard
Aug 12, 2015, 6:28:39 PM8/12/15
to ganeti
I am also getting this error message inconsistently around metad when creating an instance.
Per my other message, when instances are created:
gnt instance console hangs indefinitely
gnt instance reboot takes minutes to execute
I can fix the above error by evacuating the node (wait, no I can not)
Ganeti from Ubuntu PPA:
0-15:02 djh@ganeti06-23 ~$ aptitude show ganeti
Package: ganeti
State: installed
Automatically installed: no
Version: 2.12.4-1~trusty+1
Priority: extra
Section: admin
Maintainer: Debian Ganeti Team <pkg-gane...@lists.alioth.debian.org>
Architecture: all
Uncompressed Size: 369 k
Depends: ganeti-2.12 (= 2.12.4-1~trusty+1), ganeti-haskell-2.12 (>=
2.12.4-1~trusty+1), ganeti-haskell-2.12 (< 2.12.4-1~trusty+1.1~),
ganeti-htools-2.12 (>= 2.12.4-1~trusty+1), ganeti-htools-2.12 (<
2.12.4-1~trusty+1.1~), adduser, python
Recommends: drbd-utils | drbd8-utils (>= 8.0.7), qemu-kvm |
xen-linux-system-amd64 | xen-linux-system-686-pae,
ganeti-instance-debootstrap, ndisc6
Suggests: ganeti-doc, blktap-dkms, molly-guard
Conflicts: ganeti-htools, ganeti-htools
Breaks: ganeti-haskell, ganeti-haskell, ganeti2 (< 2.7.0-2), ganeti2 (< 2.7.0-2)
Replaces: ganeti-haskell, ganeti-haskell, ganeti-htools, ganeti-htools, ganeti2
(< 2.7.0-2), ganeti2 (< 2.7.0-2)
Provides: ganeti-htools
Description: cluster virtualization manager
Ganeti is a virtual server cluster management tool built on top of the Xen and
KVM virtualization technologies and other free software. It provides an
automated environment to manage highly available virtual machine instances.
It is designed to facilitate cluster management of virtual servers using
commodity hardware, and to provide fast and simple recovery after physical
failures. It can take care of operations such as redundant disk creation and
management, operating system installation (in cooperation with OS-specific
install scripts), startup, shutdown, and failover of instances between physical
systems.
This package installs the complete Ganeti system.
First gnt instance create generates error. Second does not:
0-14:51 djh@ganeti06-23 ~$ sudo gnt-instance add -o debootstrap+default -s 8g -B memory=1G ganeti-test1
Wed Aug 12 14:51:33 2015 - INFO: Resolved given name 'ganeti-test1' to 'ganeti-test1.mtv.qxxxxxxxxd.com'
Wed Aug 12 14:51:37 2015 - INFO: Selected nodes for instance ganeti-test1.mtv.qxxxxxxxxd.com via iallocator hail: ganeti06-24.mtv.qxxxxxxxxd.com, ganeti06-29.mtv.qxxxxxxxxd.com
Wed Aug 12 14:51:38 2015 * creating instance disks...
Wed Aug 12 14:51:44 2015 adding instance ganeti-test1.mtv.qxxxxxxxxd.com to cluster config
Wed Aug 12 14:51:44 2015 adding disks to cluster config
Wed Aug 12 14:51:45 2015 - INFO: Waiting for instance ganeti-test1.mtv.qxxxxxxxxd.com to sync disks
Wed Aug 12 14:51:45 2015 - INFO: - device disk/0: 0.50% done, 4m 31s remaining (estimated)
Wed Aug 12 14:52:46 2015 - INFO: - device disk/0: 57.50% done, 44s remaining (estimated)
Wed Aug 12 14:53:31 2015 - INFO: - device disk/0: 99.70% done, 0s remaining (estimated)
Wed Aug 12 14:53:31 2015 - INFO: - device disk/0: 100.00% done, 0s remaining (estimated)
Wed Aug 12 14:53:31 2015 - INFO: Instance ganeti-test1.mtv.qxxxxxxxxd.com's disks are in sync
Wed Aug 12 14:53:31 2015 - INFO: Waiting for instance ganeti-test1.mtv.qxxxxxxxxd.com to sync disks
Wed Aug 12 14:53:31 2015 - INFO: Instance ganeti-test1.mtv.qxxxxxxxxd.com's disks are in sync
Wed Aug 12 14:53:31 2015 * running the instance OS create scripts...
Wed Aug 12 14:57:01 2015 Could not update metadata for instance 'ganeti-test1.mtv.qxxxxxxxxd.com': Error while executing backend function: Failed to connect to metadata daemon: /var/run/ganeti/socket/ganeti-metad
Wed Aug 12 14:57:01 2015 * starting instance...
1-15:01 djh@ganeti06-23 ~$ sudo service ganeti status
* ganeti-noded is running
* ganeti-wconfd is running
* ganeti-rapi is running
* ganeti-luxid is running
* ganeti-kvmd is running
* ganeti-confd is running
* ganeti-mond is running
0-15:01 djh@ganeti06-23 ~$ sudo gnt-instance remove ganeti-test1
This will remove the volumes of the instance ganeti-test1 (including
mirrors), thus removing all the data of the instance. Continue?
y/[n]/?: y
0-15:03 djh@ganeti06-23 ~$ sudo gnt-instance add -o debootstrap+default -s 8g -B memory=1G ganeti-test1
Wed Aug 12 15:04:34 2015 - INFO: Resolved given name 'ganeti-test1' to 'ganeti-test1.mtv.qxxxxxxxxd.com'
Wed Aug 12 15:04:38 2015 - INFO: Selected nodes for instance ganeti-test1.mtv.qxxxxxxxxd.com via iallocator hail: ganeti06-24.mtv.qxxxxxxxxd.com, ganeti06-29.mtv.qxxxxxxxxd.com
Wed Aug 12 15:04:39 2015 * creating instance disks...
Wed Aug 12 15:04:45 2015 adding instance ganeti-test1.mtv.qxxxxxxxxd.com to cluster config
Wed Aug 12 15:04:45 2015 adding disks to cluster config
Wed Aug 12 15:04:45 2015 - INFO: Waiting for instance ganeti-test1.mtv.qxxxxxxxxd.com to sync disks
Wed Aug 12 15:04:45 2015 - INFO: - device disk/0: 0.60% done, 3m 28s remaining (estimated)
Wed Aug 12 15:05:46 2015 - INFO: - device disk/0: 58.50% done, 41s remaining (estimated)
Wed Aug 12 15:06:28 2015 - INFO: - device disk/0: 98.70% done, 1s remaining (estimated)
Wed Aug 12 15:06:30 2015 - INFO: Instance ganeti-test1.mtv.qxxxxxxxxd.com's disks are in sync
Wed Aug 12 15:06:30 2015 - INFO: Waiting for instance ganeti-test1.mtv.qxxxxxxxxd.com to sync disks
Wed Aug 12 15:06:30 2015 - INFO: Instance ganeti-test1.mtv.qxxxxxxxxd.com's disks are in sync
Wed Aug 12 15:06:30 2015 * running the instance OS create scripts...
Wed Aug 12 15:06:48 2015 * starting instance...
Reboot takes forever:
0-15:11 djh@ganeti06-23 ~$ time sudo gnt-instance reboot ganeti-test1
Waiting for job 797 for ganeti-test1.mtv.qxxxxxxxxd.com ...
real 2m4.438s
user 0m0.140s
sys 0m0.045s
Note this in log:
==> /var/log/ganeti/luxi-daemon.log <==
2015-08-12 15:11:50,223213000000 PDT: ganeti-luxid pid=3261/ThreadId 29649 INFO Scheduler notified of change of job 797
2015-08-12 15:11:50,223430000000 PDT: ganeti-luxid pid=3261/ThreadId 29649 INFO Rereading job 797
2015-08-12 15:11:50,256262000000 PDT: ganeti-luxid pid=3261/ThreadId 29639 INFO Successfully handled WaitForJobChange
==> /var/log/ganeti/wconf-daemon.log <==
2015-08-12 15:11:50,338241000000 PDT: ganeti-wconfd pid=3192/ThreadId 39100 INFO Successfully handled readConfig
==> /var/log/ganeti/node-daemon.log <==
2015-08-12 15:11:50,509: ganeti-noded pid=6862 INFO 10.10.6.123:43712 POST /hooks_runner HTTP/1.1 200
==> /var/log/ganeti/luxi-daemon.log <==
2015-08-12 15:12:19,314933000000 PDT: ganeti-luxid pid=3261/ThreadId 29639 INFO Successfully handled WaitForJobChange
2015-08-12 15:12:48,373565000000 PDT: ganeti-luxid pid=3261/ThreadId 29639 INFO Successfully handled WaitForJobChange
==> /var/log/ganeti/wconf-daemon.log <==
2015-08-12 15:12:50,400820000000 PDT: ganeti-wconfd pid=3192/ThreadId 39100 WARNING Error during message receiving: user error (Timeout in reading a response)
==> /var/log/ganeti/luxi-daemon.log <==
2015-08-12 15:13:17,432088000000 PDT: ganeti-luxid pid=3261/ThreadId 29639 INFO Successfully handled WaitForJobChange
Okay, so ... let us try my work-around: evacuate the node ... dang, no it doesn't work this time! :(
0-15:11 djh@ganeti06-23 ~$ time sudo gnt-instance reboot ganeti-test1
Waiting for job 797 for ganeti-test1.mtv.qxxxxxxxxd.com ...
real 2m4.438s
user 0m0.140s
sys 0m0.045s
0-15:13 djh@ganeti06-23 ~$ sudo gnt-node evacuate -p ganeti06-24.mtv.qxxxxxxxxd.com
Relocate instance(s) ganeti-test1.mtv.qxxxxxxxxd.com from node(s)
ganeti06-24.mtv.qxxxxxxxxd.com?
y/[n]/?: y
Wed Aug 12 15:14:50 2015 - INFO: Evacuating instances from node 'ganeti06-24.mtv.qxxxxxxxxd.com': ganeti-test1.mtv.qxxxxxxxxd.com
Wed Aug 12 15:14:51 2015 - INFO: Instances to be moved: ganeti-test1.mtv.qxxxxxxxxd.com (to ganeti06-29.mtv.qxxxxxxxxd.com, ganeti06-24.mtv.qxxxxxxxxd.com)
Submitted jobs 799
Waiting for job 799 ...
Wed Aug 12 15:14:53 2015 Migrating instance ganeti-test1.mtv.qxxxxxxxxd.com
Wed Aug 12 15:14:53 2015 * checking disk consistency between source and target
Wed Aug 12 15:14:53 2015 * switching node ganeti06-29.mtv.qxxxxxxxxd.com to secondary mode
Wed Aug 12 15:14:54 2015 * changing into standalone mode
Wed Aug 12 15:14:54 2015 * changing disks into dual-master mode
Wed Aug 12 15:14:55 2015 * wait until resync is done
Wed Aug 12 15:14:55 2015 * preparing ganeti06-29.mtv.qxxxxxxxxd.com to accept the instance
Wed Aug 12 15:14:55 2015 * migrating instance to ganeti06-29.mtv.qxxxxxxxxd.com
Wed Aug 12 15:14:55 2015 * starting memory transfer
Wed Aug 12 15:14:57 2015 * memory transfer complete
Wed Aug 12 15:14:57 2015 * switching node ganeti06-24.mtv.qxxxxxxxxd.com to secondary mode
Wed Aug 12 15:14:57 2015 * wait until resync is done
Wed Aug 12 15:14:57 2015 * changing into standalone mode
Wed Aug 12 15:14:58 2015 * changing disks into single-master mode
Wed Aug 12 15:14:58 2015 * wait until resync is done
Wed Aug 12 15:14:58 2015 * done
All instances evacuated successfully.
0-15:14 djh@ganeti06-23 ~$ time sudo gnt-instance reboot ganeti-test1
Waiting for job 800 for ganeti-test1.mtv.qxxxxxxxxd.com ...
real 2m4.565s
user 0m0.160s
sys 0m0.030s
0-15:17 djh@ganeti06-23 ~$ sudo gnt-node evacuate -p ganeti06-29.mtv.qxxxxxxxxd.com
Relocate instance(s) ganeti-test0.mtv.qxxxxxxxxd.com, ganeti-
test1.mtv.qxxxxxxxxd.com from node(s) ganeti06-29.mtv.qxxxxxxxxd.com?
y/[n]/?: y
Wed Aug 12 15:17:16 2015 - INFO: Evacuating instances from node 'ganeti06-29.mtv.qxxxxxxxxd.com': ganeti-test0.mtv.qxxxxxxxxd.com, ganeti-test1.mtv.qxxxxxxxxd.com
Wed Aug 12 15:17:16 2015 - INFO: Instances to be moved: ganeti-test0.mtv.qxxxxxxxxd.com (to ganeti06-23.mtv.qxxxxxxxxd.com, ganeti06-29.mtv.qxxxxxxxxd.com), ganeti-test1.mtv.qxxxxxxxxd.com (to ganeti06-24.mtv.qxxxxxxxxd.com, ganeti06-29.mtv.qxxxxxxxxd.com)
Submitted jobs 803, 804
Waiting for job 803 ...
Wed Aug 12 15:17:18 2015 Migrating instance ganeti-test0.mtv.qxxxxxxxxd.com
Wed Aug 12 15:17:19 2015 * checking disk consistency between source and target
Wed Aug 12 15:17:19 2015 * switching node ganeti06-23.mtv.qxxxxxxxxd.com to secondary mode
Wed Aug 12 15:17:19 2015 * changing into standalone mode
Wed Aug 12 15:17:19 2015 * changing disks into dual-master mode
Wed Aug 12 15:17:20 2015 * wait until resync is done
Wed Aug 12 15:17:21 2015 * preparing ganeti06-23.mtv.qxxxxxxxxd.com to accept the instance
Wed Aug 12 15:17:21 2015 * migrating instance to ganeti06-23.mtv.qxxxxxxxxd.com
Wed Aug 12 15:17:21 2015 * starting memory transfer
Wed Aug 12 15:17:28 2015 * memory transfer complete
Wed Aug 12 15:17:28 2015 * switching node ganeti06-29.mtv.qxxxxxxxxd.com to secondary mode
Wed Aug 12 15:17:28 2015 * wait until resync is done
Wed Aug 12 15:17:29 2015 * changing into standalone mode
Wed Aug 12 15:17:29 2015 * changing disks into single-master mode
Wed Aug 12 15:17:29 2015 * wait until resync is done
Wed Aug 12 15:17:30 2015 * done
Waiting for job 804 ...
Wed Aug 12 15:17:31 2015 Migrating instance ganeti-test1.mtv.qxxxxxxxxd.com
Wed Aug 12 15:17:31 2015 * checking disk consistency between source and target
Wed Aug 12 15:17:31 2015 * switching node ganeti06-24.mtv.qxxxxxxxxd.com to secondary mode
Wed Aug 12 15:17:31 2015 * changing into standalone mode
Wed Aug 12 15:17:32 2015 * changing disks into dual-master mode
Wed Aug 12 15:17:33 2015 * wait until resync is done
Wed Aug 12 15:17:33 2015 * preparing ganeti06-24.mtv.qxxxxxxxxd.com to accept the instance
Wed Aug 12 15:17:33 2015 * migrating instance to ganeti06-24.mtv.qxxxxxxxxd.com
Wed Aug 12 15:17:33 2015 * starting memory transfer
Wed Aug 12 15:17:34 2015 * memory transfer complete
Wed Aug 12 15:17:35 2015 * switching node ganeti06-29.mtv.qxxxxxxxxd.com to secondary mode
Wed Aug 12 15:17:35 2015 * wait until resync is done
Wed Aug 12 15:17:35 2015 * changing into standalone mode
Wed Aug 12 15:17:35 2015 * changing disks into single-master mode
Wed Aug 12 15:17:36 2015 * wait until resync is done
Wed Aug 12 15:17:36 2015 * done
All instances evacuated successfully.
0-15:17 djh@ganeti06-23 ~$ time sudo gnt-instance reboot ganeti-test1
Waiting for job 805 for ganeti-test1.mtv.qxxxxxxxxd.com ...
real 2m4.402s
user 0m0.155s
sys 0m0.037s
I did notice this scrolling by:
==> /var/log/ganeti/jobs.log <==
2015-08-12 15:16:02,684: job-801 pid=8948 INFO Op 1/1: opcode GROUP_VERIFY_DISKS(c9ab6fa9-3cc2-4e07-a4b3-e4e4592acddc) waiting for locks
I suspect that is something else entirely.
Regarding the console thing:
0-15:13 djh@ganeti06-23 ~$ sudo gnt-instance console ganeti-test1
[ ZZzzzZZZZ ]
But note:
==> /var/log/ganeti/commands.log <==
2015-08-12 15:23:03,216: gnt-instance console pid=14878 INFO RunCmd ssh -oEscapeChar=none -oHashKnownHosts=no -oGlobalKnownHostsFile=/var/lib/ganeti/known_hosts -oUserKnownHostsFile=/dev/null -oCheckHostIp=no -oHostKeyAlias=ganeti.mtv.qxxxxxxxxd.com -oBatchMode=yes -oStrictHostKeyChecking=yes -4 -t -t ro...@ganeti06-24.mtv.qxxxxxxxxd.com '/usr/lib/ganeti/tools/kvm-console-wrapper /usr/bin/socat ganeti-test1.mtv.qxxxxxxxxd.com /var/run/ganeti/kvm-hypervisor/ctrl/ganeti-test1.mtv.qxxxxxxxxd.com.monitor STDIO,raw,echo=0,escape=0x1d UNIX-CONNECT:/var/run/ganeti/kvm-hypervisor/ctrl/ganeti-test1.mtv.qxxxxxxxxd.com.serial'
So:
0-15:23 djh@ganeti06-23 ~$ sudo -i
root@ganeti06-23:~# ssh -i .ssh/id_dsa ro...@ganeti06-24.mtv.qxxxxxxxxd.com
Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.13.0-61-generic x86_64)
* Documentation: https://help.ubuntu.com/
Last login: Wed Aug 12 15:23:03 2015 from ganeti06-23.mtv.qxxxxxxxxd.com
root@ganeti06-24:~# /usr/lib/ganeti/tools/kvm-console-wrapper /usr/bin/socat ganeti-test1.mtv.qxxxxxxxxd.com /var/run/ganeti/kvm-hypervisor/ctrl/ganeti-test1.mtv.qxxxxxxxxd.com.monitor STDIO,raw,echo=0,escape=0x1d UNIX-CONNECT:/var/run/ganeti/kvm-hypervisor/ctrl/ganeti-test1.mtv.qxxxxxxxxd.com.serial
Nada.
Per my other message, when instances are created:
gnt instance console hangs indefinitely
gnt instance reboot takes minutes to execute
I can fix the above error by evacuating the node (wait, no I can not)
Ganeti from Ubuntu PPA:
0-15:02 djh@ganeti06-23 ~$ aptitude show ganeti
Package: ganeti
State: installed
Automatically installed: no
Version: 2.12.4-1~trusty+1
Priority: extra
Section: admin
Maintainer: Debian Ganeti Team <pkg-gane...@lists.alioth.debian.org>
Architecture: all
Uncompressed Size: 369 k
Depends: ganeti-2.12 (= 2.12.4-1~trusty+1), ganeti-haskell-2.12 (>=
2.12.4-1~trusty+1), ganeti-haskell-2.12 (< 2.12.4-1~trusty+1.1~),
ganeti-htools-2.12 (>= 2.12.4-1~trusty+1), ganeti-htools-2.12 (<
2.12.4-1~trusty+1.1~), adduser, python
Recommends: drbd-utils | drbd8-utils (>= 8.0.7), qemu-kvm |
xen-linux-system-amd64 | xen-linux-system-686-pae,
ganeti-instance-debootstrap, ndisc6
Suggests: ganeti-doc, blktap-dkms, molly-guard
Conflicts: ganeti-htools, ganeti-htools
Breaks: ganeti-haskell, ganeti-haskell, ganeti2 (< 2.7.0-2), ganeti2 (< 2.7.0-2)
Replaces: ganeti-haskell, ganeti-haskell, ganeti-htools, ganeti-htools, ganeti2
(< 2.7.0-2), ganeti2 (< 2.7.0-2)
Provides: ganeti-htools
Description: cluster virtualization manager
Ganeti is a virtual server cluster management tool built on top of the Xen and
KVM virtualization technologies and other free software. It provides an
automated environment to manage highly available virtual machine instances.
It is designed to facilitate cluster management of virtual servers using
commodity hardware, and to provide fast and simple recovery after physical
failures. It can take care of operations such as redundant disk creation and
management, operating system installation (in cooperation with OS-specific
install scripts), startup, shutdown, and failover of instances between physical
systems.
This package installs the complete Ganeti system.
First gnt instance create generates error. Second does not:
0-14:51 djh@ganeti06-23 ~$ sudo gnt-instance add -o debootstrap+default -s 8g -B memory=1G ganeti-test1
Wed Aug 12 14:51:33 2015 - INFO: Resolved given name 'ganeti-test1' to 'ganeti-test1.mtv.qxxxxxxxxd.com'
Wed Aug 12 14:51:37 2015 - INFO: Selected nodes for instance ganeti-test1.mtv.qxxxxxxxxd.com via iallocator hail: ganeti06-24.mtv.qxxxxxxxxd.com, ganeti06-29.mtv.qxxxxxxxxd.com
Wed Aug 12 14:51:38 2015 * creating instance disks...
Wed Aug 12 14:51:44 2015 adding instance ganeti-test1.mtv.qxxxxxxxxd.com to cluster config
Wed Aug 12 14:51:44 2015 adding disks to cluster config
Wed Aug 12 14:51:45 2015 - INFO: Waiting for instance ganeti-test1.mtv.qxxxxxxxxd.com to sync disks
Wed Aug 12 14:51:45 2015 - INFO: - device disk/0: 0.50% done, 4m 31s remaining (estimated)
Wed Aug 12 14:52:46 2015 - INFO: - device disk/0: 57.50% done, 44s remaining (estimated)
Wed Aug 12 14:53:31 2015 - INFO: - device disk/0: 99.70% done, 0s remaining (estimated)
Wed Aug 12 14:53:31 2015 - INFO: - device disk/0: 100.00% done, 0s remaining (estimated)
Wed Aug 12 14:53:31 2015 - INFO: Instance ganeti-test1.mtv.qxxxxxxxxd.com's disks are in sync
Wed Aug 12 14:53:31 2015 - INFO: Waiting for instance ganeti-test1.mtv.qxxxxxxxxd.com to sync disks
Wed Aug 12 14:53:31 2015 - INFO: Instance ganeti-test1.mtv.qxxxxxxxxd.com's disks are in sync
Wed Aug 12 14:53:31 2015 * running the instance OS create scripts...
Wed Aug 12 14:57:01 2015 Could not update metadata for instance 'ganeti-test1.mtv.qxxxxxxxxd.com': Error while executing backend function: Failed to connect to metadata daemon: /var/run/ganeti/socket/ganeti-metad
Wed Aug 12 14:57:01 2015 * starting instance...
1-15:01 djh@ganeti06-23 ~$ sudo service ganeti status
* ganeti-noded is running
* ganeti-wconfd is running
* ganeti-rapi is running
* ganeti-luxid is running
* ganeti-kvmd is running
* ganeti-confd is running
* ganeti-mond is running
0-15:01 djh@ganeti06-23 ~$ sudo gnt-instance remove ganeti-test1
This will remove the volumes of the instance ganeti-test1 (including
mirrors), thus removing all the data of the instance. Continue?
y/[n]/?: y
0-15:03 djh@ganeti06-23 ~$ sudo gnt-instance add -o debootstrap+default -s 8g -B memory=1G ganeti-test1
Wed Aug 12 15:04:34 2015 - INFO: Resolved given name 'ganeti-test1' to 'ganeti-test1.mtv.qxxxxxxxxd.com'
Wed Aug 12 15:04:38 2015 - INFO: Selected nodes for instance ganeti-test1.mtv.qxxxxxxxxd.com via iallocator hail: ganeti06-24.mtv.qxxxxxxxxd.com, ganeti06-29.mtv.qxxxxxxxxd.com
Wed Aug 12 15:04:39 2015 * creating instance disks...
Wed Aug 12 15:04:45 2015 adding instance ganeti-test1.mtv.qxxxxxxxxd.com to cluster config
Wed Aug 12 15:04:45 2015 adding disks to cluster config
Wed Aug 12 15:04:45 2015 - INFO: Waiting for instance ganeti-test1.mtv.qxxxxxxxxd.com to sync disks
Wed Aug 12 15:04:45 2015 - INFO: - device disk/0: 0.60% done, 3m 28s remaining (estimated)
Wed Aug 12 15:05:46 2015 - INFO: - device disk/0: 58.50% done, 41s remaining (estimated)
Wed Aug 12 15:06:28 2015 - INFO: - device disk/0: 98.70% done, 1s remaining (estimated)
Wed Aug 12 15:06:30 2015 - INFO: Instance ganeti-test1.mtv.qxxxxxxxxd.com's disks are in sync
Wed Aug 12 15:06:30 2015 - INFO: Waiting for instance ganeti-test1.mtv.qxxxxxxxxd.com to sync disks
Wed Aug 12 15:06:30 2015 - INFO: Instance ganeti-test1.mtv.qxxxxxxxxd.com's disks are in sync
Wed Aug 12 15:06:30 2015 * running the instance OS create scripts...
Wed Aug 12 15:06:48 2015 * starting instance...
Reboot takes forever:
0-15:11 djh@ganeti06-23 ~$ time sudo gnt-instance reboot ganeti-test1
Waiting for job 797 for ganeti-test1.mtv.qxxxxxxxxd.com ...
real 2m4.438s
user 0m0.140s
sys 0m0.045s
Note this in log:
==> /var/log/ganeti/luxi-daemon.log <==
2015-08-12 15:11:50,223213000000 PDT: ganeti-luxid pid=3261/ThreadId 29649 INFO Scheduler notified of change of job 797
2015-08-12 15:11:50,223430000000 PDT: ganeti-luxid pid=3261/ThreadId 29649 INFO Rereading job 797
2015-08-12 15:11:50,256262000000 PDT: ganeti-luxid pid=3261/ThreadId 29639 INFO Successfully handled WaitForJobChange
==> /var/log/ganeti/wconf-daemon.log <==
2015-08-12 15:11:50,338241000000 PDT: ganeti-wconfd pid=3192/ThreadId 39100 INFO Successfully handled readConfig
==> /var/log/ganeti/node-daemon.log <==
2015-08-12 15:11:50,509: ganeti-noded pid=6862 INFO 10.10.6.123:43712 POST /hooks_runner HTTP/1.1 200
==> /var/log/ganeti/luxi-daemon.log <==
2015-08-12 15:12:19,314933000000 PDT: ganeti-luxid pid=3261/ThreadId 29639 INFO Successfully handled WaitForJobChange
2015-08-12 15:12:48,373565000000 PDT: ganeti-luxid pid=3261/ThreadId 29639 INFO Successfully handled WaitForJobChange
==> /var/log/ganeti/wconf-daemon.log <==
2015-08-12 15:12:50,400820000000 PDT: ganeti-wconfd pid=3192/ThreadId 39100 WARNING Error during message receiving: user error (Timeout in reading a response)
==> /var/log/ganeti/luxi-daemon.log <==
2015-08-12 15:13:17,432088000000 PDT: ganeti-luxid pid=3261/ThreadId 29639 INFO Successfully handled WaitForJobChange
Okay, so ... let us try my work-around: evacuate the node ... dang, no it doesn't work this time! :(
0-15:11 djh@ganeti06-23 ~$ time sudo gnt-instance reboot ganeti-test1
Waiting for job 797 for ganeti-test1.mtv.qxxxxxxxxd.com ...
real 2m4.438s
user 0m0.140s
sys 0m0.045s
0-15:13 djh@ganeti06-23 ~$ sudo gnt-node evacuate -p ganeti06-24.mtv.qxxxxxxxxd.com
Relocate instance(s) ganeti-test1.mtv.qxxxxxxxxd.com from node(s)
ganeti06-24.mtv.qxxxxxxxxd.com?
y/[n]/?: y
Wed Aug 12 15:14:50 2015 - INFO: Evacuating instances from node 'ganeti06-24.mtv.qxxxxxxxxd.com': ganeti-test1.mtv.qxxxxxxxxd.com
Wed Aug 12 15:14:51 2015 - INFO: Instances to be moved: ganeti-test1.mtv.qxxxxxxxxd.com (to ganeti06-29.mtv.qxxxxxxxxd.com, ganeti06-24.mtv.qxxxxxxxxd.com)
Submitted jobs 799
Waiting for job 799 ...
Wed Aug 12 15:14:53 2015 Migrating instance ganeti-test1.mtv.qxxxxxxxxd.com
Wed Aug 12 15:14:53 2015 * checking disk consistency between source and target
Wed Aug 12 15:14:53 2015 * switching node ganeti06-29.mtv.qxxxxxxxxd.com to secondary mode
Wed Aug 12 15:14:54 2015 * changing into standalone mode
Wed Aug 12 15:14:54 2015 * changing disks into dual-master mode
Wed Aug 12 15:14:55 2015 * wait until resync is done
Wed Aug 12 15:14:55 2015 * preparing ganeti06-29.mtv.qxxxxxxxxd.com to accept the instance
Wed Aug 12 15:14:55 2015 * migrating instance to ganeti06-29.mtv.qxxxxxxxxd.com
Wed Aug 12 15:14:55 2015 * starting memory transfer
Wed Aug 12 15:14:57 2015 * memory transfer complete
Wed Aug 12 15:14:57 2015 * switching node ganeti06-24.mtv.qxxxxxxxxd.com to secondary mode
Wed Aug 12 15:14:57 2015 * wait until resync is done
Wed Aug 12 15:14:57 2015 * changing into standalone mode
Wed Aug 12 15:14:58 2015 * changing disks into single-master mode
Wed Aug 12 15:14:58 2015 * wait until resync is done
Wed Aug 12 15:14:58 2015 * done
All instances evacuated successfully.
0-15:14 djh@ganeti06-23 ~$ time sudo gnt-instance reboot ganeti-test1
Waiting for job 800 for ganeti-test1.mtv.qxxxxxxxxd.com ...
real 2m4.565s
user 0m0.160s
sys 0m0.030s
0-15:17 djh@ganeti06-23 ~$ sudo gnt-node evacuate -p ganeti06-29.mtv.qxxxxxxxxd.com
Relocate instance(s) ganeti-test0.mtv.qxxxxxxxxd.com, ganeti-
test1.mtv.qxxxxxxxxd.com from node(s) ganeti06-29.mtv.qxxxxxxxxd.com?
y/[n]/?: y
Wed Aug 12 15:17:16 2015 - INFO: Evacuating instances from node 'ganeti06-29.mtv.qxxxxxxxxd.com': ganeti-test0.mtv.qxxxxxxxxd.com, ganeti-test1.mtv.qxxxxxxxxd.com
Wed Aug 12 15:17:16 2015 - INFO: Instances to be moved: ganeti-test0.mtv.qxxxxxxxxd.com (to ganeti06-23.mtv.qxxxxxxxxd.com, ganeti06-29.mtv.qxxxxxxxxd.com), ganeti-test1.mtv.qxxxxxxxxd.com (to ganeti06-24.mtv.qxxxxxxxxd.com, ganeti06-29.mtv.qxxxxxxxxd.com)
Submitted jobs 803, 804
Waiting for job 803 ...
Wed Aug 12 15:17:18 2015 Migrating instance ganeti-test0.mtv.qxxxxxxxxd.com
Wed Aug 12 15:17:19 2015 * checking disk consistency between source and target
Wed Aug 12 15:17:19 2015 * switching node ganeti06-23.mtv.qxxxxxxxxd.com to secondary mode
Wed Aug 12 15:17:19 2015 * changing into standalone mode
Wed Aug 12 15:17:19 2015 * changing disks into dual-master mode
Wed Aug 12 15:17:20 2015 * wait until resync is done
Wed Aug 12 15:17:21 2015 * preparing ganeti06-23.mtv.qxxxxxxxxd.com to accept the instance
Wed Aug 12 15:17:21 2015 * migrating instance to ganeti06-23.mtv.qxxxxxxxxd.com
Wed Aug 12 15:17:21 2015 * starting memory transfer
Wed Aug 12 15:17:28 2015 * memory transfer complete
Wed Aug 12 15:17:28 2015 * switching node ganeti06-29.mtv.qxxxxxxxxd.com to secondary mode
Wed Aug 12 15:17:28 2015 * wait until resync is done
Wed Aug 12 15:17:29 2015 * changing into standalone mode
Wed Aug 12 15:17:29 2015 * changing disks into single-master mode
Wed Aug 12 15:17:29 2015 * wait until resync is done
Wed Aug 12 15:17:30 2015 * done
Waiting for job 804 ...
Wed Aug 12 15:17:31 2015 Migrating instance ganeti-test1.mtv.qxxxxxxxxd.com
Wed Aug 12 15:17:31 2015 * checking disk consistency between source and target
Wed Aug 12 15:17:31 2015 * switching node ganeti06-24.mtv.qxxxxxxxxd.com to secondary mode
Wed Aug 12 15:17:31 2015 * changing into standalone mode
Wed Aug 12 15:17:32 2015 * changing disks into dual-master mode
Wed Aug 12 15:17:33 2015 * wait until resync is done
Wed Aug 12 15:17:33 2015 * preparing ganeti06-24.mtv.qxxxxxxxxd.com to accept the instance
Wed Aug 12 15:17:33 2015 * migrating instance to ganeti06-24.mtv.qxxxxxxxxd.com
Wed Aug 12 15:17:33 2015 * starting memory transfer
Wed Aug 12 15:17:34 2015 * memory transfer complete
Wed Aug 12 15:17:35 2015 * switching node ganeti06-29.mtv.qxxxxxxxxd.com to secondary mode
Wed Aug 12 15:17:35 2015 * wait until resync is done
Wed Aug 12 15:17:35 2015 * changing into standalone mode
Wed Aug 12 15:17:35 2015 * changing disks into single-master mode
Wed Aug 12 15:17:36 2015 * wait until resync is done
Wed Aug 12 15:17:36 2015 * done
All instances evacuated successfully.
0-15:17 djh@ganeti06-23 ~$ time sudo gnt-instance reboot ganeti-test1
Waiting for job 805 for ganeti-test1.mtv.qxxxxxxxxd.com ...
real 2m4.402s
user 0m0.155s
sys 0m0.037s
I did notice this scrolling by:
==> /var/log/ganeti/jobs.log <==
2015-08-12 15:16:02,684: job-801 pid=8948 INFO Op 1/1: opcode GROUP_VERIFY_DISKS(c9ab6fa9-3cc2-4e07-a4b3-e4e4592acddc) waiting for locks
I suspect that is something else entirely.
Regarding the console thing:
0-15:13 djh@ganeti06-23 ~$ sudo gnt-instance console ganeti-test1
[ ZZzzzZZZZ ]
But note:
==> /var/log/ganeti/commands.log <==
2015-08-12 15:23:03,216: gnt-instance console pid=14878 INFO RunCmd ssh -oEscapeChar=none -oHashKnownHosts=no -oGlobalKnownHostsFile=/var/lib/ganeti/known_hosts -oUserKnownHostsFile=/dev/null -oCheckHostIp=no -oHostKeyAlias=ganeti.mtv.qxxxxxxxxd.com -oBatchMode=yes -oStrictHostKeyChecking=yes -4 -t -t ro...@ganeti06-24.mtv.qxxxxxxxxd.com '/usr/lib/ganeti/tools/kvm-console-wrapper /usr/bin/socat ganeti-test1.mtv.qxxxxxxxxd.com /var/run/ganeti/kvm-hypervisor/ctrl/ganeti-test1.mtv.qxxxxxxxxd.com.monitor STDIO,raw,echo=0,escape=0x1d UNIX-CONNECT:/var/run/ganeti/kvm-hypervisor/ctrl/ganeti-test1.mtv.qxxxxxxxxd.com.serial'
So:
0-15:23 djh@ganeti06-23 ~$ sudo -i
root@ganeti06-23:~# ssh -i .ssh/id_dsa ro...@ganeti06-24.mtv.qxxxxxxxxd.com
Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.13.0-61-generic x86_64)
* Documentation: https://help.ubuntu.com/
Last login: Wed Aug 12 15:23:03 2015 from ganeti06-23.mtv.qxxxxxxxxd.com
root@ganeti06-24:~# /usr/lib/ganeti/tools/kvm-console-wrapper /usr/bin/socat ganeti-test1.mtv.qxxxxxxxxd.com /var/run/ganeti/kvm-hypervisor/ctrl/ganeti-test1.mtv.qxxxxxxxxd.com.monitor STDIO,raw,echo=0,escape=0x1d UNIX-CONNECT:/var/run/ganeti/kvm-hypervisor/ctrl/ganeti-test1.mtv.qxxxxxxxxd.com.serial
Nada.
Reply all
Reply to author
Forward
0 new messages