gnt-instance add claims not enough disk on target node while there is available free space
110 views
Skip to first unread message
Azeez Zakariyau
Mar 22, 2017, 6:10:39 AM3/22/17
to ganeti
Hello
I tried to build a ganeti VM from iso and when i ran the gnt-instance add command ganeti reported
failure: prerequisites not met for this operation:
error type: insufficient_resources, error details:
Not enough disk space on target node node2.xxx.xx.xx vg ganeti: required 4096 MiB, available 0 MiB
I tried to list the storage on the nodes with gnt-node list-storage and I got
Node Type Name Size Used Free Allocatable
node2.xx.xxxx.xx lvm-vg ganeti 1.1T 1.1T 0M Y
node4.xx.xxxx.xx lvm-vg ganeti 1.1T 1.1T 0M Y
node1xx.xxxx.xx lvm-vg ganeti 1.1T 1.1T 0M Y
node3.xxx.xxxx.xx lvm-vg ganeti 1.1T 1.1T 0M Y
Basically ganeti reported all space on volume group has been used up,but this is not true as these are new nodes with nothing on them, as shown below.
oot@node2:~# pvs
PV VG Fmt Attr PSize PFree
/dev/sda3 ganeti lvm2 a-- 1.09t 0
root@node2:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/dm-0 6.3G 5.0G 983M 84% /
udev 10M 0 10M 0% /dev
tmpfs 9.5G 9.3M 9.5G 1% /run
tmpfs 24G 80K 24G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 24G 0 24G 0% /sys/fs/cgroup
/dev/sda2 237M 34M 191M 16% /boot
/dev/sda1 511M 132K 511M 1% /boot/efi
/dev/mapper/ganeti-var 2.7G 305M 2.3G 12% /var
/dev/mapper/ganeti-tmp 360M 2.1M 335M 1% /tmp
/dev/mapper/ganeti-home 996G 130M 945G 1% /home
tmpfs 4.8G 16K 4.8G 1% /run/user/118
tmpfs 4.8G 0 4.8G 0% /run/user/0
tmpfs 4.8G 0 4.8G 0% /run/user/1001
I am working on a debian jessie system and had installed the synnefo version of ganeti
Thank you
Azeez Zakariyau
Linux and Network Administrator
Eko-Konnect Research and Education Initiative
e-mail: az...@eko-konnect.org.ng
Tel: +234 13428566
Mob: +234 8132754552
Yiannis Tsiouris
Mar 22, 2017, 6:28:23 AM3/22/17
to gan...@googlegroups.com
Hi Azeez,
On 03/21, Azeez Zakariyau wrote:
> I tried to build a ganeti VM from iso and when i ran the gnt-instance add
> command ganeti reported:
>
> root@node1:~# gnt-instance add -t plain -o noop -s 4G -B
> minmem=256M,maxmem=512M -n node2.xx.xxxx.xx.xx --no-start --no-name-check
> --no-ip-check testdebian1
> Failure: prerequisites not met for this operation:
cannot be created. Keep in mind that Ganeti uses similar commands to determine
the available space usage.
> root@node2:~# df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/dm-0 6.3G 5.0G 983M 84% /
> udev 10M 0 10M 0% /dev
> tmpfs 9.5G 9.3M 9.5G 1% /run
> tmpfs 24G 80K 24G 1% /dev/shm
> tmpfs 5.0M 0 5.0M 0% /run/lock
> tmpfs 24G 0 24G 0% /sys/fs/cgroup
> /dev/sda2 237M 34M 191M 16% /boot
> /dev/sda1 511M 132K 511M 1% /boot/efi
> /dev/mapper/ganeti-var 2.7G 305M 2.3G 12% /var
> /dev/mapper/ganeti-tmp 360M 2.1M 335M 1% /tmp
> /dev/mapper/ganeti-home 996G 130M 945G 1% /home
> tmpfs 4.8G 16K 4.8G 1% /run/user/118
> tmpfs 4.8G 0 4.8G 0% /run/user/0
> tmpfs 4.8G 0 4.8G 0% /run/user/1001
There might be available space in other volumes but not in the VG...
Can you post the output of 'mount' in node2?
Regards,
Yiannis
On 03/21, Azeez Zakariyau wrote:
> I tried to build a ganeti VM from iso and when i ran the gnt-instance add
>
> root@node1:~# gnt-instance add -t plain -o noop -s 4G -B
> minmem=256M,maxmem=512M -n node2.xx.xxxx.xx.xx --no-start --no-name-check
> --no-ip-check testdebian1
> Failure: prerequisites not met for this operation:
> error type: insufficient_resources, error details:
> Not enough disk space on target node node2.xx.xxxx.xx.xx vg ganeti:
> required 4096 MiB, available 0 MiB
>
> I tried to list the storage on the nodes with gnt-node list-storage and I
> got
>
> Node Type Name Size Used Free Allocatable
> node2.xx.xxxx.xx lvm-vg ganeti 1.1T 1.1T 0M Y
> node4.xx.xxxx.xx lvm-vg ganeti 1.1T 1.1T 0M Y
> node1xx.xxxx.xx lvm-vg ganeti 1.1T 1.1T 0M Y
> node3.xxx.xxxx.xx lvm-vg ganeti 1.1T 1.1T 0M Y
>
> Basically ganeti reported all space on volume group has been used up,but
> this is not true as these are new nodes with nothing on them, as shown
> below.
>
> oot@node2:~# pvs
> PV VG Fmt Attr PSize PFree
> /dev/sda3 ganeti lvm2 a-- 1.09t 0
The ganeti VG seems to be full (as reported by pvs command). Thus, the 4 GB disk
>
> I tried to list the storage on the nodes with gnt-node list-storage and I
> got
>
> Node Type Name Size Used Free Allocatable
> node2.xx.xxxx.xx lvm-vg ganeti 1.1T 1.1T 0M Y
> node4.xx.xxxx.xx lvm-vg ganeti 1.1T 1.1T 0M Y
> node1xx.xxxx.xx lvm-vg ganeti 1.1T 1.1T 0M Y
> node3.xxx.xxxx.xx lvm-vg ganeti 1.1T 1.1T 0M Y
>
> Basically ganeti reported all space on volume group has been used up,but
> this is not true as these are new nodes with nothing on them, as shown
> below.
>
> oot@node2:~# pvs
> PV VG Fmt Attr PSize PFree
> /dev/sda3 ganeti lvm2 a-- 1.09t 0
cannot be created. Keep in mind that Ganeti uses similar commands to determine
the available space usage.
> root@node2:~# df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/dm-0 6.3G 5.0G 983M 84% /
> udev 10M 0 10M 0% /dev
> tmpfs 9.5G 9.3M 9.5G 1% /run
> tmpfs 24G 80K 24G 1% /dev/shm
> tmpfs 5.0M 0 5.0M 0% /run/lock
> tmpfs 24G 0 24G 0% /sys/fs/cgroup
> /dev/sda2 237M 34M 191M 16% /boot
> /dev/sda1 511M 132K 511M 1% /boot/efi
> /dev/mapper/ganeti-var 2.7G 305M 2.3G 12% /var
> /dev/mapper/ganeti-tmp 360M 2.1M 335M 1% /tmp
> /dev/mapper/ganeti-home 996G 130M 945G 1% /home
> tmpfs 4.8G 16K 4.8G 1% /run/user/118
> tmpfs 4.8G 0 4.8G 0% /run/user/0
> tmpfs 4.8G 0 4.8G 0% /run/user/1001
Can you post the output of 'mount' in node2?
Regards,
Yiannis
Yiannis Tsiouris
Mar 22, 2017, 6:56:09 AM3/22/17
to gan...@googlegroups.com
Regards,
Yiannis
candlerb
Mar 22, 2017, 7:44:00 AM3/22/17
to ganeti
oot@node2:~# pvsPV VG Fmt Attr PSize PFree/dev/sda3 ganeti lvm2 a-- 1.09t 0
Zero space free in the physical volume which makes up the 'ganeti' volume group. ganeti needs to create a new logical volume for each instance, and there's no unallocated space for it to do so.
root@node2:~# df -h...
/dev/mapper/ganeti-home 996G 130M 945G 1% /home
You made a logical volume of size 996G for your /home directory. This is using most of the space on your 1TB drive.
Therefore you need to shrink your /home logical volume.
Unfortunately, shrinking an ext4 filesystem isn't as easy as growing it. But since there's only 130M there, I suggest tarring it up into your root partition, blasting it away and creating a new smaller logical volume for /home.
Regards,
Brian.
Karsten Heymann
Mar 23, 2017, 4:03:50 AM3/23/17
to gan...@googlegroups.com
Hi,
2017-03-22 12:44 GMT+01:00 candlerb <b.ca...@pobox.com>:
> Unfortunately, shrinking an ext4 filesystem isn't as easy as growing it. But
> since there's only 130M there, I suggest tarring it up into your root
> partition, blasting it away and creating a new smaller logical volume for
> /home.
If you want to shrink it as an exercise, it's actually not that hard.
The biggest problem is that the filesystem cannot be mounted during
shrinking, so you have to do it from a root console or a live boot
system. Afterwards, it's just
resize2fs /dev/xxx/home 1G (or any size larger than what you are
currently using but smaller than what you are planning to allocate for
it)
lvresize -L 2G /dev/xxx/home
resize2fs /dev/xxx/home
I've done this a lot of time succesfully, although pease backup your
data before trying it.
(While looking up the syntax of lvresize, I found that it also has an
"-r"-Option which should remove the need to call resize2fs at all. But
I havn't used that one myself yet.)
+Karsten
2017-03-22 12:44 GMT+01:00 candlerb <b.ca...@pobox.com>:
> Unfortunately, shrinking an ext4 filesystem isn't as easy as growing it. But
> since there's only 130M there, I suggest tarring it up into your root
> partition, blasting it away and creating a new smaller logical volume for
> /home.
The biggest problem is that the filesystem cannot be mounted during
shrinking, so you have to do it from a root console or a live boot
system. Afterwards, it's just
resize2fs /dev/xxx/home 1G (or any size larger than what you are
currently using but smaller than what you are planning to allocate for
it)
lvresize -L 2G /dev/xxx/home
resize2fs /dev/xxx/home
I've done this a lot of time succesfully, although pease backup your
data before trying it.
(While looking up the syntax of lvresize, I found that it also has an
"-r"-Option which should remove the need to call resize2fs at all. But
I havn't used that one myself yet.)
+Karsten
Azeez Zakariyau
Apr 7, 2017, 3:54:29 AM4/7/17
to ganeti
Hi Everyone,
Thank you all for the suggestions and advice, we have decided to do a complete reinstall of debian 8.7 and now a separate partition (782G) has been carved out as a physical volume for ganeti volume group as shown below.
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 92G 3.8G 84G 5% /
udev 10M 0 10M 0% /dev
tmpfs 9.5G 9.3M 9.5G 1% /run
tmpfs 24G 80K 24G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 24G 0 24G 0% /sys/fs/cgroup
/dev/sda3 14G 67M 13G 1% /boot
/dev/sda2 46G 87M 44G 1% /home
/dev/sda4 92G 419M 87G 1% /var
/dev/sda6 4.7G 132K 4.7G 1% /boot/efi
tmpfs 4.8G 16K 4.8G 1% /run/user/118
tmpfs 4.8G 0 4.8G 0% /run/user/1001
root@node2:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 1.1T 0 disk
├─sda1 8:1 0 93.1G 0 part /
├─sda2 8:2 0 46.6G 0 part /home
├─sda3 8:3 0 14G 0 part /boot
├─sda4 8:4 0 93.1G 0 part /var
├─sda5 8:5 0 83.8G 0 part [SWAP]
├─sda6 8:6 0 4.7G 0 part /boot/efi
└─sda7 8:7 0 782.5G 0 part
sr0 11:0 1 1024M 0 rom
However I have encountered another challenge while running a burn-in test after initializing the ganeti cluster. The cluster details and error are shown below:
root@node1:~# gnt-cluster verify
Submitted jobs 569, 570
Waiting for job 569 ...
Fri Apr 7 08:17:05 2017 * Verifying cluster config
Fri Apr 7 08:17:05 2017 * Verifying cluster certificate files
Fri Apr 7 08:17:05 2017 * Verifying hypervisor parameters
Fri Apr 7 08:17:05 2017 * Verifying all nodes belong to an existing group
Waiting for job 570 ...
Fri Apr 7 08:17:05 2017 * Verifying group 'default'
Fri Apr 7 08:17:05 2017 * Gathering data (4 nodes)
Fri Apr 7 08:17:06 2017 * Gathering disk information (4 nodes)
Fri Apr 7 08:17:06 2017 * Verifying configuration file consistency
Fri Apr 7 08:17:06 2017 * Verifying node status
Fri Apr 7 08:17:06 2017 * Verifying instance status
Fri Apr 7 08:17:06 2017 * Verifying orphan volumes
Fri Apr 7 08:17:06 2017 * Verifying N+1 Memory redundancy
Fri Apr 7 08:17:06 2017 * Other Notes
Fri Apr 7 08:17:06 2017 * Hooks Results
oot@node1:~# gnt-cluster version
Software version: 2.10.7
Internode protocol: 2100000
Configuration format: 2100000
OS api version: 20
Export interface: 0
VCS version: (ganeti) version v2.10.3-27-gfcf69d1
I had installed the synnefo (grnet) version of ganeti from the repo http://apt.dev.grnet.gr jessie/. (plan is to deploy synnefo components as VMs in the cluster). The result of the burn-in is shown below:
root@node1:~# /usr/lib/ganeti/tools/burnin -o debootstrap+default -H kvm:kernel_path=/vmlinuz,initrd_path=/initrd.img --disk-size 1024m --no-name-check --no-ip-check burn-in.example.com
- Testing global parameters
- Creating instances
* instance burn-in.example.com
- Setting instance runtime memory
* instance burn-in.example.com
Set memory to 128 MB
- Replacing disks on the same nodes
* instance burn-in.example.com
run replace_on_secondary
run replace_on_primary
- Changing the secondary node
* instance burn-in.example.com
run replace_new_secondary node3.example.com
- Growing disks
* instance burn-in.example.com
increase disk/0 by 128 MB
- Failing over instances
* instance burn-in.example.com
- Migrating instances
* instance burn-in.example.com
migration and migration cleanup
- Exporting and re-importing instances
* instance burn-in.example.com
export to node node3.example.com
remove instance
remove export
- Non-idempotent opcode failed, aborting
- Error detected: opcode buffer follows:
Thu Apr 6 14:14:06 2017 * creating instance disks...
Thu Apr 6 14:14:09 2017 adding instance burn-in.example.com to cluster config
Thu Apr 6 14:14:09 2017 - INFO: Waiting for instance burn-in.example.com to sync disks
Thu Apr 6 14:14:10 2017 - INFO: - device disk/0: 0.80% done, 3m 57s remaining (estimated)
Thu Apr 6 14:15:10 2017 - INFO: Instance burn-in.example.com's disks are in sync
Thu Apr 6 14:15:10 2017 * running the instance OS create scripts...
Thu Apr 6 14:25:33 2017 * starting instance...
Thu Apr 6 14:25:36 2017 Replacing disk(s) 0 for instance 'burn-in.example.com'
Thu Apr 6 14:25:36 2017 Current primary node: node1.example.com
Thu Apr 6 14:25:36 2017 Current seconary node: node2.example.com
Thu Apr 6 14:25:36 2017 STEP 1/6 Check device existence
Thu Apr 6 14:25:36 2017 - INFO: Checking disk/0 on node1.example.com
Thu Apr 6 14:25:37 2017 - INFO: Checking disk/0 on node2.example.com
Thu Apr 6 14:25:37 2017 - INFO: Checking volume groups
Thu Apr 6 14:25:37 2017 STEP 2/6 Check peer consistency
Thu Apr 6 14:25:37 2017 - INFO: Checking disk/0 consistency on node node1.example.com
Thu Apr 6 14:25:37 2017 STEP 3/6 Allocate new storage
Thu Apr 6 14:25:37 2017 - INFO: Adding storage on node2.example.com for disk/0
Thu Apr 6 14:25:38 2017 STEP 4/6 Changing drbd configuration
Thu Apr 6 14:25:38 2017 - INFO: Detaching disk/0 drbd from local storage
Thu Apr 6 14:25:38 2017 - INFO: Renaming the old LVs on the target node
Thu Apr 6 14:25:38 2017 - INFO: Renaming the new LVs on the target node
Thu Apr 6 14:25:39 2017 - INFO: Adding new mirror component on node2.example.com
Thu Apr 6 14:25:39 2017 STEP 5/6 Sync devices
Thu Apr 6 14:25:39 2017 - INFO: Waiting for instance burn-in.example.com to sync disks
Thu Apr 6 14:25:39 2017 - INFO: - device disk/0: 0.80% done, 43m 41s remaining (estimated)
Thu Apr 6 14:26:40 2017 - INFO: - device disk/0: 82.20% done, 8s remaining (estimated)
Thu Apr 6 14:26:48 2017 - INFO: Instance burn-in.example.com's disks are in sync
Thu Apr 6 14:26:48 2017 STEP 6/6 Removing old storage
Thu Apr 6 14:26:48 2017 - INFO: Remove logical volumes for disk/0
Thu Apr 6 14:26:49 2017 Replacing disk(s) 0 for instance 'burn-in.example.com'
Thu Apr 6 14:26:49 2017 Current primary node: node1.example.com
Thu Apr 6 14:26:49 2017 Current seconary node: node2.example.com
Thu Apr 6 14:26:49 2017 STEP 1/6 Check device existence
Thu Apr 6 14:26:49 2017 - INFO: Checking disk/0 on node2.example.com
Thu Apr 6 14:26:49 2017 - INFO: Checking disk/0 on node1.example.com
Thu Apr 6 14:26:50 2017 - INFO: Checking volume groups
Thu Apr 6 14:26:50 2017 STEP 2/6 Check peer consistency
Thu Apr 6 14:26:50 2017 - INFO: Checking disk/0 consistency on node node2.example.com
Thu Apr 6 14:26:50 2017 STEP 3/6 Allocate new storage
Thu Apr 6 14:26:50 2017 - INFO: Adding storage on node1.example.com for disk/0
Thu Apr 6 14:26:51 2017 STEP 4/6 Changing drbd configuration
Thu Apr 6 14:26:51 2017 - INFO: Detaching disk/0 drbd from local storage
Thu Apr 6 14:26:51 2017 - INFO: Renaming the old LVs on the target node
Thu Apr 6 14:26:51 2017 - INFO: Renaming the new LVs on the target node
Thu Apr 6 14:26:52 2017 - INFO: Adding new mirror component on node1.example.com
Thu Apr 6 14:26:52 2017 STEP 5/6 Sync devices
Thu Apr 6 14:26:52 2017 - INFO: Waiting for instance burn-in.example.com to sync disks
Thu Apr 6 14:26:53 2017 - INFO: - device disk/0: 0.80% done, 43m 40s remaining (estimated)
Thu Apr 6 14:27:53 2017 - INFO: Instance burn-in.example.com's disks are in sync
Thu Apr 6 14:27:53 2017 STEP 6/6 Removing old storage
Thu Apr 6 14:27:53 2017 - INFO: Remove logical volumes for disk/0
Thu Apr 6 14:27:54 2017 Replacing disk(s) 0 for instance 'burn-in.example.com'
Thu Apr 6 14:27:54 2017 Current primary node: node1.example.com
Thu Apr 6 14:27:54 2017 Current seconary node: node2.example.com
Thu Apr 6 14:27:54 2017 STEP 1/6 Check device existence
Thu Apr 6 14:27:54 2017 - INFO: Checking disk/0 on node1.example.com
Thu Apr 6 14:27:54 2017 - INFO: Checking volume groups
Thu Apr 6 14:27:54 2017 STEP 2/6 Check peer consistency
Thu Apr 6 14:27:54 2017 - INFO: Checking disk/0 consistency on node node1.example.com
Thu Apr 6 14:27:55 2017 STEP 3/6 Allocate new storage
Thu Apr 6 14:27:55 2017 - INFO: Adding new local storage on node3.example.com for disk/0
Thu Apr 6 14:27:55 2017 STEP 4/6 Changing drbd configuration
Thu Apr 6 14:27:55 2017 - INFO: activating a new drbd on node3.example.com for disk/0
Thu Apr 6 14:27:56 2017 - INFO: Shutting down drbd for disk/0 on old node
Thu Apr 6 14:27:56 2017 - INFO: Detaching primary drbds from the network (=> standalone)
Thu Apr 6 14:27:56 2017 - INFO: Updating instance configuration
Thu Apr 6 14:27:56 2017 - INFO: Attaching primary drbds to new secondary (standalone => connected)
Thu Apr 6 14:27:57 2017 STEP 5/6 Sync devices
Thu Apr 6 14:27:57 2017 - INFO: Waiting for instance burn-in.example.com to sync disks
Thu Apr 6 14:27:57 2017 - INFO: - device disk/0: 1.20% done, 2m 3s remaining (estimated)
Thu Apr 6 14:28:57 2017 - INFO: Instance burn-in.example.com's disks are in sync
Thu Apr 6 14:28:58 2017 STEP 6/6 Removing old storage
Thu Apr 6 14:28:58 2017 - INFO: Remove logical volumes for 0
Thu Apr 6 14:28:59 2017 Growing disk 0 of instance 'burn-in.example.com' by 128M to 1.1G
Thu Apr 6 14:29:00 2017 - INFO: Waiting for instance burn-in.example.com to sync disks
Thu Apr 6 14:29:01 2017 - INFO: - device disk/0: 9.10% done, 14s remaining (estimated)
Thu Apr 6 14:29:15 2017 - INFO: Instance burn-in.example.com's disks are in sync
Thu Apr 6 14:29:16 2017 Failover instance burn-in.example.com
Thu Apr 6 14:29:16 2017 * checking disk consistency between source and target
Thu Apr 6 14:29:16 2017 * shutting down instance on source node
Thu Apr 6 14:31:18 2017 * deactivating the instance's disks on source node
Thu Apr 6 14:31:19 2017 * activating the instance's disks on target node node3.example.com
Thu Apr 6 14:31:20 2017 * starting the instance on the target node node3.example.com
Thu Apr 6 14:31:23 2017 Migrating instance burn-in.example.com
Thu Apr 6 14:31:23 2017 * checking disk consistency between source and target
Thu Apr 6 14:31:23 2017 * switching node node1.example.com to secondary mode
Thu Apr 6 14:31:24 2017 * changing into standalone mode
Thu Apr 6 14:31:24 2017 * changing disks into dual-master mode
Thu Apr 6 14:31:25 2017 * wait until resync is done
Thu Apr 6 14:31:25 2017 * preparing node1.example.com to accept the instance
Thu Apr 6 14:31:26 2017 * migrating instance to node1.example.com
Thu Apr 6 14:31:26 2017 * starting memory transfer
Thu Apr 6 14:31:30 2017 * memory transfer complete
Thu Apr 6 14:31:31 2017 * switching node node3.example.com to secondary mode
Thu Apr 6 14:31:31 2017 * wait until resync is done
Thu Apr 6 14:31:31 2017 * changing into standalone mode
Thu Apr 6 14:31:31 2017 * changing disks into single-master mode
Thu Apr 6 14:31:32 2017 * wait until resync is done
Thu Apr 6 14:31:32 2017 * done
Thu Apr 6 14:31:33 2017 - INFO: Not checking memory on the secondary node as instance will not be started
Thu Apr 6 14:31:33 2017 Migrating instance burn-in.example.com
Thu Apr 6 14:31:33 2017 * checking where the instance actually runs (if this hangs, the hypervisor might be in a bad state)
Thu Apr 6 14:31:33 2017 * instance confirmed to be running on its primary node (node1.example.com)
Thu Apr 6 14:31:33 2017 * switching node node3.example.com to secondary mode
Thu Apr 6 14:31:33 2017 * wait until resync is done
Thu Apr 6 14:31:33 2017 * changing into standalone mode
Thu Apr 6 14:31:34 2017 * changing disks into single-master mode
Thu Apr 6 14:31:34 2017 * wait until resync is done
Thu Apr 6 14:31:35 2017 * done
Thu Apr 6 14:31:36 2017 Shutting down instance burn-in.example.com
Thu Apr 6 14:33:38 2017 Creating a snapshot of disk/0 on node node1.example.com
Thu Apr 6 14:33:38 2017 Starting instance burn-in.example.com
Thu Apr 6 14:33:39 2017 Exporting snapshot/0 from node1.example.com to node3.example.com
Thu Apr 6 14:33:43 2017 snapshot/0 is now listening, starting export
Thu Apr 6 14:33:51 2017 snapshot/0 sent 0M, 0.0 MiB/s
Thu Apr 6 14:33:56 2017 - WARNING: export 'export-disk0-2017-04-06_14_33_44-Zntkkx' on node1.example.com failed: Exited with status 1
Thu Apr 6 14:33:56 2017 snapshot/0 failed to send data: Exited with status 1 (recent output: DUMP: Date of this level 0 dump: Thu Apr 6 14:33:45 2017
DUMP: Dumping /dev/mapper/ganeti-6346d65f--15a9--468a--a7d2--c4b7f7555896.disk0_data.snap-1 (an unlisted file system) to standard output
DUMP: Label: none
DUMP: Writing 10 Kilobyte records
DUMP: mapping (Pass I) [regular files]
DUMP: mapping (Pass II) [directories]
DUMP: estimated 192396 blocks.
DUMP: Volume 1 started with block 1 at: Thu Apr 6 14:33:45 2017
DUMP: dumping (Pass III) [directories]
socat: E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small
dd: dd: error writing 'standard output': Broken pipe
DUMP: Broken pipe
DUMP: The ENTIRE dump is aborted.)
Thu Apr 6 14:33:56 2017 Removing snapshot of disk/0 on node node1.example.com
Thu Apr 6 14:33:56 2017 - WARNING: Aborting import 'import-disk0-2017-04-06_14_33_39-lKCYBL' on 6c87e66e-dbf5-4cd1-9345-9fb62c753014
Thu Apr 6 14:33:57 2017 - WARNING: import 'import-disk0-2017-04-06_14_33_39-lKCYBL' on node3.example.com failed: Exited due to signal 15
Thu Apr 6 14:33:57 2017 snapshot/0 failed to receive data: Exited due to signal 15 (recent output: socat: W exiting on signal 15)
Thu Apr 6 14:33:57 2017 Finalizing export on node3.example.com
Removing instances
* instance burn-in.example.com
Traceback (most recent call last):
File "/usr/lib/ganeti/tools/burnin", line 21, in <module>
sys.exit(main.Main())
File "/usr/share/ganeti/2.10/ganeti/tools/burnin.py", line 1174, in Main
return Burner().BurninCluster()
File "/usr/share/ganeti/2.10/ganeti/tools/burnin.py", line 1113, in BurninCluster
self.BurnImportExport()
File "/usr/share/ganeti/2.10/ganeti/tools/burnin.py", line 283, in wrapper
val = fn(self, *args, **kwargs)
File "/usr/share/ganeti/2.10/ganeti/tools/burnin.py", line 303, in batched
val = fn(self, *args, **kwargs)
File "/usr/share/ganeti/2.10/ganeti/tools/burnin.py", line 815, in BurnImportExport
self.ExecOrQueue(instance, [exp_op, rem_op, imp_op, erem_op])
File "/usr/share/ganeti/2.10/ganeti/tools/burnin.py", line 411, in ExecOrQueue
val = self.ExecOp(self.queue_retry, *ops) # pylint: disable=W0142
File "/usr/share/ganeti/2.10/ganeti/tools/burnin.py", line 403, in ExecOp
return self.MaybeRetry(rval, "opcode", self._ExecOp, *ops)
File "/usr/share/ganeti/2.10/ganeti/tools/burnin.py", line 360, in MaybeRetry
val = fn(*args)
File "/usr/share/ganeti/2.10/ganeti/tools/burnin.py", line 385, in _ExecOp
results = cli.PollJob(job_id, cl=self.cl, feedback_fn=self.Feedback)
File "/usr/share/ganeti/2.10/ganeti/cli.py", line 2291, in PollJob
return GenericPollJob(job_id, _LuxiJobPollCb(cl), reporter)
File "/usr/share/ganeti/2.10/ganeti/cli.py", line 2113, in GenericPollJob
errors.MaybeRaise(msg)
File "/usr/share/ganeti/2.10/ganeti/errors.py", line 510, in MaybeRaise
raise errcls(*args)
ganeti.errors.OpExecError: Export failed, errors in disk export: disk(s) 0
candlerb
Apr 10, 2017, 3:47:15 AM4/10/17
to ganeti
On Friday, 7 April 2017 08:54:29 UTC+1, Azeez Zakariyau wrote:
Thank you all for the suggestions and advice, we have decided to do a complete reinstall of debian 8.7
You probably want to raise your issues on the synnefo mailing list, since you are using the synnefo fork of ganeti.
Currently only wheezy is "supported" for synnefo, but they are working towards making a new jessie-based release; I think they would appreciate feedback for problems with their jessie work-in-progress.
Thu Apr 6 14:33:43 2017 snapshot/0 is now listening, starting exportThu Apr 6 14:33:51 2017 snapshot/0 sent 0M, 0.0 MiB/sThu Apr 6 14:33:56 2017 - WARNING: export 'export-disk0-2017-04-06_14_33_44-Zntkkx' on node1.example.com failed: Exited with status 1
...
DUMP: dumping (Pass III) [directories]socat: E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too smalldd: dd: error writing 'standard output': Broken pipe
I guess the new version of openssl with jessie isn't compatible with the options given by the old version of ganeti you're using, which is exactly the sort of problem the burn-in test is good at finding :-)
A simple google for "ganeti dh key too small" finds this:
And there's a solution here:
... but it involves recompiling Haskell stuff :-(
Personally I think it would be good if the synnefo people could simply jump to ganeti 2.15 and minimise the number of local patches they need to apply, but that's a separate issue.
Regards,
Brian.
candlerb
Apr 10, 2017, 3:48:21 AM4/10/17
to ganeti
Possibly better answer here:
Azeez Zakariyau
Apr 10, 2017, 4:01:44 AM4/10/17
to ganeti
Hi All,
It turns out there must be a generated diffie hellman params file: (https://weakdh.org/ ) because of logjam attack
openssl dhparam -out dhparams.pem 2048
and then added to server.pem on every node:
cat dhparams.pem >> /var/lib/ganeti/server.pem
After adding dh to every node - import/export works fine.
Thank you
azeezza...@gmail.com
Feb 14, 2023, 3:33:27 AM2/14/23
to ganeti
Reply all
Reply to author
Forward
0 new messages