Problem with ganeti not detecting file in a path

[Equand]

So I've fought hours on the end and the issue still persists.

After upgrade to Stretch (all updates successful and no issues) we rebooted the nodes. First the issue showed that some vps's won't start due to to not finding the mount point, we emptied the cdrom_image_path and restarted the sshfs (it failed on some nodes due to problems with keys and host unknown, we had to perform some magic on the cluster keys and root ssh keys due to version jump and I assume OpenSSH change).

2 out of 8 systems (same config) return the following error

Wed Jan 10 16:27:12 2018   - ERROR: node vds-1: hypervisor kvm parameter verify failure (source instance system2): Parameter 'cdrom2_image_path' fails validation: not found or not a file or URL (current value: '/mnt/nfs/isos/debian-9.3.0-amd64-netinst.iso'

Wed Jan 10 16:27:12 2018   - ERROR: node vds-2: hypervisor kvm parameter verify failure (source instance system2): Parameter 'cdrom2_image_path' fails validation: not found or not a file or URL (current value: '/mnt/nfs/isos/debian-9.3.0-amd64-netinst.iso'

Any other node doesn't report any issues at all

When I do gnt-instance modify -H cdrom_image_path=/mnt/nfs/isos/win2008r2.iso system2

Hypervisor parameter validation failed on node vds-1: Parameter 'cdrom_image_path' fails validation: not found or not a file or URL (current value: '/mnt/nfs/isos/win2008r2.iso')

Now the /mnt/nfs/isos is actually connecting using SSHFS. And running the command ala gnt-cluster command file /mnt/nfs/isos/win2008r2.iso returns

/mnt/nfs/isos/win2008r2.iso: ISO 9660 CD-ROM filesystem data 'GRMSXVOL_EN_DVD' (bootable)

on all 8 nodes

It is working perfectly on other systems and all /etc/fstab settings are the same. Ganeti version is 2.15 and the cluster is also 2.15

# gnt-cluster version

Software version: 2.15.2

Internode protocol: 2150000

Configuration format: 2150000

OS api version: 20

Export interface: 0

VCS version: (ganeti) version 2.15.2-7+deb9u1

Linux version on all nodes the same 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04) x86_64 GNU/Linux

Python version except for one node with 2.7.13 is 2.7.14+

I don't even know where to look because it works everywhere except master node and second node. It fails only in Ganeti and it is impossible to declutter Ganeti OP code structure with it's tuple magic for me. So if someone could help me debug this issue it would be greatly appreciated.

The level of frustration this nonsense brings in deeply underestimated by me. We cannot use files on 2 out of 8 nodes which have been used there before upgrade. I've restarted ganeti on all nodes and same. I've restarted sshfs (umount and mount, check ps for leftover processes)

When I modify in hypervisor/hv_base.py:

_FILE_OR_URL_CHECK = (lambda x: utils.IsNormAbsPath(x) or utils.IsUrl(x),

                      "must be an absolute normalized path or a URL",

                      lambda x: os.path.isfile(x) or utils.IsUrl(x),

                      "not found or not a file or URL")

to 

_FILE_OR_URL_CHECK = (lambda x:True,

                      "must be an absolute normalized path or a URL",

                      lambda x: True,

                      "not found or not a file or URL")

The issue goes away but this will be the cause of problems in the future. I don't understand why this is not working as intended.

If I run utils.IsNormAbsPath('/mnt/nfs/isos/win2008r2.iso') and os.path.isfile('/mnt/nfs/isos/win2008r2.iso') in python session I get

>>> x = '/mnt/nfs/isos/win2008r2.iso'

>>> ganeti.utils.IsNormAbsPath(x) or ganeti.utils.IsUrl(x), os.path.isfile(x) or  ganeti.utils.IsUrl(x)

(True, True)

Checked ganeti logs but there is nothing useful there, just the text from hv_base.py which doesn't make sense (if I run individual op code checks in python session they all return true except for URL check). Maybe I'm missing something?

[Karsten Heymann]

Hi Equand,

Did you make sure the files can be accessed by the users the ganeti processes run with? 

Karsten

[Equand .]

The paths are absolutely identical with same permissions on all 8 systems. All packages are the same. Weirdest part is that when perform any action triggering the error ganeti doesn't even hit fusefs or sshfs. When i run that in debug there's no action. Doing file operation is working without issue. So it's definitely not in fuse or sshfs. I eliminated caching in fstab on the node so there's no host or kernel cache.

Only difference is that both of 2 first nodes were used with ganeti pre 2.11 and went through upgrades. But i purged 2.11 from packages and there's still no change. I also reinstalled 2.15... I'm completely baffled by the difficulty of debugging this issue.

[Equand]

To add insult to confusion I attached to ganeti-noded via strace -f -p <pid>

and got this

[pid 22141] stat("/mnt/nfs/isos/win2008r2.iso", 0x7ffd44ef9000) = -1 ENOENT (No such file or directory)

While if I do os.stat from system stat command or python I get:

  File: /mnt/nfs/isos/win2008r2.iso

  Size: 3166720000 Blocks: 6185000    IO Block: 4096   regular file

Device: 28h/40d Inode: 18          Links: 1

Access: (0644/-rw-r--r--)  Uid: ( 1004/ UNKNOWN)   Gid: ( 1004/ UNKNOWN)

Access: 2016-08-14 11:51:01.000000000 +0200

Modify: 2016-08-14 12:01:36.000000000 +0200

Change: 2016-08-14 12:01:36.000000000 +0200

 Birth: -

>>> os.stat('/mnt/nfs/isos/win2008r2.iso')

posix.stat_result(st_mode=33188, st_ino=18, st_dev=40, st_nlink=1, st_uid=1004, st_gid=1004, st_size=3166720000, st_atime=1471168261, st_mtime=1471168896, st_ctime=1471168896)

And even more confusing, if we compile a simple statting C program it will do the stat without issue!

stat("/mnt/nfs/isos/win2008r2.iso", {st_mode=S_IFREG|0644, st_size=3166720000, ...}) = 0

There's no chroot on ganeti-noded. We have no chroot or user limitation, it's run as root...

[Ansgar Jazdzewski]

Hi,

do you allow root access on nff?
by dfault this is NOT allowed

Hope it helps,
Ansgar

[Equand .]

Sorry it seems the nfs in path confused you. That's legacy path name from since we used proxmox. We don't use nfs. We use sshfs.

Also ill add that we had a few processes of sshfs running on other systems. After we did umount -f and killall sshfs and mount -a those nodes except for one fail as well. So I'm going to check what's the difference between them as i successfully reproduced it on other similar nodes. Might be the unstable repository we used for some software...

[Equand]

Seems like a terrible problem with one of the /unstable packages which installed by accident on the systems.

Can't downgrade but seems like ganeti is not friends with unstable mount for some reason.

This package util-linux/unstable,now 2.30.2-0.1 amd64 prevents proper stat call in ganeti-noded...

We can't downgrade the server unfortunately- managed to downgrade one but another gets into deadlock of packages.

Currently our main plan to avoid this is to run a new another vanilla system to migrate all users there and run them on a brand new cluster or maybe even not ganeti (due to its apparent complexity, why do you need virtual cpu in a vm management software? liked inception movie? :D)

Thanks anyone for trying to help. Still can't understand how it is possible that one program sees something other can't...

[candlerb]

> run them on a brand new cluster or maybe even not ganeti (due to its apparent complexity, why do you need virtual cpu in a vm management software? liked inception movie? :D)

No disrespect, but can you be specific what you mean by that comment? It appears to make no sense as written.

All hypervisors provide virtual CPUs, and ganeti is nothing more than a front-end to the KVM or Xen hypervisor.  You will find the same with libvirt, e.g.

  <vcpu placement='static'>8</vcpu>

to assign 8 virtual CPUs to a virtual machine.  The number of emulated CPU cores each VM sees is a parameter you can tweak. However, ganeti itself is not a virtual machine; it is just a bunch of python and haskell code which starts and stops kvm/xen instances and manages virtual disks.

That's not to say that some software other than ganeti may be more suitable for your application, and/or that you may find something else easier to use.  I just don't understand what aspect of ganeti leads you to your Inception analogy.

[Equand]

Sorry, I'm talking about mcpu.py and OP_codes in ganeti, not the virtualization. It's funny when it is easier to debug qemu through attaching gdb rather than completely unable to debug ganeti due to inability to use pdb to find out where the error happens. I had to strace the gnt-noded process on the node to find out where the problem happens, still unclear why it happens to ganeti only (all other software sees it perfectly). gnt-debug is completely useless and lacks any documentation. That's why I'm talking about inception. Ganeti is running a pseudo-virtual cpu to process opcodes inside which are processed in a non-transparent way inside the system. There is no direct function call, there are mills and queues which process tuples of precomputed lambdas with functions. There's so much abstraction it's impossible to debug the issue without using low level tools. Errors and logs are completely useless as they don't report which call actually creates the problem and a way to debug it. libvirt is written in C so it makes sense to create a virtual machine inside of it. Ganeti though is written in Python. It doesn't make sense to write a virtual machine inside of a virtual machine of the language interpreter because it makes debugging useless.

Ah it's just a rant. I'm terribly confused by an error where selectively one app stops statting a file while others do it without an issue. Seems to be some kind of a bug I guess.

[Federico Pareschi]

This is a really weird problem and a puzzling issue. I've been reading this thread for the past couple of days and honestly I have no ideas.

>This package util-linux/unstable,now 2.30.2-0.1 amd64 prevents proper stat call in ganeti-noded...

I'm a bit confused by this line, were you able to verify that this was indeed what was stopping ganeti's stat syscall from working properly? Any hint on the reason why?

[Equand]

The only evidence of that is that if I downgrade to stable completely then ganeti is able to stat the file. However if I don't do that completely (they have a deadlock of packages, can't upgrade mount because of systemd and can't upgrade systemd because you need to install a replacement and can't install a replacement because it brings even more unstable package which bring more deadlocks. can't install stable because can't install different versions of packages which both systemd and another init system requires) and only these packages are left then ganeti is unable to stat the file. I don't see any other method rather than risk reboot or basically reimage the hypervisor. Currently we are waiting for upgrade node to consolidate all 8 systems on one which I think is the right answer due to craziness of the situation and required time to fix it all... Frankly I have no idea why and how ganeti is dependent on any package just that the only command I see is failing is the stat call from a child of ganeti-noded on the node where the request is sent.

[candlerb]

> Ganeti is running a pseudo-virtual cpu to process opcodes

Oh I see what you mean.  As I understand it, that's what enables the front-end to be separated from the back-end via a job queue, so the backend can work on tasks asynchronously; and it gives the backend the ability to retry steps on transient errors.

Openstack has a full-blown workflow engine for the same purpose (it's called Mistral); arguably more general-purpose but more complex.  https://docs.openstack.org/mistral/latest/architecture.html

If you find ganeti over-complicated, definitely you should not touch openstack!

It's certainly annoying in Ganeti when sometimes I don't get a decent error message from the front-end when something has gone wrong at the back-end.  These cases should be fixable.

I guess you could also consider mcpu.py as further proof of Greenspun's Tenth Rule :-)

Sorry I can't help you on the sshfs problem, it looks really weird.  If you were running this on CentOS then I would be checking SELinux, but that's not going to be it on Debian.

Since I presume sshfs itself is running in userland, then maybe you could try stracing the sshfs process itself (instead of or as well as the ganeti-noded process), to see if there is a hint as to why sshfs is returning an ENOENT.  Or maybe you'll find sshfs doesn't receive the request at all, which might point the problem being somewhere else.

Cheers,

Brian.

[Equand .]

It doesn't receive the request at all. Not even fuse. I'm utterly confused currently.

I don't think the point of "yeah but that software is more dreadful" is a good one. Shouldn't we thrive for simplicity? It takes 1 second to perform a simple ganeti request and the problem is not in the speed of the language. There are plenty of issues with the current ganeti and because future support by Google is questionable the whole system looks like a future debugging disaster. Stable SSH support is abysmal. TBH i don't even understand why ganeti uses ssh commands instead of paramiko for some of the work. For some reason some requests hit rapi. A lot of why's appear when you look at the codebase...

[Ivan Vučica]

On Friday, January 12, 2018 at 11:49:45 AM UTC, Equand wrote:

The only evidence of that is that if I downgrade to stable completely then ganeti is able to stat the file.

So if you downgrade the *whole system* to stable, stat() works? Not just Ganeti?

 

That means something else might be interfering. Perhaps it pulled in a new libc, or your system accidentally got an isolation system like AppArmor deployed on it? candlerb mentioned SELinux should not be there on Debian -- but maybe it is?

You've tried running Python and a simple C program manually. Just as a safety check, you're 100% sure you ran it under the same user as ganeti-noded -- in your case, under root? 

If so, can you risk stopping ganeti-noded, then temporarily and carefully swapping the ganeti-noded binary with your test C program (or with a symlink to Python in which you just run os.stat() like you did)?

You may have tried all this already, in which case, apologies. :-)

[candlerb]

On Friday, 12 January 2018 13:28:31 UTC, Equand wrote:

It doesn't receive the request at all. Not even fuse. I'm utterly confused currently.

Well, that's good (in some ways).  It means you know the stat() call is hitting the kernel but the kernel is returning ENOENT straight away, which sounds like either it doesn't know that sshfs is mounted at that point, or it is doing some sort of negative caching.  The suggestion that different versions of the mount utils make it behave differently means that maybe it's mounted with some slightly different options.

I'm afraid I don't use userland filesystems, so can't really say any more.  You could always mount your ISO images via a read-only NFS share: NFS has its own problems of course, but generally they are fixable.

 

I don't think the point of "yeah but that software is more dreadful" is a good one. Shouldn't we thrive for simplicity?

"Everything should be made as simple as possible, but no simpler."  -- Einstein (misquoted)

"Anything that can go wrong will go wrong" -- Murphy's Law

There is some irreducible complexity here.  Consider what happens, for example, when the ganeti master (running on node A) needs to change an plain instance (on node B) to a drbd instance (replicated between B and C).  It needs to issue a long sequenced set of operations to the various backends; each step must have completed successfully before the next one completes; and some steps may take an extremely long time (especially waiting for drbd to sync).

You need a full-blown state machine of some sort to handle this properly.  Otherwise when something goes wrong, your chance of either fixing and moving forward or successfully rolling back is minimal.

Furthermore, in a big cluster you may have multiple operations like this going on in parallel.

It takes 1 second to perform a simple ganeti request

*Some* simple requests.  Even a request like "start instance" is not as simple as it might sound at first - ganeti may first need to activate the disks, which in turn is a multi-step process across two nodes to activate drbd.

And even if some simple requests could be handled synchronously, that would mean having two different code paths - one for "quick" requests and one for "slow" requests, where you still keep the state machine for the "slow" requests.  That would increase the complexity, over having a single mechanism to handle everything in a consistent fashion.

 

There are plenty of issues with the current ganeti and because future support by Google is questionable the whole system looks like a future debugging disaster.

I am not going to argue with that, although the plan to get rid of Haskell hopefully improves debuggability.  And the fact that there are no big magic layers like libvirt *could* be argued as an advantage too; ultimately, ganeti daemons directly spawn your kvm processes, and you can trace it all the way through easily.

Stable SSH support is abysmal. TBH i don't even understand why ganeti uses ssh commands instead of paramiko for some of the work.

Quite possibly it's historical - ganeti 1.x dates from a long, long time ago, and maybe paramiko was not an option back then.  It also seems ludicrous to me how much ganeti spawns out to external shell commands to handle things like lvm and drbd - until you look at the kernel ABIs and work out how much more difficult it would be to interact with them directly.

At the end of the day, there are a number of options out there for running a "cluster of VMs", for some sort of definition of "cluster".  Unfortunately, all of them are bad in their own different ways.  Choose the one which you find to be least bad - or you can always start your own :-)