Ganeti Future

[Redfish Bluefish]

Hi all,

I've a small mostly test ganeti system up and running, and I'm considering reinstalling.

Can anyone say what the development state is of this really rather excellent project? The git repo is not frenetic, and this appears to be the only active discussion area (at least, that I've found).
I'm happy to use 2.15 (which I believe is the current stable), but my queries include:
* Is this still used by Google internally?
* Is there a known roadmap - I'm thinking of release timetable for 2.16/7 etc?
* Do any of you have any opinion on continuing down this road?

I selected Ganeti before based on relative ease for the feature set. It's a beautiful project, and its shortcomings are easily overtaken by its benefits.
I'm just left wondering if now would be a good time to jump to the docker type of ship - all I'm after is a way to run a bunch of machines for light web and system services (LDAP. DHCP etc).

Cheers,

Tom.

[candlerb]

You're right: there has been no release since 2.15.2 in December 2015.  On the plus side, this means the latest released Ganeti version is in the Ubuntu 16.04 standard repositories :-)

Ganeti and Docker address different use cases. Ganeti is for long-lived VMs, and is hypervisor based; Docker is more for application delivery, and is container based (cgroups/namespaces).

Hypervisors are able to run different OSes as guests (think: Windows, FreeBSD etc), and offer strong isolation.  Containers are Linux only, and offer less strong isolation: I would not run applications from untrusted sources in containers.

Ganeti also offers the disk-level replication between multiple nodes without a SAN, which is in some ways its killer feature.

I still use Ganeti because I like its design and am familiar with the CLI, and I need to run full VMs - although inside some of those VMs I run lxd and docker.

However, the development of Ganeti does indeed appear to have ground to a halt, and the community is quite small.  Therefore, if your needs can be met by Docker alone then I'd suggest you use it.  Docker definitely isn't going away, and there is a good choice of platforms for managing and deploying containers across multiple hosts (rancher.com is definitely worth a look)

HTH,

Brian.

[Daniel Howard]

Hello,

No Ganeticon this year?

https://sites.google.com/site/ganeticon/

At the 2016 Ganeticon, I learned that Google has 2 full time developers dedicated to Ganeti, and a lot of internal use. They had migrated off of DRBD backends to hosting VMs on reliable filers via NFS. The roadmap included notions to move the config data into etcd, as that is more scalable, and to have the nodes dynamically rebalance instances on a cluster in response to resource use. The Google devs also made the very big (for Google) move of migrating from their google repos to github.

One challenge for Ganeti, or for anything at Google, is the developers rotate projects every few years. At the 2016 conference, the previous generation of Ganeti developers had shown up.

Ganeti is also weird in that it is super useful but nobody seems to use it, outside  of Greece. It feels like I'm the only user in Silicon Valley, (besides Google,) for instance. But it does a great job for us.

-danny

--

http://dannyman.toldme.com

[micah]

Daniel Howard <dann...@toldme.com> writes:

> Ganeti is also weird in that it is super useful but nobody seems to use it,
> outside of Greece. It feels like I'm the only user in Silicon Valley,
> (besides Google,) for instance. But it does a great job for us.

Maybe not everyone is public about who uses it. I know an organization
in Canada that uses it, and another in the US that uses it, neither is
in Greece or Silicon Valley. :)

[Jake Anderson]

I've been using it in Australia though not for a while (went physical).

Perhaps some sort of default on "popularity contest" or something might
be useful? put it in big letters that it's here and how to turn it off.

I wouldn't object too strongly if ganeti phoned home every few days/week
from each node perhaps with some basic info along the way (# CPU's, ram,
total VMs managed, perhaps system load? I dunno anything useful I guess).

I wouldn't want any identifying info collected, no hostnames or anything
like that

I see it as showing, hey there are 40k users in the world (or whatever)
they are all just super quiet because for the most part everything works
so we should keep this project running.

[sascha...@web.de]

Hi Ganeti community,

I think every body on this list is concerned about the future of ganeti. I'm
glade that someone else started this question, because I felt too busy ...

We all know that ganeti is underestimated, not widespread and does a very
well job. Ganeti has done 10 years ago, what we call hyper-converged today
(with DRBD). And even 10 years later there is no competitive alternative to
manage VMs on DRBD, isn't it?

So what are the cons?

* Development seems stopped. No further releases and with that no further
bugfixes for distributions (i.e. debian stretch or upcoming ubuntu 18.04).
v2.15 has tons of fixes in heads/stable-2.15 not released yet, some of them
seriously (i.e. "Fix failover in case the source node is offline").

* google might decide to drop this project and jump onto another hype. This
is what google has done with other big (internet facing) projects. The only
safeguard seems to be, that google depends on ganeti for internal things.
But nobody knows.

* the user base seems very small. Greece with the famous synnefo, Germany
about 5 installations I know. A view others around the globe.

* the burden of haskell makes contributions very hard

And what are the pros?

* ganeti comes with debian/ubuntu. I've heard, that ganeti is used within
debian infra, so at least there are good people doing packaging and
bugfixing.

* regardless of a ganeti release it may be valid to say that with ubuntu
18.04 there are at least 5 years you can count on ganeti within
functionality of 2.15. Who knows how far containers/distributions are in
2023?

* And finally there are we, the ganeti community. Having the chance to make
ganeti more known/comfortable to others.

So my point of view is:
* use it as it is
* the next years will show, where to go
* try to spend some time spreading ganeti

Thanks, Sascha.

[Randy Bush]

i will confess to helping manage clusters in seattle, ashburn, dallas,
and one on wheels that ships around.

but to me, the issue is not if it is used, as it is obvious. it is how
it goes forward or are we doomed to vmware?

[Ivan Rossi]

Running a few clusters, both internally and on site to a few clients in Italy and Germany.

Share the same worries as you all, and , unless I am mistaken, none of the Google people commented in this thread yet. 

Hope to see some comments from them in the next days...

[__ __]

We are running two clusters with about 10 nodes at a university in germany. Ganeti works great since some years but we are also a little concerned about future development.

[Gianfilippo]

Just a "me-too" post - we employ ganeti for production service VMs on a
10 nodes cluster backed by RBD . Having a better developed RBD backend
would be very useful for us (snapshots?)
Cheers

Il 07/11/2017 10:11, __ __ ha scritto:
> We are running two clusters with about 10 nodes at a university in
> germany. Ganeti works great since some years but we are also a little
> concerned about future development.
>
> On Tue, Nov 7, 2017 at 9:51 AM, Ivan Rossi <roug...@gmail.com
> <mailto:roug...@gmail.com>> wrote:
>
> Running a few clusters, both internally and on site to a few clients
> in Italy and Germany.
> Share the same worries as you all, and , unless I am mistaken, none
> of the Google people commented in this thread yet.
> Hope to see some comments from them in the next days...
>
> 2017-11-06 23:52 GMT+01:00 Randy Bush <ra...@psg.com

> <mailto:ra...@psg.com>>:

[candlerb]

> Google ... had migrated off of DRBD backends to hosting VMs on reliable filers via NFS

That's a useful data point - thank you.  Good to know where the focus is.

If you decide to run with shared storage backend, then the job of the front-end is mainly just starting and stopping the VMs.  There are a bunch of options for this, some free (e.g. oVirt, archipel), and some non-free or semi-free (e.g. Proxmox).  We definitely don't need to go to VMware :-)

However I do like running with local disk, without the SPOF of a shared file store.

In my ideal world, each VM image would be stored on its own separate ZFS filesystem, which was replicated periodically to one or more backup locations, using something like syncoid.

> The roadmap included notions to move the config data into etcd

That would be very nice.  The built-in config store has been a source of bugs in the past; being able to bolt onto an existing etcd cluster would be attractive.

[Philip Hansen]

Hey everyone,

I happen to be subscribed to this list, although I'm not actually involved with Ganeti, so I cannot say anything on behalf of Google in direct context to the project.

What I can say is, we still get these emails, and we are seeing them.

I know a colleague who is actually involved with the project has seen your messages, and is working on a response to this.

Just so you all know you're not shouting into the void :)

Philip Hansen | IT Resident | philip...@google.com | +353 1 519 9980

[Sylvain Costard]

We are testing ganeti cluster here in france and this cluster should be in prodcutino in 2018. I 've started to make a php/laravel GUI under ganeti's API.

--

Sylvain Costard
Administrateur Système Unix
D.S.I - Pôle Infrastructures - Domaine unix
02.99.14.13.47

--

Sylvain Costard
Administrateur Système Unix
D.S.I - Pôle Infrastructures - Domaine unix
02.99.14.13.47

[Ansgar Jazdzewski]

Hi *,

we had just build a new cluster in Germany based on ubuntu16.04 and ganeti 2.16 (~60Nodes ~450Instances) and we are going to build a 2nd. cluster in Dallas in 2018 (same size)

some things that are spinning around my head:

- if we get more help from google-folks enable us to send more patches and getting it into the codebase would be cool (and will help speed things up)

- i like ganeti because of the hbal, hspace, hroller stuff as far as i know tair is no free vm-manage how is capable to do the same

- from my point of vier 2.16 is stable (we use it in ~6clusters and ~600instance in total, all kind of sorage-backends)

Things i miss in ganeti to bring it to the current state of the Art:

- macvlan/macvtab

- vlan bridging

- limit IO and traffic (KVM)

- allow >4TB disk on DRBD

but this ist all for a 2.17 release ;-)

Have a nice one,

Ansgar

2017-11-07 13:02 GMT+01:00 Sylvain Costard <sylvain...@univ-rennes2.fr>:

We are testing ganeti cluster here in france and this cluster should be in prodcutino in 2018. I 've started to make a php/laravel GUI under ganeti's API.

--

Sylvain Costard
Administrateur Système Unix
D.S.I - Pôle Infrastructures - Domaine unix
02.99.14.13.47

Le 07/11/2017 à 12:46, 'Philip Hansen' via ganeti a écrit :

Hey everyone,

I happen to be subscribed to this list, although I'm not actually involved with Ganeti, so I cannot say anything on behalf of Google in direct context to the project.

What I can say is, we still get these emails, and we are seeing them.

I know a colleague who is actually involved with the project has seen your messages, and is working on a response to this.

Just so you all know you're not shouting into the void :)

Philip Hansen | IT Resident | philiphansen@google.com | +353 1 519 9980

On 7 November 2017 at 11:22, candlerb <b.ca...@pobox.com> wrote:

> Google ... had migrated off of DRBD backends to hosting VMs on reliable filers via NFS

That's a useful data point - thank you.  Good to know where the focus is.

If you decide to run with shared storage backend, then the job of the front-end is mainly just starting and stopping the VMs.  There are a bunch of options for this, some free (e.g. oVirt, archipel), and some non-free or semi-free (e.g. Proxmox).  We definitely don't need to go to VMware :-)

However I do like running with local disk, without the SPOF of a shared file store.

In my ideal world, each VM image would be stored on its own separate ZFS filesystem, which was replicated periodically to one or more backup locations, using something like syncoid.

> The roadmap included notions to move the config data into etcd

That would be very nice.  The built-in config store has been a source of bugs in the past; being able to bolt onto an existing etcd cluster would be attractive.

[Benjamin Redling]

Small 6-node cluster running 2.15 on Debian Stretch (thanks to Guido and
Apollon!) serving everything here.

On 11/7/17 12:22 PM, candlerb wrote:
[...]

> There are a bunch of options for this, some free (e.g. oVirt,
> archipel), and some non-free or semi-free (e.g. Proxmox).

[...]

while I'm not having anything against Proxmox and their business model
I'd prefer someone would monetize on the same level (ca. 50€ per socket,
year to have a "Enterprise" repo. with backports / or ~1200€/a without
support cases) and have dependable releases of Ganeti/(+Syneffo?) for
the latest Debian stable.

Sorry, I need to justify spending. I don't think I'm allowed to do
donations.
Or something like (co-)sponsoring a 2.16 backport would be possible --
we are able to do a "Werkvertrag" / works contract.
It would be utterly cool to have (a greek? ;) company able to grow on that.

Currently the only questions that counts for me:
what's needed to have a future Debian 10 with a released Ganeti 2.16?

> However I do like running with local disk, without the SPOF of a
> shared file store.

Me too. Because I don't have a whole team handling storage issues like
Google -- there's only me.

> In my ideal world, each VM image would be stored on its own separate
> ZFS filesystem, which was replicated periodically to one or more
> backup locations, using something like syncoid.

Never heard of that before. Very interesting!

>> The roadmap included notions to move the config data into etcd

> That would be very nice. The built-in config store has been a source
> of bugs in the past;

[...]

As far as I remember during Ganeticon 2016 Brian Foley said that he
wished that would happen, as it would ease Ganeti code maintenance.
Rather than worrying that it isn't a Python or Haskell code base,
that would improve the situation by reusing a proven technology -- with
a bigger community, even if outside of Ganeti.
(But careful, just my memory... )

Considering that promoting 2.16rc to stable was mentioned during
GanetiCon 2016 to be possible in that same year or early 2017 because it
was running at Google for 1.5years(?), that willingness to bring things
is worthy of support.

Regards,
Benjamin
--
FSU Jena | JULIELab.de/Staff/Benjamin+Redling.html
☎ +49 3641 9 44323

[George K.]

Ganeti is in a very strange, or even difficult, situation. There's a ton of people who use it, and are really happy about it, but the people who are supposed to drive the development and organize the rest of us don't want to take the project any further. They also don't want any publicity about the project, for example ganeti website is definitely not what it should be.

Google clearly expressed that it doesn't have a lot (or even any) interest in ganeti at Ganeticon 2016. Since then Google has neither
 * put more people working on the project (makes sense since they don't really care about it)
 * given commit access to people/companies/institutions that actively care about ganeti's future GRNET/Debian/Skroutz/whatever (even though we were promised that)
 * packaged any new releases
 * released any roadmap for any changes (organizational or technical)

They way the issues were migrated from code.google.com to github is also a sign of not so proper planning/management. But even new issues that are opened on github don't really get answers, probably because nobody feels responsible enough or cares enough to answer. A community project should be providing prompt answers if it wants to grow. ganeti looks like it's trying to avoid having new users for the software.
 
There has not been an official release for years. 2.16 is still in RC, and 2.15 has serious bugs that are fixed in 2.16-rc. So what are people supposed to use? What are distros supposed to do? why would they package something that seems/looks/is abandoned or poorly maintained?

We (at GRNET) have more than 800 hardware nodes, in 30+ clusters, in 4 separate datacenters running either synnefo [1] or ganetimgr [2] to manage those clusters. We're heavily interested in ganeti and we're doing our best to fix stuff and publish our fixes. This is not enough though since Google who holds the keys to the project doesn't act at all.

Our current focus is using ganeti with userspace RBD using extstorage provider [3], improve networking using gnt-networking [4], modular backend support for snf-image [5] and so on...

IMO and I'm not speaking for GRNET now, this is strictly my personal opinion, the only hope for the project after so much idle time has passed, is if some people/companies/institutions finally decide to fork it and start publishing releases with fixes in order to gather the rest of the interested parties around them. Most companies/organizations that I know who are heavily using ganeti are already in a situation where they keep their own repository with their own patches, so this won't be something new to them.

GRNET's latest efforts are here:
https://github.com/grnet/snf-ganeti/tree/prepare-2.16 (not stable yet! but certainly usable)

I really hope that something actually happens and this project progresses as it deserves, so much work has been put into it and so many things have been accomplished. It will be a shame to let this project deteriorate.

Regards,

George Kargiotakis

[1] https://www.synnefo.org/
[2] https://github.com/grnet/ganetimgr
[3] https://github.com/grnet/gnt-extstorage-rbd
[4] https://github.com/grnet/gnt-networking
[5] https://github.com/grnet/snf-image

[Davide Bozzelli]

So George, Why GRNET does not fork ganeti at all ? 

Seems It has all the skills and the interest to do this.

Regards

--

Got problems with Windows? - ReBooT
Got problems with Linux? - Be RooT

[Federico Pareschi]

Hello everyone, I'm glad to see this conversation pop up in this mailing list. 

I'm not sure if everybody knows me but I'm the person that is currently owning and maintaining the Ganeti project for Google. As all of you can probably guess we're in a very tight spot and we need the community more than ever. It makes me happy to see so much interest from the community for this project and it makes me sad that I haven't been able to keep up with all the expectations.

As we had mentioned last year, Google had only two full time developers dedicated to the project as of Ganeticon 2016. Since then, those developers have transitioned to other projects and I've been put in charge of maintaining the open source side of Ganeti (among other things). Rafael has been helping me with some code reviews on github and we also had an intern (Calum) for some months working on a kvm postcopy implementation project as well. 

At Google currently, as Philip mentioned, we still greatly use Ganeti internally. 

As we mentioned at Ganeticon 2016 we have shifted our focus to NFS as opposed to DRBD clusters and we do have active interest in bugfixing and improving Ganeti as we keep using it. The bad news is that our resources for this scope are limited as I'm the only person currently maintaining it, the project is big and alone it takes great effort (both internally and externally), especially as I juggle it with other projects I am currently involved with. I acknowledge I haven't been able to do my best and for this I am very sorry. 

Our main focus for the last year was to migrate all the Ganeti stuff (Documentation, website, issues, codebase, development process, etc) to Github as we want to make it easier for the community to participate and help us develop it, however there have been a few hiccups on the way that caused delays (developers leaving, mostly). However, I'm still on track to do this and we hope to be finished by the end of the year. 

I honestly did not expect as much interest and so many different groups coming out of this email thread, it gives me more hope to see more activity (development, reviews, etc) on Github as well. As far as a 2.16 release goes, Viktor was handling it at the beginning of 2017 but he left the company and the release never happened. Due to a cycle of constant bugfixing on 2.16, right now I'm planning to finish merging 2.16 into 2.17 and then finalise the 2.16 release. I plan to get this done by the end of this month and I'll be looking for feedback from the community as far as 2.16 stability goes. We do use 2.16rc internally and so do others as reported in this thread, so I don't think there will be any problem with that. 

I'm always open to help everybody who needs support in getting involved in this project since it's open source, as I said we do have limited resources for development and implementation of new features, however we'd be extremely happy to see participation from you, the community, on Github. Not just with reporting issues, but also submitting pull requests, doing reviews and in general bringing the project forward. 

As I said before, I'd like to reiterate that I'm sorry I haven't been as active with the community and I want to change this moving forward. Expect some changes coming soon and don't be afraid to participate. Ganeti is a great project and it's not just a Google thing, the community is what drives it forward now. 

Please reach out to me if you have any further questions, 

Morg.

[Phil Regnauld]

'Federico Pareschi' via ganeti (ganeti) writes:
>
> As we had mentioned last year, Google had only two full time developers
> dedicated to the project as of Ganeticon 2016. Since then, those developers
> have transitioned to other projects and I've been put in charge of
> maintaining the open source side of Ganeti (among other things).

Oh, I was wondering why Viktor and Brian had not chimed in.
Bit of a shame we didn't hear about their being pulled off the
project :(

> Our main focus for the last year was to migrate all the Ganeti stuff
> (Documentation, website, issues, codebase, development process, etc) to
> Github as we want to make it easier for the community to participate and
> help us develop it, however there have been a few hiccups on the way that
> caused delays (developers leaving, mostly). However, I'm still on track to
> do this and we hope to be finished by the end of the year.

That's good to hear!

> I honestly did not expect as much interest and so many different groups
> coming out of this email thread, it gives me more hope to see more activity
> (development, reviews, etc) on Github as well. As far as a 2.16 release
> goes, Viktor was handling it at the beginning of 2017 but he left the
> company and the release never happened. Due to a cycle of constant
> bugfixing on 2.16, right now I'm planning to finish merging 2.16 into 2.17
> and then finalise the 2.16 release. I plan to get this done by the end of
> this month and I'll be looking for feedback from the community as far as
> 2.16 stability goes. We do use 2.16rc internally and so do others as
> reported in this thread, so I don't think there will be any problem with
> that.

Ack, thanks for the update.

Cheers,
Phil

[Phil Regnauld]

candlerb (b.candler) writes:
>
> In my ideal world, each VM image would be stored on its own separate ZFS
> filesystem, which was replicated periodically to one or more backup
> locations, using something like syncoid.

I'm currently prototyping this as an ext-storage provider for
doing what you describe, but over NFS, with 1 image / mountpoint.

This allows for per-image snapshotting. I hadn't actually considered
doing ZFS *on the host* [1] but that could also work, obviously.

I'm pondering how wise/reliable it is to have potentially hundreds
of mountpoints -- technically nothing wrong with it, but it requires
writing a handler to create the FS, exporting it, and then deciding
whether to lazy mount it when a node starts a VM, and unmount when
it stops, or just go whole hog and mount all VMs on all nodes.

[1] well, there is the abandoned https://github.com/ffzg/ganeti-extstorage-zfs

[John N.]

Not quite abandoned, the ZFS ganeti external storage has a fork:

https://github.com/brigriffin/ganeti-extstorage-zfs

and I use it since over 1,5 year with a 3 node ganeti 2.12 cluster on Debian 8 with over 50 instances. So far it works nicely and I am planning to setup a new 3 node ganeti cluster with Debian 9 and ganeti 2.15 also using that very same ZFS external storage provide. Have a try at it I can only recommend it. By the way the ganeti documentation maybe be updated to point at this fork instead.

Then I have to add a "me-too" that I am using ganeti since 2012 and happy with it. I do not want to use something else and hope that it can continue. I have a few clusters, mostly small from 2 nodes up to 6 nodes, with Debian 7, Debian 8 and soon Debian 9. Ganeti being in the Debian repo is also a big plus I must say.

Long life to ganeti!

Best,
John

[Equand]

We use ganeti in production for years. However the small scale of it and move from python to haskell had to curb our enthusiasm about it. Now we are looking at less systems with Proxmox instead of ganeti due to basically lvm+drbd only solution for the systems. We are looking at dropping COW even more and to do that we can't keep double the space on another system so the idea is to have zfs systems with zvols, backups as snapshots and remote snapshots with post-online migration in case of need of expansion. High availability is not needed much and space is much more important. Over years statistically speaking there was nothing lost which couldn't have been recovered from backups and systems overall are pretty reliable. Live migration is only needed to keep the systems up without downtime in cases of node upgrades, overpopulation and noticeable degradation of hardware (raid or ram problems).

For what it was ganeti was awesome (low footprint, ease of setup, sure some quirks but still). But now our goals have changed and proxmox looks like a better solution for this. Or even maybe raw libvirt.

[candlerb]

On Wednesday, 8 November 2017 10:28:31 UTC, John N. wrote:

Not quite abandoned, the ZFS ganeti external storage has a fork:

https://github.com/brigriffin/ganeti-extstorage-zfs

If I understand correctly, that uses zvol rather than an image file on top of zfs filesystem.

That's reasonable, given that a zvol block device is closer to the lvm/drbd service that ganeti natively uses.  In fact, possibly the extstorage interface of ganeti *only* supports block devices (I'm not sure about that)

Unfortunately, zvols don't play entirely nicely with snapshots:

http://jrs-s.net/2016/06/16/psa-snapshots-are-better-than-zvols/

Also: that reminds me about a long-standing feature request for ganeti: the ability to detach disks and treat them as independent entities.  This would let you have, for example, one disk as drbd and a second disk as plain, on the same instance.

http://docs.ganeti.org/ganeti/2.17/html/design-disks.html

[candlerb]

On Tuesday, 7 November 2017 22:37:31 UTC, Phil Regnauld wrote:

        I'm pondering how wise/reliable it is to have potentially hundreds
        of mountpoints -- technically nothing wrong with it, but it requires
        writing a handler to create the FS, exporting it, and then deciding
        whether to lazy mount it when a node starts a VM, and unmount when
        it stops, or just go whole hog and mount all VMs on all nodes.

With nfs4, you might be able to have a single /export directory, have all the zfs filesystems mounted under there, and export the whole thing.  Or bind-mount all the zfs filesystems under a single export point.

Otherwise, if you have to mount them separately, I would be inclined to mount only those filesystems which are required for instances running on that node - for scalability reasons. 

[Phil Regnauld]

candlerb (b.candler) writes:
>
> Unfortunately, zvols don't play entirely nicely with snapshots:
> http://jrs-s.net/2016/06/16/psa-snapshots-are-better-than-zvols/

The key statement being:

"If you don’t have at least as much free space in a pool as the REFER of a ZVOL on that pool, you can’t snapshot the ZVOL, period"

TIL something ;)

[Phil Regnauld]

candlerb (b.candler) writes:
> >
> With nfs4, you might be able to have a single /export directory, have all
> the zfs filesystems mounted under there, and export the whole thing. Or
> bind-mount all the zfs filesystems under a single export point.

I think the way NFS works (at least v3) it passes an inode reference,
so I wonder how that would work. Worth a try :)

> Otherwise, if you have to mount them separately, I would be inclined to
> mount only those filesystems which are required for instances running on
> that node - for scalability reasons.

Yes, that's a hook in the storage provider. It would definitely be
easier with just a plain NFS mount and using the shared file storage
provider.

[Phil Regnauld]

Phil Regnauld (regnauld) writes:
> candlerb (b.candler) writes:
> > >
> > With nfs4, you might be able to have a single /export directory, have all
> > the zfs filesystems mounted under there, and export the whole thing. Or
> > bind-mount all the zfs filesystems under a single export point.
>
> I think the way NFS works (at least v3) it passes an inode reference,
> so I wonder how that would work. Worth a try :)

From the FreeBSD exports(5) manpage:

"NFSv4 does not use the mount protocol and does permit clients to cross server mount point boundaries, although not all clients are capable of crossing the mount points."

... so indeed it might be possible.

[Phil Regnauld]

[while the goal of this experiment is to use a ZFS filer with
Ganeti, some might find it a bit off topic - let me know if
you do and I'll take it off-list]

candlerb (b.candler) writes:
>
> With nfs4, you might be able to have a single /export directory, have all
> the zfs filesystems mounted under there, and export the whole thing. Or
> bind-mount all the zfs filesystems under a single export point.

Ok, the latter doesn't seem to work very well.

I have, on an NFS server (FreeBSD):

/zroot/vms/a
/zroot/vms/b
/zroot/vms/c
/zroot/vms/d

... each of these ZFS FSes contains a single dummy image
a.img, b.img, etc...

I have then mount_nullfs'ed them all as follows:

mount_nullfs /zroot/vms/a /nfs/a
mount_nullfs /zroot/vms/b /nfs/b
mount_nullfs /zroot/vms/c /nfs/c
mount_nullfs /zroot/vms/d /nfs/d

... and exported them using nfs:

/nfs -maproot=0:0 -network 10.10.0.0 -mask 255.255.0.0

(This is on the FreeBSD server).

On the Linux server:

mount -o nfsvers=3 10.10.0.133:/nfs /nfs

then:

root# ls -l /nfs/
total 2
drwxr-xr-x 2 root root 2 Nov 9 00:15 a
drwxr-xr-x 2 root root 2 Nov 9 00:15 b
drwxr-xr-x 2 root root 2 Nov 9 00:15 c
drwxr-xr-x 2 root root 2 Nov 9 00:15 d

Ok, so far so good:

root# ls -l /nfs/a
total 0

Oops, seems nullfs re-exporting isn't DTRT. Could be that mount_nullfs
is aware this is being NFS exported. Now, it might work with bind mount
on Linux, but I haven't tried.


Much more interesting is using NFSv4.

On the server, /etc/exports is set up as:

V4: /
/zroot/vms -network 10.10.0.0 -mask 255.255.0.0
/zroot/vms/a -network 10.10.0.0 -mask 255.255.0.0
/zroot/vms/d -network 10.10.0.0 -mask 255.255.0.0

Notice I only explicitly export a and d.

On the Linux client, /etc/fstab

10.10.0.133:/zroot/vms /vms nfs nfsvers=4 0 0

Then

root# mkdir /vms
root# mount /vms

root# df -t nfs
Filesystem 1K-blocks Used Available Use% Mounted on
10.10.0.133:/zroot/vms 48171520 0 48171520 0% /vms

root# ls -l /vms/
total 2
drwxr-xr-x 2 nobody 4294967294 3 Nov 8 23:40 a
drwxr-xr-x 2 nobody 4294967294 3 Nov 8 23:40 b
drwxr-xr-x 2 nobody 4294967294 3 Nov 8 23:41 c
drwxr-xr-x 2 nobody 4294967294 3 Nov 9 00:09 d

Ok, looking good.

root# ls -l /vms/a
total 1
-rw-r--r-- 1 nobody 4294967294 0 Nov 8 23:40 a.img

root# df -t nfs
Filesystem 1K-blocks Used Available Use% Mounted on
10.10.0.133:/zroot/vms 48171520 0 48171520 0% /vms
10.10.0.133:/zroot/vms/a 48171520 0 48171520 0% /vms/a

Notice how the client dynamically cross-mounted vms/a

root@dnshost:/# ls -l /vms/b
ls: reading directory '/vms/b': Input/output error
total 0

Argh.

root# ls -l /vms/d
total 1
-rw-r--r-- 1 nobody 4294967294 0 Nov 9 00:09 d.img

So, while the client dynamically mounts sub-FSes, mountd on the server
needs to be told to export the FSes as they are created, then a "service mountd
reload" needs to be issued, otherwise attempts to access the mounted sub-FS
results in I/O error.

Not too critical I guess, but wish there was a v4 equivalent of "-alldirs"
here.

I will be experimenting a bit more with this, and see if I can cobble together
a storage provider on this. Ideally shared-file-storage could be used, but
then there's no hook to create the server-side exports line needed.

Probably can reuse https://code.grnet.gr/projects/extstorage/repository/revisions/master/show/shared-filer

Finally, since all disks for an instance will be created under the same
directory (one per instance), snapshotting granularity will be all
disks or nothing.

[John N.]

That's correct the ZFS external storage provider uses zvols for each ganeti instance and not image files. In fact you have two zvols one for the data and one for the metadata of DRBD, analog as if you where using plain LVM. Last year I posted my experience and modifications with this provider here:

https://groups.google.com/d/topic/ganeti/ZMgiLQ5ueZ0/discussion

That's not entirely true either regarding the fact that zvols don't play nicely with snapshots. In a virtualization context such as a ganeti node your zvols are not so big. For example in my case I have ganeti instances ranging from 10 GB up to 500 GB. This means that I have to make sure that my zpool has at least 500 GB of free space. Nowadays my ganeti nodes have at least 3 TB of storage, 500 GB out of 3 TB is approx. 16%, if I have a ganeti node with less than 20% storage space available I know anyway it's time to add more disks. So in fact I never encountered this problem just by using best practices in monitoring and capacity planning.

Furthermore and AFAIK, ZFS itself does not claim better performance by using zvols. I think this is also nowadays a myth (such as LVM storage vs plain image on FS). It's more about how you want to manage your instances' storage, with LVM or zvols you have the snapshot bonus and with ZFS, the send/receive extra-bonus for off-site storage of your snapshot.

That's my thoughts for now :)

Cheers,
J.

[candlerb]

On Tuesday, 7 November 2017 14:33:00 UTC, Davide Bozzelli wrote:

So George, Why GRNET does not fork ganeti at all ? 

Seems It has all the skills and the interest to do this.

I would support GRNET forking Ganeti to carry it forward.

I would also be fine with Google continuing to steward the project, as long as:

1. It merges reasonable patches in a timely fashion

2. It makes regular releases

There are also quite a few "big picture" architectural changes which have already been designed in outline (including some already mentioned: separating off disks into own top-level entity, changing to etcd).  I don't know if there's anyone who wants to step up and make those changes.

[George K.]

GRNET has forked ganeti since many years ago, it called the fork "snf-ganeti". 

AFAIK this decision was driven by the need to have faster, internal, release cycles and to be able to diverge from the upstream in order to have experimental/test code for long periods of time. Most of the stuff developed for snf-ganeti was at some point pushed upstream. When ganeti progressed, snf-ganeti was rebased on upstream, and so on and so on. In the past 2-3 years snf-ganeti development had stalled for quite some time, but things have changed since then ffffand there has been a lot of progress recently. That's another indication that a single company's fork is not the solution.

Can GRNET maintain a truly public fork and gather tons of issues/requests from users ? Certainly not. It doesn't have the resources to support other use cases apart from their own. And I guess that most other companies that use ganeti are more or less in the same situation. That's why I personally believe that a fork would be feasible only if a collection of people/companies/institutions create a steering committee to drive the project further.  All major stakeholders should gather and discuss how many resources they can devote, if any. Create a roadmap, drop technical debt, plan release cycles, answer open issues.

The situation can't be improved by individuals because the project is too big. Morg's efforts are great, but unless more people get commit access to the project and feel responsible for the project, there's no way it can progress. Right now, who would invest time/resources on ganeti since they're not sure when, if ever, their changes will be merged ? If google still wants the project to move forward without putting more of their people on it, it should give access to it. If it has changed its mind about giving access it should openly say so. People should stop waiting though, it's already been a very long time.

It's very important that ganeti keeps being included in Debian and possibly other distros, because this is a major force to move stuff forward. Since ganeti is not releasing stuff it gets closer to getting dropped, and that would mean fewer users, which means fewer people interested and eventually it will get abandoned if favor of other more maintained solutions. It is of utmost importance that ganeti gets automated builds/packages of its latest versions for at least the last 2 debian stable versions.

If google decides to give commit access, I'm pretty sure grnet can dedicate hardware resources for automated builds, a repository for release archives, and so on. If someone else can provide a decent website for the project that would be an awesome new beginning. Everyone is waiting for google to express what their plans for commit access is though.

Ganeti is still the best opensource virtualization cluster solution, but it certainly needs some love...

--

George Kargiotakis

[Equand]

Use sparse volumes

[candlerb]

But... but... live migration is so *cool* :-)

I definitely get where you're coming from though.  And libvirt can supposedly do live migration of block storage now (haven't tested it myself though).  If that works, then having the real-time replication of drbd is less exciting, when you could instead just take a zfs snapshot every 5 minutes and replicate that.

BTW, an interesting approach I've just come across is running kvm directly inside docker containers:

https://github.com/rancher/vm
http://rancher.com/introducing-ranchervm-package-and-run-virtual-machines-as-docker-containers/

Potentially then you can use generic docker "clouds" like swarm, kubernetes, rancher for running VMs.  It's a sort of prototype, and it's a bit inflexible (e.g. you can't easily change RAM/CPU settings after creating a VM), but I thought it was a neat idea.

[Sarah Newman]

On Monday, November 6, 2017 at 10:03:18 AM UTC-8, Daniel Howard wrote:

Ganeti is also weird in that it is super useful but nobody seems to use it, outside  of Greece. It feels like I'm the only user in Silicon Valley, (besides Google,) for instance. But it does a great job for us.

You aren't. We aren't "silicon valley" in the sense of we've never taken money from investors, but my company migrated to ganeti because we could fall back on managing instances by hand if something went wrong and all the various hooks and external script options made it relatively easy to integrate with previously existing infrastructure.

A big concern I have is python 3 support. There are 2 years before python 2 is supposed to be officially deprecated. Google does not appear to be devoting enough resources at this point to handle that time frame. If we have to figure out what to do for ourselves, I'm not sure if we'd try to convert to python 3 or to make something with the minimal feature set we need in a faster language, like golang.

[Randy Bush]

> A big concern I have is python 3 support. There are 2 years before
> python 2 is supposed to be officially deprecated. Google does not
> appear to be devoting enough resources at this point to handle that
> time frame. If we have to figure out what to do for ourselves, I'm not
> sure if we'd try to convert to python 3 or to make something with the
> minimal feature set we need in a faster language, like golang.

i do not think speed of ganeti orchestration is of concern. and i am
not a big chaser of language fads of the day (and i have seen many days
and many fads; wanna try PL/I :)

the cf of group decision making and folk finding time to actually work
will be the critical factors.

i am waiting to see if the sentiment out of the mini-con is to fork and
who will be able to make the serious committment of time. that is the
critical gating item for many things.

randy

[Equand]

Ok python 3 is not a fad, it was released almost 10 years ago.

By this time transition should've happened already, however python 2.7 brought a lot of crutches to help port some stuff... nevertheless they are incompatible