Over the past 6 months, we have discussed (at length) on what is the
best direction for Ganeti, given that the current code-base, while
working well for us, has accumulated a lot of technical debt in terms
of internal architecture deficiencies, programming language issues,
testability, etc.

The main pain points we have identified are:

- the master daemon is a big, monolithic entity that contains a lot of
  components; changes in any component (OpCodes, Logical Units,
  objects, etc.) have cascading and sometimes unpredictable effects on
  the other components
- the dynamic nature of Python makes any static analysis very hard (or
  impossible), which makes the previous point even more painful
- the multi-threaded nature of the master daemon creates scalability
  problems in Python (which cannot take advantage of SMP/multi-core
  environments)
- currently, the internal architecture relies on mutable objects,
  which creates additional problems regarding the consistency of
  internal data structures

As you know, Ganeti is written mainly in Python, with a sub-component
in Haskell and small bits on other glue languages (autoconf, automake,
make, shell, etc.). Over the (shorter) history of the htools
component, we have seen significant advantages from the functional
aspects, especially related to the above pain points.

So we have decided to make an experiment and see whether using Haskell
in more parts of the code base would help some of the issues we
have. We have in mind a staged use of Haskell for other parts of
Ganeti (besides htools), which goes roughly like follows:

- 2.6 release (April/May):
  - alternative "confd" daemon implementation in Haskell; this is
    fully optional and people can select at build time whether to use
    the Python or Haskell version
- 2.7 release (summer/autumn):
  - move query functionality from (Python) master daemon to the
    "confd" daemon (Haskell); again, this will be optional and
    selected at build time
  - provide one new, optional CLI tool (``gnt-*``) in Haskell; this
    will have some bits of new functionality that will not be exposed
    in a Python version

After the 2.7 release, we should be able to make a reasonable
assessment on the feasibility of using Haskell more pervasively in the
Ganeti code base, and on the downsides of doing so. Assuming it all
goes well, it means that starting with the 2.8 release, Haskell will
become required for building Ganeti. Of course, if we stumble and this
plan doesn't work well, we can still revert the changes, since most of
the new components are optional/extra.

Any interface that Ganeti provides right now (CLI: the ``gnt-*``
programs, LUXI: internal interface, cluster RPC: internal interface,
RAPI: external interface, IAllocator: external interface) will remain
the same, so any program that talks to Ganeti will still be able to do
so in the new world. We will keep providing the current Python RAPI
client (and we might look into providing a Haskell RAPI client).

From the point of view of building Ganeti from source and using it, it
will mean that:

- the Haskell compiler/platform will be needed at ./configure and
  build time, together will all libraries
- on the machines that Ganeti will actually run on, there will be no
  extra dependencies, on the contrary, we might have some reduction in
  libraries needed at run-time

From the point of view of using Ganeti from a distribution:

- we will continue to provide Ganeti packages in Debian, and if
  possible, in backports
- any distribution that has the Haskell platform plus the list of
  libraries we use (not too long) should have no issues in building
  and providing packages
- if you are a distribution user, installing Ganeti should bring it at
  worse the same dependency list, at best fewer dependencies

This is a long-term plan (2.7 will not come before autumn), so it
would be a good idea to discuss this now. We're especially looking at
feedback from users and contributors/developers on:

- as an end-user, would this change impact you at all? do you have any
  concerns regarding it?
- would it impact you if Ganeti requires Haskell to build/install? do
  you see deployment issues or such?
- would it impact you if the Ganeti code-base is written mostly in
  Haskell, with regards to contributing small patches or adding new
  features?

Of course, any other feedback on this topic is welcome.

First, I am somewhat of two minds as to whether to respond to this
thread or not.  I am not a ganeti developer, although I did push a
small fix last year.

My real concern is that ganeti is too monolithic.  Perhaps my
understanding is wrong here, but I think there needs to be more of a
separation between configuration and operations.  By my way of
thinking, it would really help maintainability if the process model
for ganeti operations were more like qmail.

Again, I am probably speaking from lack of knowledge here, but I would
find it much easier to navigate ganeti code if each operation were a
separate command-line driven module written in any language.  As a
simple example, I think that booting an instance should be a simple
script that is shelled on the target host with all parameters on the
command line.  This way, interactions between the operations and the
management functions are well defined.  This also allows you to
develop/test these operational scripts without dragging in the world.

This type of separation would probably hurt performance but in the
end, I think overall performance would improve because individual
operations could be tuned without impacting the center.  For example,
migrating a virtual from server to server would be a black box and
replacing dd over ssh with some other, faster, transport would become
a safe optimization that did not involve the central code base.
Similarly, if operations are shelled black boxed, I can envision a lot
of new features that are all doable in the context of KVM/XEN but are
difficult to plumb into the current structure.

None of these have much to do with the programming language involved.
I suspect that many of the operation "scripts" would end up in bash.

One other aspect of designing stuff this way is that debugging a live
server is a lot easier.  'ps' becomes a very useful tool that shows
you what is actually happening.

Again, my two cents, which are worth even less.

My "feature" list that I would like to implement involve exploiting
how KVM/XEN and more virtuals around and how this should allow neat
stuff like "live convert from plain to DRBM" or migration of a plain
instance to another instance without shutting it down by converting
the host to DRBD and back on the fly.  A smaller function would be
writing a socket helper to replace dd over ssh for copying volumes.

I know how to do this in the current code, but having a separation of
configuration and operations would make this a lot easier.

Finally, I don't want anyone taking any of this as a complaint.  I
will work with, and hopefully work to help any code base that I can
find time for.

Thanks for the feedback!

That means, shutting down a 1000-instances cluster would take around two
minutes just in _starting_ the jobs (in practice, it is much much slower
due to other issues). My goal is to have a job engine that can process
null opcodes in the realm of 1000 jobs per second.

We discussed about splitting the masterd (the logical units) into
separate processes, however there are a few downsides for doing that at
the current moment.

The proposed change for 2.7 is to split the query paths outside masterd,
so that what masterd does becomes more focused - a pure job execution
engine.

Later, we have some rough plans of splitting the configuration and
locking entirely out of masterd, so that all the "core" logic is
separated out of the individual job execution part. At that stage, the
practical difference between running multi-threaded or running
multi-process is low, so we could change to whatever model is best.

But we can't make this transition today, or very quickly. Hence the
staged approach, because in the meantime we still have to release and
deliver features incrementally.

Making a bunch of architectural changes at the same time as language
changes seems particularly fraught.
I'd be inclined to see how it goes breaking down the master daemon, and
the other architectural changes goes towards reducing the desire for
adding languages.

I wonder if your seeing the grass as greener with the new language, this
will solve all out problems etc etc.

Things I'd like to see moving forwards in terms of features are
primarily better support for generic instances.
I am using ganeti to run a clients office, they have a nice pair of Dell
R210 II's and 8 VM's, all of different flavours, windows, linux, bsd etc
so I have always used the instance+image but I've never really felt like
its the way its meant to be used, integrating instance+image into the
base distro would probably help with this.

A simple way to snapshot instances would be nice (even if it was disk
only, and even offline only would be ok) so I could snapshot the mail
server, do the upgrade and roll back if it all goes horribly wrong.

Oh the ability to change the drbd cache mode. Disk writes are pretty
crappy and I'll take the performance gain over the risk of failure,
particularly if it can be easily changed, IE if I'm installing windows
or something it doesn't really matter if it gets hosed or copying in a
bunch of files from the old file server, I can just re-do it on the .01%
chance of a failure, but in the meantime the performance gain would be
really nice lol.

That said.
Its really working rather well providing HA without needing a dedicated
storage back end (which is the reason I picked it), they had a power
failure the other week that took out one server, everything was up and
running on the remaining node inside 10 minutes which isn't bad for the
first time I've needed to do it.

On 20/04/12 03:18, Iustin Pop wrote:

Thanks for the feedback. Comments inline.

If everything goes well, and we see the improvements we expect, then
yes, we'll have more work to do (conversion). But we don't know yet, so
we can't say "the end goal is a full conversion".

Rest assured that continued stability is in our own direct interest!

But since we anyway have to refactor _heavily_, we want to see if
refactoring in a language which we know will improve on _some_ aspects
sounds like something worth trying.

Again, thanks a lot for the feedback, especially the missing features!

To put it differently, it's more of a matter of non-blocking operation of the
daemon: the daemon issues requests and blocks for responses. To make the
analogy, wouldn't the model of a select()-based event loop inside a *single*
process work for the master? Is there something that makes it computationally
intensive, so scalability with python-based threading suffers? It seems
the master's scalability is more impacted by locking, more on this below.

How do you see handling locking evolve in the future, with and without
Haskell's increased presence in the code? you mentioned separating the
locking mechanism from the rest of the master daemon. How would this impact
lock handling, and do you have plans to explore less strict locking in favor of
higher-throughput processing?

Regarding support for generic instances, please take a look at
snf-image:

https://code.grnet.gr/projects/snf-image/wiki

It's a Ganeti OS definition we have developed, which tries to simplify
the deployment of Ganeti instances from predefined images. We've been
running it for some time now in production (Debian/Ubuntu/Fedora/Windows
instances), and it has proven quite helpful.

It has a number of features [untrusted image deployment in volatile VM,
setting hostname/passwords, file injection], it'd be great if you could
give it a try and give any feedback you may have.

And for locking, oh yes, I totally agree.

Indeed, we will take a hit from the point of view of easy contributions.
We have had many discussions trying to understand how serious this
impact will be, but we are unclear.

What we hope to gain is however, on this topic, easier integration of
code patches (once they are written). For one thing, getting the code to
compile means already a lot of trust that it is not broken, and that it
doesn't break any assumptions in the existing code base (e.g. modifying
global state incorrectly, etc.).

To solve this issue, we had the node resource design model, which was
supposed to be implemented in 2.6. This design was intended to remove
all long-duration locks related to the iallocator, and would have
allowed to do "many" iallocator runs in parallel, and during other
operations. The design was modeled on how htools does the 'predictive'
allocation, so the design itself is, we believe, sound.

Michael worked and tried to implement this design in the current Python
code base, and after fighting with it, came to the conclusion that due
to the current model (resource/locks) inside Python and due to the fact
that we can't "protect" data structures in Python, we cannot *reliably*
implement it.

So short story long, htools can predict how the cluster will look after
100 balancing steps, whereas the current Python code doesn't. If we end
up with the locking/config daemon in Haskell, I already know how the
design would look to remove long-lived locks during iallocator run.

Some updates on this plan. 2.7 was delayed as 2.6 itself was a couple of
months late. As such, we've decided to slightly tweak this plan.

2.6 was released with the Haskell-based confd, and we run it in
production with no issues so far, including some new functionality not
provided in the Python version. Furthermore, we have some maintenance
overhead with the fact that htools is still optional, so we can't depend
on some feature that it provides.

So the new plan is as follows: the base Haskell environment (GHC
compiler and the base htools dependencies: the json, network, bytestring
and parallel library) will become required for Ganeti 2.7. The
additional packages used for extra functionality, curl, hslogger, crypto
and similar, will remain optional.

This means that in order to build Ganeti, you will need a Haskell
compiler plus some basic libraries that should be available in existing
stable distributions. For "full" enabling of the haskell components, you
will need to have the extra libraries (either from cabal or by running a
newer distribution).

This will have the following advantages:

- allow base Ganeti to depend on htools, and hence promote integration
- simplify the build configurations and requirements (hopefully they
  will be simpler, due to the reduction in the possible combinations)
- allow exploration of more Haskell use in the project, by allowing
  critical components to be implemented in this language; this is not
  possible, nowadays, due to the "optionality" of htools

The disadvantage, of course, is the requirement for GHC and said
libraries. To compensate, we aim to ensure that our dependency list is
sane for existing stable/long term distributions.

We'll have more precise details on the new requirements as we near the
release (not soon, by the way).

Feedback welcome, as always.

Thanks for the update!

If the former, all is good - curl won't be a 'base' dependency, but only
an extended one. If the latter, hmm…

Just to recap: needed will be: ghc, json, network, bytestring (comes
with ghc), parallel.

The plan sounds well thought-out.
We've been trying to become more capable in Haskell, in an effort to
contribute to the codebase in the future.

We're fine with the Haskell dependencies, as long as they are available
for current Debian releases. It seems there are a few problems trying to
get everything right when running Squeeze. Can you comment on whether
things be better in Wheezy wrt the Haskell dependencies of Ganeti?

Finally, it would be very helpful if there was a single point detailing
all current dependencies, Haskell-related or otherwise. Do you plan on
keeping an updated list somewhere, as new features get added to master
and perhaps new dependencies are introduced?