Ongoing efforts towards Fedora 20

[Peter Law]

Hi,

For as long as we've had it, badger has been running Fedora 17. While
that was fine for a while, it's nowhere near current, and reached its
end of life late last year. Importantly, this means that it's not
receiving updates any longer which means we can't make use of the
latest features of various things.

As you're hopefully aware, most of badger's configuration is in puppet
[1], which somewhat simplifies the process of setting up its
replacement. Unfortunately, the version of puppet on F20 is much newer
than on F17, and as a result, our puppet config needs some work before
we can use it.

There is a collection of tickets relating to this work, under the
milestone "SR2014 Server Update" [2], which includes a number of
things in this area, not just the OS upgrade (though that is the big
ticket item).

I'm currently working towards having working puppet state, and I've
emitted a number of patches to that end. I'd originally thought about
having a branch in the puppet repo for F20 issues (and this is what
the fedora-20 branch was for), but I've since realised it's much more
sane to just keep everything as patches on master.

The main reason for this is that the large tidyup I did as part of the
early work (in order to reduce the number of deprecation warnings and
lint errors) mean that merges of mast branch changes almost always
conflict, but also that puppet is very much meant to work cross
platform, and has mechanisms for detecting which OS it's running on.

The upshot of this is that there are an increasing number of puppet
patches which need attention.

So, this is a call to anyone who is interested in badger things to
please go and try out the patches [3] and provide feedback.
Alternatively, if you'd like to get involved with the upgrade, reply
to this thread so we don't end up duplicating effort (there are some
complications around the LDAP module which I'll post about shortly).

Thanks,
Peter


[1] https://www.studentrobotics.org/cgit/server/puppet.git
[2] https://www.studentrobotics.org/trac/query?status=!closed&milestone=SR2014+Server+Update&group=status&col=id&col=summary&col=status&col=owner&col=type&col=priority&col=component
[3] https://www.studentrobotics.org/gerrit/#/q/status:open+project:server/puppet,n,z

[Peter Law]

Hi,

> As you're hopefully aware, most of badger's configuration is in puppet
> [1], which somewhat simplifies the process of setting up its
> replacement. Unfortunately, the version of puppet on F20 is much newer
> than on F17, and as a result, our puppet config needs some work before
> we can use it.

> (there are some
> complications around the LDAP module which I'll post about shortly).

It appears that the LDAP/PAM/NSS setup on badger isn't entirely
simple, and I by no means truly understand all of it. I believe that
Jeremy does, so hopefully he'll dive in and correct anything i get
wrong.

We currently use a modified version of an external module [2] to
handle the initial setup of the LDAP services, and then a custom type
& provider to configure some of the actual entries in the database. We
then also load pre-canned values into the database from the secrets
repo to fill out the data.

The last part is somewhat standard, since the data for a dev server is
going to be different than that on the live server, and this mechanism
is also intended to be used for restoring from a backup.
Unfortunately, the separation between the things in config vs in the
data isn't really in the right place, with some things being in data
where they should probably be config. Thankfully while annoying that
doesn't actually cause any issues.

The real complications come from the combination of the changes we've
made to the ldap module and the fact that the upstream has since been
massively refactored [3]. The changes break down into roughly two
groups: those made were to work around a bug around having a client
and server setup on the same machine, and changes to allow local
definitions of logins to badger to work at the same time as LDAP
logins.

First up, it doesn't appear that the server-client bug was ever
reported upstream, so I've done that [4] and I'll submit a pull
request for my fix [5] shortly.

I've also attempted to reapply our local-login related patches to the
upstream, these appear in the 'srobo-tweaks-on-latest' of my fork [6].
Some review on wether or not that does the right things would be great
-- I didn't want to submit that to gerrit as I'd likely have to submit
the entire upstream for review first.

However, it's notable that unfortunately those changes on their own
still don't get the ldap module working on F20, due to the nss_ldap
package having been retired. I've not yet managed to find out why that
occurred, and I'm not really sure what it achieved so I'm unsure how
to work out what (if anything is needed to replace it). Input here
would be most welcome.

The combined result of my work on getting the ldap module to work on
F20 as we need it can be found in the 'srobo-tweaks+f20' branch [7] of
my github fork. This is the branch which I'm using, and which appears
to work; if you'd like to get involved, this is the branch to drop
into puppet.git/modules/ldap to move beyond the LDAP errors. Again
though, this needs some review from someone who better understands
what's actually happening to LDAP as a result of the specified
configs.

Peter

[1] https://www.studentrobotics.org/cgit/server/puppet.git
[2] Specifically, the upstream is here:
https://github.com/jfryman/puppet-openldap, though that's not
documented anywhere I'm aware of, and our modifications live in
https://www.studentrobotics.org/cgit/server/ldap.git
[3] Ironically, this appears to have happened within weeks of us
having applied our modifications.
[4] https://github.com/jfryman/puppet-openldap/issues/12
[5] https://github.com/PeterJCLaw/puppet-openldap/tree/fix-duplicate-package
[6] https://github.com/PeterJCLaw/puppet-openldap/tree/srobo-tweaks-on-latest
[7] https://github.com/PeterJCLaw/puppet-openldap/tree/srobo-tweaks+f20

[Peter Law]

I wrote:
> The upshot of this is that there are an increasing number of puppet
> patches which need attention.
>
> So, this is a call to anyone who is interested in badger things to
> please go and try out the patches [3] and provide feedback.

I probably should have noted that I'm intending that all of these
should work fine against F17, but that for the moment none of them
solve all of the F20 issues, and you may need to apply some of the
others in order to make the later ones work.

> Alternatively, if you'd like to get involved with the upgrade, reply
> to this thread so we don't end up duplicating effort (there are some
> complications around the LDAP module which I'll post about shortly).

Aside from the LDAP complications, the main items remaining (at least
the things I'm currently getting errors for) are gerrit and apache.
I'm expecting that the apache issues are around the newer version
having a slightly different config file format.

Peter

[Murray Colpman]

On 06/07/14 15:58, Peter Law wrote:
> Unfortunately, the version of puppet on F20 is much newer
> than on F17, and as a result, our puppet config needs some work before
> we can use it.

Out of interest, will the new version fix the issues we were having
involving the IPv6 firewall not being configurable? (I can't remember
the specifics and I wasn't involved - only heard about it afterwards -
so it might not be entirely accurate). Now that v6 is actually working
in ECS, it's making using not-HTTP rather annoying as many tools try to
connect on v6 first (with a long timeout) - git for example.

Murray.

[Peter Law]

I wrote:
>> Unfortunately, the version of puppet on F20 is much newer
>> than on F17, and as a result, our puppet config needs some work before
>> we can use it.

Murray wrote:
> Out of interest, will the new version fix the issues we were having
> involving the IPv6 firewall not being configurable? (I can't remember
> the specifics and I wasn't involved - only heard about it afterwards -
> so it might not be entirely accurate). Now that v6 is actually working
> in ECS, it's making using not-HTTP rather annoying as many tools try to
> connect on v6 first (with a long timeout) - git for example.

I've no idea. I have had to update the firewall module to make that
work, so it's possible that that will enable this sort of thing.

I highly encourage that you have a look at the docs for it and find
out how to make this work!

Peter

[Peter Law]

Hi,

ITT: Progress occurs.

I've now got a (mostly) functional F20 badger equivalent (code name to
be decided AFAIK, suggestions welcome).

For details on how to get one set up yourself, please see my latest
entry on the relevant ticket [1480].

Most of the things which aren't working are quick and simple fixes
where there have been minor changes in the underlying config
mechanisms, or package names have changed.

The only currently outstanding thing I know is non-functional is the
team git http repo access, though it's entirely likely that this will
be a similarly quick fix -- I've not yet looked at why it's failing
(but will do later today (Sunday) after some sleep).

I previously wrote about various LDAP complexities, including an issue
where you couldn't have a server and client on the same machine with
the ldap module we're using. I'm pleased to say the upstream
maintainer accepted my pull request so that's now fixed.

Our own tweaks on top remain, and I've updated my srobo-tweaks
branches accordingly [1]. Review from someone who actually knows this
stuff would be appreciated (and certainly required before we can use
them in production).

I'll continue poking my VM tomorrow to try to work out what else (if
anything) needs to be done to get this shippable.
I think we're fast approaching a point where we're ready to ship this,
which is good, though I'd like to see others able to reproduce having
a working VM first!

Peter

[1] Without any F20 fixes:
https://github.com/PeterJCLaw/puppet-openldap/tree/srobo-tweaks-on-latest
With F20 fixes (but ought to still work on F17):
https://github.com/PeterJCLaw/puppet-openldap/tree/srobo-tweaks+f20
[1480] http://trac.srobo.org/ticket/1480

[Jeremy Morse]

Hi,

I'm gearing up to deploying this -- great thanks to Peter for delving
into what was going wrong with LDAP, IIRC it was hell last time I had to
deal with it. I've pushed your LDAP fixes onto badger and bumped the
submodule in puppet, and everything Just Works (TM), which is excellent.

> Unfortunately, the separation between the things in config vs in

> [LDAP] data isn't really in the right place, with some things being

> in data where they should probably be config. Thankfully while
> annoying that doesn't actually cause any issues.

It is indeed a mix -- users and groups are clearly data, but also so are
things like the ide, anon, and tickets users, which need to exist for
the puppet configuration to work (and thus are config), plus I
occasionally rotate their passwords. Things like the structure (o=sr,
ou={users,groups}) are definitely config too.

Some of where to draw the line is debateable, I'd enjoy having a
definition of what is data and what is config, or some other way to
classify it.

I haven't touched the v6 config as it's no-where near high priority.
Patches welcome.

I'll be feature testing this over the next few days, and testing how
good the backup/restore facility is. Hint: it's going to require much
more work.

--
Thanks,
Jeremy

[Jeremy Morse]

Hi,

And having said LDAP was ok, it now turns out that in certain puppet
orderings, /etc/openldap/slapd.conf is installed before the
openldap-servers package. And the package moves slapd.conf. Hurrah.

--
Thanks,
Jeremy