ROADMAP/REDESIGN: Puppetlabs-mysql

[Ashley Penney]

MySQL Roadmap/Redesign

[This email follows a consensus amongst people attending the Module Triages that we would like to explore the possibility of cutting edge engineering techniques such as "design before you start mashing manifests together" and "lets ask the public what features they'd like before deciding on their behalf".  As such this is an attempt at pulling together an initial explanation of the pain points of the current module and where we should go from here.]

Background

The puppetlabs-mysql (we'll refer to it simple as the mysql module from now on) module accreted from a number of PRs that solved immediate and instant pain points for users over a number of years.  A couple of months back I took a stab at a fairly large refactoring of the module.  The two things I wanted to solve was:

* Improve the type/providers so they could be used via puppet resource and fix numerous bugs.

* Rewrite the module to follow our fledgling best practices.

* Make it possible to pass in arbitrary configuration values so we didn't need a parameter per my.cnf entry.

This refactoring/rewrite semi-worked.  It exposed a metric ton of issues and problems along the way, and our introduction of an acceptance framework helped us to fix most of those.  It did solve the goal of making the module easier to work with, and it did improve the situation of arbitrary configuration, but the general consensus of myself, and the users, is that there's still a large amount of significant problems that fundamentally stem from the fact the module has no base design, it just kind of happened, and the rewrite attempted to retain backwards compatibility with that API.

Problems we know about

A user of the module asked his local DBA expert to help us review the module, which lead to a number of issues which I'll list out below.  I've taken liberties with this review, rewriting parts and condensing others to make it more suitable here.

Configuration Variables

MySQL doesn't treat all configuration values equally, but we do.  We don't handle:

Initial startup configuration.

Certain settings, like innodb_log_file_size need to be set before we ever start the service.  If they change after the service is up and running you have to be able to shutdown MySQL, then delete the appropriate ib_logfile files, then restart.

Dynamic configuration

MySQL allows a whole bunch of stuff to be set via mysql set. We should use this where possible and not restart MySQL for no reason.

Defaults

Our $default_options don't reflect what DBAs consider good defaults.

Password hashes

We require plain text passwords in most places in the module, rather than letting people pass in the hash directly if they want to.

Client Certificates

We don't have any facility to use these.

Configuration in general

We're still just relying heavily on my.cnf, rather than breaking things up into a conf.d.

Smaller concerns

A lot of these relate to the module as it is today, not what we'd do with any redesign work:

* server/account_security.pp inflexible.

* server/root_password.pp the endless troublemaker.  Only works as root thanks to /root, should use /etc/something and --defaults-file.

* db.pp has lots of issues.

* types need further work.

Where do we go from here?

I don't want to constrain the design based on my pre-conceptions so first I want to continue to gather requirements and criticisms of the module in this email thread.  I've started writing up my own thoughts and those are basically:

* Configuration is the biggest problem we face and we need to tackle it head on.

* Ordering is the next largest problem, we have a number of use cases and it's difficult to handle them all.  (stuff like handling existing mysql setups on a box, people want to take those other with puppet surprisingly often).

I have a lot more words written down but that's basically the two things I suspect we need to address out of the gate.  Anyway, let me know what you people think puppetlabs-mysql needs.

--

Ashley Penney

ashley...@puppetlabs.com

Module Engineer

Join us at PuppetConf 2014, September 23-24 in San Francisco

[Gareth Rushgrove]

One of the questions/conversations that came up at Config Management
Camp around large modules like this one was: should/could they be
split up into multiple sub-modules?

For instance one of the usecases was around managing mysql databases
without managing the mysql server, for instance in cases like Amazon
RDS or similar. Or I'd guess the opposite, manage the servers but let
others manage the databases (for instance in a multi-tenant
environment)

Any thoughts on that?

Gareth

>
> --
> Ashley Penney
> ashley...@puppetlabs.com
> Module Engineer
>
> Join us at PuppetConf 2014, September 23-24 in San Francisco
>

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-dev+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-dev/CAC9eg%2B%3DLhKvGjCPGmziu4MdFcbhEbU-BRd67d6ntchZANaoNaA%40mail.gmail.com.
> For more options, visit https://groups.google.com/groups/opt_out.



--
Gareth Rushgrove
@garethr

devopsweekly.com
morethanseven.net
garethrushgrove.com

[Jeremy T. Bouse]

[Jeremy T. Bouse]

On 01.03.2014 05:03, Gareth Rushgrove wrote:
>

> One of the questions/conversations that came up at Config Management
> Camp around large modules like this one was: should/could they be
> split up into multiple sub-modules?
>
> For instance one of the usecases was around managing mysql databases
> without managing the mysql server, for instance in cases like Amazon
> RDS or similar. Or I'd guess the opposite, manage the servers but let
> others manage the databases (for instance in a multi-tenant
> environment)
>

I can echo this as a use case for most any database including MySQL and
PostgreSQL. I do actually use RDS in some environments and there are
other environments where the ability to do one or the other is desirable
if not required and thus leave modules unable to perform that way not
considered.

Regards,
Jeremy

PS. Apologizes for the empty email earlier hit send too quickly.

[Ashley Penney]

I would absolutely like us to explore breaking down some of these modules into smaller pieces.  For -apache for example we've talked about breaking out the vhost stuff, a lot of the more complex mod_* stuff, and so on into separate modules. 

Another piece to this that we've talked about internally is an ARM proposal that Hunter was working on (I'll just throw him under a bus here) to introduce a kind of include system that would allow you to do things like:

include puppetlabs-mysql as mysql

include mycustommodule-mysql::server as mysql::server

In order to make it easier to override parts of modules or just use pieces of them.  I think something like this (He's got a much better syntax/explanation somewhere) combined with smaller modules would go a long way towards making them more usable.  We'd still probably ship a meta-module that includes all of the smaller modules as a whole, but you'd be able to just nab the types/providers or the backup piece or whatever you needed.

To bring this back to mysql I guess I'd see:

mysql-provider

mysql-server

mysql-server-accounts

mysql-server-backup

mysql-client

As potential places to split a module like this into smaller bits.  It would definitely be more complex to have a bunch of smaller modules and ensure API compatibility between them all, but it's nothing that we couldn't manage.  Do you have a suggestion on how detailed a split you'd like to see for something like MySQL?

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/e58569e96ca0273607ed3081caab15fa%40undergrid.net.

For more options, visit https://groups.google.com/groups/opt_out.

--

[Peter Meier]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> To bring this back to mysql I guess I'd see:
>
> mysql-provider mysql-server mysql-server-accounts
> mysql-server-backup mysql-client
>
> As potential places to split a module like this into smaller bits.
> It would definitely be more complex to have a bunch of smaller
> modules and ensure API compatibility between them all, but it's
> nothing that we couldn't manage. Do you have a suggestion on how
> detailed a split you'd like to see for something like MySQL?

+1 for this kind of split!

~pete

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlMTsnYACgkQbwltcAfKi3/VjQCfTHusIPOLogHddwGJnRXZ04iI
o40An2aXND/arDnGI2w40VERndPezLG1
=xKis
-----END PGP SIGNATURE-----

[Daniele Sluijters]

Hi,

That user that asked his DBA was me so you know most of our issues with the module by

now. One that's especially high on my list is the SSL client certificate stuff, both for

c2s and s2s connections but also for user authentication.

I'm also liking the idea of splitting this module up, it would remove part of the complexity.

With a bit of luck we might even be able to get to the point where we can have an abstract

type like 'db', 'db_user', 'db_database', 'db_permissions' and so forth that have different

providers for MySQL, Postgres etc. allowing people to use the same module with a different

backend if that better fits their environment.

MySQL is an incredibly complex beast to configure and maintain, especially if we want to

expose all the nobs and dials people might need. With that in mind, if we're going to do the

redesign though I'd very much like to propose we ditch support for old versions of Puppet,

so at least pre-3.4, and stdlib pre 4.x. I'd even go as far as state that if there are features in

future parser that will make our life significantly easier we might want to consider branching

the current release of the module to an 'LTS' version and truly start anew.

[Igor Galić]

To bring this back to mysql I guess I'd see:

mysql-provider

mysql-server

mysql-server-accounts

mysql-server-backup

mysql-client

Do we have a precedent for these kinds of patterns?

--

Igor Galić

Tel: +43 (0) 664 886 22 883
Mail: i.g...@brainsware.org
URL: http://brainsware.org/
GPG: 8716 7A9F 989B ABD5 100F  4008 F266 55D6 2998 1641