puppet-dev status the week ending 14-11-14

[Henrik Lindberg]

** Next PR Triage Wednesday, November 19th @ 10:00 am Pacific. **

Priorities

1. Puppet 3.7.4

2. Native Facter 0.3.0

3. Puppet Server 1.0.0

4. Code removal for puppet 4

5. New puppet doc implementation

Commentary

Puppet:

We are going to keep 3.7.4 in parallel with 4.0.0 to ensure we get a 3x release that contains

a future parser that is as close as possible to what is released in 4.0.0. It will be released in close proximity to 4.0.0 (unless there is a burning need to release it sooner).

Please note that some changes in puppet 4 mean that puppet 3 agents won’t be able to talk to puppet 4 masters (or vice-versa). This probably means you don’t want to be updating puppet itself with “ensure => latest” :)

One of these changes is for the URL structure for the master/agent HTTP API.  Specifically, we’d like to move the environment name away from being the first segment of the URL, so that the URL space is more well-contained and predictable.  This will allow us to begin separating the monolithic “master” service into more fine-grained services.

Code removals for puppet 4 is picking up more steam (yay, burn all the things…) - 17k lines removed so far. See a heads up below if you are doing advanced things with any of the classes under puppet/parser (an additional 15k removal).

We are doing major triage for 4.0.0 as we have accumulated quite a bit of scope creep. You will be seeing tickets move to target 4x as we like to reserve the 4.0.0 tag for things that we feel are required for this release. This means that we may pull in tickets/PRs marked 4x, but we will not block the release if they do not make it in time.

We are also engaged in yet more work around environments - this time removing support for legacy environments.

Puppet Server:

Puppet Server 0.4.0 was released!  The changes were mostly plumbing; packaging improvements, tweaks to make the CA behavior more consistent with the old CA, and some improvements related to SSL transports.

We’re planning for the next release to be our 1.0 release.  It will include a few cool new features related to directory environment caching, plus a few minor tweaks and bug fixes.  The 1.0 series will be compatible with the Puppet 3.7.x, so next up will be to start working on Puppet Server 2.0 for Puppet 4.x compatibility.

Puppet Doc:

We have worked on the scope of the new puppet doc implementation (puppet strings), there are now tickets, and we are just starting to work on those.

Native Facter:

Solaris, Windows, ec2/gce facts are all in - there are a handful of misc fix-it tickets remaining before we tag 0.3.0.

Also, we now have Native Facter nightlies!

• http://nightlies.puppetlabs.com/cfacter/

• http://nightlies.puppetlabs.com/cfacter-latest/

Right now it’s rhel6/rhel7/wheezy with more packages coming. Try it out and tell us what we’re missing :)

Other conversations of note:

• Henrik taking over after Andy

• Yes, I am stepping in after Andy which means that I now will be saying no to my own crazy ideas - or face the consequences :-)

• Rosetta lands Philae on Comet 67P

 

Heads up on Code Removals

With the fix in PUP-2906 being merged (reimplementation of collections) we now freed up for the removal of the old AST classes and work is now ongoing on ticket PUP-3274, which removes the AST and switches puppet 4.0.0 over to always using the future parser.

As a heads up (if you have written code that relies on any of the parser classes), this means that:

• The --parser setting will be removed (it is always parser=future from this point. (Well unless there is the need to again do language feature work and have a "new future")

• The old Puppet::Parser::Parser is removed (if you have written code using the parser directly you should have switched to using the ParserFactory a long time ago.

• The old Puppet::Parser::Lexer is removed - if you relied to on low level lexing you need to use the new Lexer instead.

• Almost all of the AST classes have been removed except those that are used by the compiler and some of the functions. (List below of what remains of the AST).

• The AstTransformer has shrunken to only deal with a few 4x to 3x tasks. It may be refactored and go away completely.

• A large number of tests have been removed.

• RDoc support no longer produce output for any .pp files (to be replaced with the Puppet Strings module).

• Since we are also dropping support for Ruby 1.8.7 in Puppet 4.0.0 the support for RDoc1 has been removed.

Remaining AST:

• AST::Regexp - a class we would like to also remove in favor of a different construct, but it is currently expected for supporting Node.

• AST::BlockExpression - is used as the container of code in the top level constructs that the compiler cares about. Will stay around probably until the compiler is rewritten.

• AST::HostClass - a top level construct the compiler cares about.

• AST::AstArray - used in the top level constructs - I would like to see this go away, not sure when that will happen.

• AST::ResourceInstance, is currently used by the create_resources function to get the same behavior as in 3x Puppet Languag when creating a resource. May be rewritten to instead use the same logic used in the new evaluator.

• AST::ResourceParam, these are used to instantiate resource attributes, they are registered as part of overrides and defaults. These will stay around until the compiler is rewritten.

• AST::Branch, an abstact class used by ResourceParam (and others).

• AST::TopLevelConstruct, an abstract class for some of the others.

ALL OTHER AST CLASSES HAVE BEEN REMOVED. If you have logic that makes use of those you need to do things differently. Please ping if you are

in this situation and need help with the transition.

Regards

Henrik Lindberg
henrik....@puppetlabs.com

-- 

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[Henrik Lindberg]

Sorry a bout the botched format... investigating

[Henrik Lindberg]

Hope this works better

- henrik

[Felix Frank]

On 11/14/2014 07:09 PM, Henrik Lindberg wrote:
>
> Please note that some changes in puppet 4 mean that puppet 3 agents
> won’t be able to talk to puppet 4 masters (or vice-versa). This probably
> means you don’t want to be updating puppet itself with “ensure => latest” :)
>

Hi,

point taken, but what will the update plan look like for a large
installation? Will users be required to incorporate MCollective or
similar to be able to upgrade their agents?

Or should they try making all their agents update themselves until each
last one won't talk to the master again, which can then be "safely" updated.

The latter will not fly.

Cheers,
Felix

[Erik Dalén]

I would probably install a Puppet 4 master in parallell with my 3.x master and test Puppet 4.x agents against that. And then when ready to migrate I'd upgrade the agents and point them to use that master. And then decommission the old master when there's no 3.x agents left.

Way more reasonable than trying to upgrade every client and server at the same time, and also gives you opportunity to actually test out the new version before migrating to it.

That's also how I did the 2.7.x to 3.x migration btw.

 

[Chris Price]

On Fri, Nov 14, 2014 at 4:02 PM, Erik Dalén <erik.gus...@gmail.com> wrote:

On Fri Nov 14 2014 at 7:16:07 PM Felix Frank <felix...@alumni.tu-berlin.de> wrote:

On 11/14/2014 07:09 PM, Henrik Lindberg wrote:
>
> Please note that some changes in puppet 4 mean that puppet 3 agents
> won’t be able to talk to puppet 4 masters (or vice-versa). This probably
> means you don’t want to be updating puppet itself with “ensure => latest” :)
>

Or should they try making all their agents update themselves until each
last one won't talk to the master again, which can then be "safely" updated.

 

I would probably install a Puppet 4 master in parallell with my 3.x master and test Puppet 4.x agents against that. And then when ready to migrate I'd upgrade the agents and point them to use that master. And then decommission the old master when there's no 3.x agents left.

 

That's the pattern that I was hoping would work for people.  I was even thinking about hacking up a module that would do the agent upgrade and switch the server host config over as part of one agent run, just to see how that'd work.

 

[Byron Miller]

I for one would love to feel more safe when tempting fate upgrading my puppet infrastructure. I like the idea of rolling upgrades (if upgraded, point here), but i would love to have some sort of process in puppet to handle this. A configuration/feature flag that says "hey puppet, we're upgrading, lets converge". Have some sort of orchestration/reporting capability in this migration state (a light convergence catalog - hey, my file service in upgrade mode is only serving new agent code and if i see you have new agent code, converge to new master and call home)

MCollective does make things slightly easier/more flexible in this regard but i hardly come across many people using it.

i'm mostly thinking out loud remembering my last big upgrading but kicking myself for not writing down my wishlist

-byron

[Chris Price]

On Mon, Nov 17, 2014 at 6:22 AM, Byron Miller <byr...@gmail.com> wrote:

I for one would love to feel more safe when tempting fate upgrading my puppet infrastructure. I like the idea of rolling upgrades (if upgraded, point here), but i would love to have some sort of process in puppet to handle this. A configuration/feature flag that says "hey puppet, we're upgrading, lets converge". Have some sort of orchestration/reporting capability in this migration state (a light convergence catalog - hey, my file service in upgrade mode is only serving new agent code and if i see you have new agent code, converge to new master and call home)

What I'd been thinking (pretty much the same as what Erik said) is that you'd do something like this:

1. Set up a new Puppet 4 master (alongside your Puppet 3 master, as opposed to replacing it).

2. Have a module with a class in it called "puppet4::agent"; it requires a parameter called "puppet4_server", and the implementation of the class is to basically do "package {'puppet': ensure => 4.0 } -> ini_setting {'puppet4_server': file => 'puppet.conf', section => 'agent', setting => 'server', value => $puppet4_server }".

3. You add this class to the classification of the nodes in your fleet at whatever cadence makes sense for you.

I'd like to play around with step 2 there and prove that out a bit more, but it seems like it should work.  Also, since Puppet 4 will have the future parser enabled by default, it seems like it would be highly desirable to move nodes over from 3 to 4 a few at a time and very deliberately, just to make sure that the language changes don't break your existing manifests.  This strategy would encourage that approach over the more risky approach of simply trying to upgrade your whole fleet all at once.

We were also batting around the idea of possibly putting the puppet4 packages in separate yum/apt repos from the puppet3 packages, so that ensure=>latest wouldn't catch anyone by surprise.  We haven't talked through this in much detail yet (I recall that there was a spirited debate around this idea when we bumped from puppet2 to puppet3), so I don't know if it has any legs, but it did come up as an idea.

 

[Peter Meier]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

>> Please note that some changes in puppet 4 mean that puppet 3
>> agents won’t be able to talk to puppet 4 masters (or vice-versa).
>> This probably means you don’t want to be updating puppet itself
>> with “ensure => latest” :)
>>

> point taken, but what will the update plan look like for a large
> installation? Will users be required to incorporate MCollective or
> similar to be able to upgrade their agents?

And how will the release plan of PL looks like regarding 4.0? Will we
end up with having 4.0 in the repositories one day and all the people
that sync the repo (or directly update from it) will just update?

Or will someone take over the duty and run around with a red flag the
weeks before the release, as it is going to be problem?
(Henrik actually just started that... ;-) )

How does actually the time schedule for 4.0 look like?

Currently, the project says beginning of January:

https://tickets.puppetlabs.com/browse/PUP/ 2015/01/07

Just to be able to be aware of breaking changes and one might actually
monitor their repos more closely.

~pete

-----BEGIN PGP SIGNATURE-----

iEYEARECAAYFAlRqdFcACgkQbwltcAfKi392RACgkAZoRpJB2NXRbkjBPPWwz4ho
oQ0AnjypKItk6GUDKvYmyN9dy9q9v/Pr
=P0yE
-----END PGP SIGNATURE-----

[Eric Sorenson]

On Nov 17, 2014, at 2:19 PM, Peter Meier <peter...@immerda.ch> wrote:

And how will the release plan of PL looks like regarding 4.0? Will we
end up with having 4.0 in the repositories one day and all the people
that sync the repo (or directly update from it) will just update?

Or will someone take over the duty and run around with a red flag the
weeks before the release, as it is going to be problem?
(Henrik actually just started that... ;-) )

How does actually the time schedule for 4.0 look like?

Currently, the project says beginning of January:

https://tickets.puppetlabs.com/browse/PUP/ 2015/01/07

Just to be able to be aware of breaking changes and one might actually
monitor their repos more closely.

~pete

Good question. Let me break this off into a separate thread so it gets its own conversation.

Eric Sorenson - eric.s...@puppetlabs.com - freenode #puppet: eric0
puppet platform // coffee // techno // bicycles

[Mike Hendon]

   * Please note that some changes in puppet 4 mean that puppet 3 agents won’t be able to talk to puppet 4 masters (or vice-versa).

This might cause an issue.

I would suggest that the puppet RPMs are renamed to something like puppet4 so that unexpected breakages don't occur.

[John Bollinger]

Yes, we've been down that road before, and it produced considerably uproar.  As I said then, repository maintainers undertake a responsibility to protect their clients from breaking changes.  Ideally, users should take care to protect themselves, as well, but why invite criticism when you don't need to do?  Either introduce puppet 4 to the repositories under a different name, or provide altogether different repositories for it.


John