Jira (PUP-9055) Compile catalogs on demand with user-specified facts

4 views
Skip to first unread message

Josh Cooper (JIRA)

unread,
Aug 11, 2018, 12:59:03 AM8/11/18
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Puppet / Bug PUP-9055
Compile catalogs on demand with user-specified facts
Change By: Josh Cooper
Summary: puppetserver should have the ability to compile Compile catalogs on demand with user-specified facts
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Josh Cooper (JIRA)

unread,
Aug 11, 2018, 1:00:06 AM8/11/18
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Team: Server Coremunity

Josh Cooper (JIRA)

unread,
Aug 11, 2018, 1:00:07 AM8/11/18
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-9055
 
Re: Compile catalogs on demand with user-specified facts

Moving to PUP since it will be implemented either in the puppet catalog application or catalog-preview. If it's the latter, it will be moved to the MODULES project.

Rob Braden (JIRA)

unread,
Aug 13, 2018, 5:07:03 PM8/13/18
to puppe...@googlegroups.com
Rob Braden updated an issue
 
Change By: Rob Braden
Sprint: Platform Core Grooming

Rob Braden (JIRA)

unread,
Aug 13, 2018, 5:08:03 PM8/13/18
to puppe...@googlegroups.com
Rob Braden updated an issue
Change By: Rob Braden
Sprint: Platform Core Grooming Hopper

Rob Braden (JIRA)

unread,
Aug 13, 2018, 5:09:03 PM8/13/18
to puppe...@googlegroups.com
Rob Braden updated an issue
Change By: Rob Braden
Sprint: Platform Core Hopper

Rob Braden (JIRA)

unread,
Aug 13, 2018, 5:11:02 PM8/13/18
to puppe...@googlegroups.com

Josh Cooper (JIRA)

unread,
Aug 30, 2018, 5:33:04 PM8/30/18
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-9055
 
Re: Compile catalogs on demand with user-specified facts

Adding a facts option to the catalog find action is a little bit harder than expected, because the application is implemented as a face, and find is one of the standard actions. Will need to research if it's easy to override the find behavior, though honestly the easiest thing might be to rewrite the useful parts of puppet catalog as a non-face application.

Kenn Hussey (JIRA)

unread,
Sep 10, 2018, 9:30:08 AM9/10/18
to puppe...@googlegroups.com
Kenn Hussey updated an issue
 
Change By: Kenn Hussey
Flagged: Impediment

Jorie Tappa (JIRA)

unread,
Sep 10, 2018, 4:44:04 PM9/10/18
to puppe...@googlegroups.com
Jorie Tappa updated an issue
Change By: Jorie Tappa
Component/s: DOCS

Jorie Tappa (JIRA)

unread,
Sep 10, 2018, 4:44:05 PM9/10/18
to puppe...@googlegroups.com
Jorie Tappa updated an issue
Change By: Jorie Tappa
Fix Version/s: PUP 6.0.0

Rob Braden (JIRA)

unread,
Sep 10, 2018, 4:45:04 PM9/10/18
to puppe...@googlegroups.com
Rob Braden updated an issue
Change By: Rob Braden
Fix Version/s: PUP 6.y

Kenn Hussey (JIRA)

unread,
Sep 13, 2018, 9:46:06 AM9/13/18
to puppe...@googlegroups.com
Kenn Hussey updated an issue
Change By: Kenn Hussey
Flagged: Impediment

Henrik Lindberg (JIRA)

unread,
Sep 18, 2018, 10:28:05 AM9/18/18
to puppe...@googlegroups.com
Henrik Lindberg commented on Bug PUP-9055
 
Re: Compile catalogs on demand with user-specified facts

I think the easiest is to use the PAL since it now supports catalog compilation. Since master --compile was something I have been using a lot, I hacked something up using PAL when master --compile was dropped. Here is what I am using for small snippets at the moment:

Puppet.initialize_settings
 
Puppet[:rich_data]=true
 
result = Puppet::Pal.in_tmp_environment('production', modulepath: [], facts: {}) do |pal|
 
  pal.with_catalog_compiler do |c|
 
    c.evaluate_string(ARGV[0])
 
    c.with_json_encoding { |encoder| encoder.encode }
 
  end
 
end
 
puts result

For example used as:

bundle exec ruby to_catalog.rb 'notify { test: message => "we have a catalog"}'

Obviously the example is very simplistic - but all the options are there to feed in facts from any source, to use `code` or `manifest`, specify a configured environment etc. So to write a new application it is basically the CLI UX work on that application that is needed.

Josh Cooper (JIRA)

unread,
Sep 24, 2018, 6:23:04 PM9/24/18
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-9055

Some cons of using the PAL approach:

  • It must be run on the master (same as puppet master --compile))
  • binary content is handled differently for agents that don't support richdata (PSON vs Binary data type)
  • It doesn't support per-environment configuration (rich-data, manifest, modulepath, static catalogs)
  • It doesn't perform the same transformations that the compiler terminus does:
    • removing virtual resources
    • inlining static content
    • adding server and trusted facts
    • collecting performance metrics

Some cons of using the puppet catalog find approach:

  • It writes facts to puppetdb as a side-effect
  • Need to have a clientcert for the catalog being requested

One big problem is the ability to request a catalog in an arbitrary environment. In the PAL case, we can control that, however, it's misleading as the agent won't necessary be classified in that environment when it next runs. In the catalog find case, we'll use whatever environment the agent is classified for, which may be different than what we requested.

Henrik Lindberg (JIRA)

unread,
Sep 24, 2018, 11:54:05 PM9/24/18
to puppe...@googlegroups.com

Some cons of using the PAL approach:

  • It must be run on the master (same as puppet master --compile))

How is this a con if it is the same? Don't get this point. (Actually I do, but if it is to be a replacement for master --compile then REST isn't strictly needed is it? Running over REST also makes this less useful - see comment below for "catalog find").

  • binary content is handled differently for agents that don't support richdata (PSON vs Binary data type)

Easy to add "withPsonEncoding". I assume you want to control rich/not-rich - my sample had rich_data hardcoded.

  • It doesn't support per-environment configuration (rich-data, manifest, modulepath, static catalogs)

It really should if PAL is called to use an existing environment on disk - like "production". Complicates things a lot though as I would rather not make
PAL more dependent on the settings system. (This to be able to one day swap it out). My sample runs in a tmp env, it can naturally not have settings like that.

  • It doesn't perform the same transformations that the compiler terminus does:
    • removing virtual resources

I thought it did that. It should - if not its a bug.

    • inlining static content
    • adding server and trusted facts

It does use trusted facts, you have to give them in an actual node - the PAL part does not know things like how the node was obtained and how the fact where authenticated. I thought it did add the server facts already. If not, then that is a bug.

    • collecting performance metrics

What happens if you try? Should be fixed in general in PAL I suppose if it does not work.

Some cons of using the puppet catalog find approach:

  • It writes facts to puppetdb as a side-effect
  • Need to have a clientcert for the catalog being requested

Additional con: It requires having a running and configured Puppet Server to get a catalog. That is much slower and there is no possibility to test "just the Ruby part of compilation" to eliminate problems with JRuby/MRI or the state of a running Puppet Server. (The reasons why I wrote the small PAL thing to quickly get a catalog).

One big problem is the ability to request a catalog in an arbitrary environment. In the PAL case, we can control that, however, it's misleading as the agent won't necessary be classified in that environment when it next runs. In the catalog find case, we'll use whatever environment the agent is classified for, which may be different than what we requested.

Yeah, that is confusing - same confusion with lookup; use env node is in, or env node will be in.

Josh Cooper (JIRA)

unread,
Sep 26, 2018, 8:55:06 PM9/26/18
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Change By: Josh Cooper
Fix Version/s: PUP 6.y
Fix Version/s: PUP 6.1.0

Josh Cooper (JIRA)

unread,
Oct 8, 2018, 5:42:08 PM10/8/18
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Sprint: Coremunity Hopper Grooming

Kenn Hussey (JIRA)

unread,
Dec 10, 2018, 10:29:03 AM12/10/18
to puppe...@googlegroups.com
Kenn Hussey commented on Bug PUP-9055
 
Re: Compile catalogs on demand with user-specified facts

Josh Cooper this needs to land before EOD on Wednesday in order to make the cut for Puppet 6.1.

Josh Cooper (JIRA)

unread,
Dec 10, 2018, 4:52:06 PM12/10/18
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Fix Version/s: PUP 6.1.0
Fix Version/s: PUP 6.y

Maggie Dreyer (JIRA)

unread,
Dec 19, 2018, 2:01:03 PM12/19/18
to puppe...@googlegroups.com
Maggie Dreyer commented on Bug PUP-9055
 
Re: Compile catalogs on demand with user-specified facts

We are looking at adding several new catalog compilation endpoints that compile more arbitrary catalogs (rather than just the one for the requesting node). This feels somewhat related, and the code that winds up being used to do that might also be exposable via CLI.

Since webrick/rack were fully removed in Puppet 6 even without this capability, I'm going to move this to another epic and close PUP-3834 out.

Maggie Dreyer (JIRA)

unread,
Mar 18, 2019, 1:28:05 PM3/18/19
to puppe...@googlegroups.com
Maggie Dreyer commented on Bug PUP-9055

In Puppet Server 6.3.0, we are releasing the endpoint mentioned above. See details on SERVER-2434.

Alan Smith (JIRA)

unread,
Mar 26, 2019, 7:28:04 PM3/26/19
to puppe...@googlegroups.com
Alan Smith commented on Bug PUP-9055

Our upgrade from 4 to 6 is on hold because `puppet master --compile` is gone, as we rely on it heavily to compile catalogs offline.

I see the updates about a http endpoint on puppetserver but creating temporary puppetservers is both impractical and slow (`puppet master --compile` does not have the startup overhead that puppetserver does) on a developer's workstation. Our usage now involves repeatedly calling `puppet master --compile` with different facts/ENC data and puppet code, offline. This is both cheap on resources and fast (no puppetserver setup/teardown costs). If it was rearchitected to work offline with puppetserver, it would be neither cheap nor fast.

What are the alternatives to puppetserver and its use of JRuby? Can the code needed for `puppet master --compile` specifically be put back without the rest of the webrick/rack support code?

Justin Stoller (JIRA)

unread,
Mar 26, 2019, 7:51:03 PM3/26/19
to puppe...@googlegroups.com

Just to clarify: with the API call you don't have to stand up a new puppetserver every time you would have called `puppet master --compile` you can leave a puppetserver running and, as long as it has access to the code needed to compile, continually hit the API for additional catalogs.

Alan Smith (JIRA)

unread,
Mar 28, 2019, 4:46:05 PM3/28/19
to puppe...@googlegroups.com
Alan Smith commented on Bug PUP-9055

Justin Stoller Thanks for the clarification, but the issue remains of needing a long-running puppetserver available to perform what used to be possible with a one-off ruby CLI. Starting, stopping, and configuring a puppetserver for this task involves very different configuration/supporting code than does a simple CLI.

Josh Cooper (JIRA)

unread,
Apr 9, 2019, 2:20:05 AM4/9/19
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-9055

I've added a PR https://github.com/puppetlabs/puppet/pull/7476 which provides the ability to compile a catalog for an arbitrary node with arbitrary facts (as YAML). It must be executed on the puppetserver host (or otherwise have access to the same environments, modules, manifests, hiera data, etc that puppetserver does). For example:

compile catalog for the current node

$ bx puppet catalog compile
 
bx puppet catalog compile
 
Notice: Compiled catalog for localhost in environment production in 0.20 seconds
 
{
 
  "tags": [
 
    "settings",
 
    "default",
 
    "node"
 
  ],
 
  "name": "localhost",
 
  ...
 
}

compile catalog for a different node (pluto) using pre-generated facts

$ head pluto.yaml
 
--- !ruby/object:Puppet::Node::Facts
 
name: pluto
 
values:
 
  puppetversion: 6.5.0
 
  kernelversion: 18.2.0
 
$ bx puppet catalog compile --facts pluto.yaml --certname pluto
 
Notice: Compiled catalog for pluto in environment production in 0.16 seconds
 
{
 
  "tags": [
 
    "settings",
 
    "default",
 
    "node"
 
  ],
 
  "name": "pluto",
 
  ...
 
}

Trevor Vaughan (JIRA)

unread,
Apr 9, 2019, 8:20:04 AM4/9/19
to puppe...@googlegroups.com

Josh Cooper I'm not sure if this can be done, but would it be possible to generate a fact template based on a target manifest set?

Basically, if I need a bunch of facts, it would be nice to have a 'fill in the blanks' YAML file that could be output by puppet itself.

Ideally, you could amend the file so that it adds on new facts as the compile gets further along.

I'm assuming that this would be an extension to the Facter hooks in Puppet.

Alan Smith (JIRA)

unread,
Apr 9, 2019, 1:22:05 PM4/9/19
to puppe...@googlegroups.com
Alan Smith commented on Bug PUP-9055

Josh Cooper I've also [submitted a PR](https://github.com/puppetlabs/puppet/pull/7477), seconds ago, that takes a slightly different approach: it replicates the old `puppet master --compile` behavior by setting the "run mode". I think it gives everyone who relied on `master --compile` something to work with and migrate to. It would do everything I needed it to do. Let me know what you think!

Josh Cooper (JIRA)

unread,
Apr 19, 2019, 12:25:05 PM4/19/19
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Sprint: Coremunity Grooming Platform Core KANBAN

Josh Cooper (JIRA)

unread,
Apr 19, 2019, 12:25:06 PM4/19/19
to puppe...@googlegroups.com
Josh Cooper assigned an issue to Josh Cooper
Change By: Josh Cooper
Assignee: Josh Cooper

Josh Cooper (JIRA)

unread,
Apr 19, 2019, 12:30:08 PM4/19/19
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Sprint: Platform Core KANBAN

Josh Cooper (JIRA)

unread,
Apr 19, 2019, 12:30:15 PM4/19/19
to puppe...@googlegroups.com

Josh Cooper (JIRA)

unread,
Apr 23, 2019, 11:03:03 PM4/23/19
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-9055
 
Re: Compile catalogs on demand with user-specified facts

Watchers on the ticket: Alan Smith contributed a PR to restore the functionality of puppet master --compile using puppet catalog compile, and it's been merged to master in https://github.com/puppetlabs/puppet/commit/5d30e1e091199729b9d10dbe7c1566b4bcd7cded. It has the same limitation that puppet master had where you have to "preload" the facts for the node being compiled by saving a facts file with the name $certname.yaml. But at least we've restored the capabilities that were removed when puppet master --compile was removed.

If folks are interested in explicitly passing the facts to use on the command line, let's file a new ticket and link to this one.

Josh Cooper (JIRA)

unread,
Apr 23, 2019, 11:03:04 PM4/23/19
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Fix Version/s: PUP 6.y
Fix Version/s: PUP 6.5.0

Josh Cooper (JIRA)

unread,
Apr 23, 2019, 11:05:05 PM4/23/19
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Release Notes Summary: The "puppet catalog compile" action will compile a catalog in the same way that the "puppet master --compile" application did, before being removed in Puppet 6. The command must be run on the puppetserver with access to modules, hiera data, etc
Release Notes: Enhancement

Heston Hoffman (JIRA)

unread,
Jun 11, 2019, 7:54:04 PM6/11/19
to puppe...@googlegroups.com
Heston Hoffman updated an issue
Change By: Heston Hoffman
Labels: resolved-issue-added

David Sandilands (Jira)

unread,
May 6, 2022, 8:31:02 AM5/6/22
to puppe...@googlegroups.com
David Sandilands commented on Bug PUP-9055
 
Re: Compile catalogs on demand with user-specified facts

I have a question from JPMC relating to this ticket, I think I am misunderstanding what can be done here, can I run
puppet catalog compile --run_mode master pe-server-davidsand-0-ff18d2.ty4x0b4hbguu1p40qhokw3zp2f.xx.internal.cloudapp.net
and as long as I have a  pe-server-davidsand-0-ff18d2.ty4x0b4hbguu1p40qhokw3zp2f.xx.internal.cloudapp.net.json file with facts it should behave like puppet compile master command?
This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages