puppetserver puppetdb and storeconfigs=true

[Steph Gosling]

Hi all,

I'm attempting a migration from a PuppetDB 2.x and rack Puppet 3.8.1
install over to the all new 'pc1' puppetserver puppet-agent PuppetDB v3
stack[0].

On the two nodes I've tried so far (the master itself and a test node)
I'm getting the following error:

Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node <node>

This is broadly the same error as PDB-949[1] and at least one other user
[2] has had this issue with the same setup. If I disable storeconfigs
then my manifests run with both agents just fine. With storeconfigs
enabled (and even if the environment is completely empty) this is what
puppetserver logs

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:58:in `process'
file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/puppet-server-lib/puppet/server/master.rb:39:in `handleRequest'
Puppet$$Server$$Master_35370687.gen:13:in `handleRequest'
request_handler_core.clj:274:in `invoke'
request_handler_service.clj:14:in `handle_request'
request_handler.clj:3:in `invoke'
request_handler.clj:3:in `invoke'
core.clj:626:in `invoke'
core.clj:2468:in `doInvoke'
master_core.clj:47:in `invoke'
ring.clj:22:in `invoke'
ring.clj:13:in `invoke'
comidi.clj:267:in `invoke'
ringutils.clj:106:in `invoke'
ringutils.clj:62:in `invoke'
ringutils.clj:68:in `invoke'
ringutils.clj:118:in `invoke'
jetty9_core.clj:408:in `invoke'
2015-07-23 17:09:45,728 ERROR [puppet-server] Puppet Attempt to assign to a reserved variable name: 'trusted' on node <node>
2015-07-23 17:09:45,737 ERROR [puppet-server] Puppet Attempt to assign to a reserved variable name: 'trusted' on node <node>

PuppetDB logs only successful updating of facts and the report:

2015-07-23 17:09:45,224 INFO [p.p.command] [64604053-f1cb-4588-8c09-e7fc95c6b348] [replace facts] <node>
2015-07-23 17:09:45,833 INFO [p.p.command] [4f7df7c3-715a-4b97-8eff-917329667a9f] [store report] puppet v4.2.1 - <node>

The above URLs both allude to trusted facts clobbering other facts
and/or timing issues. I definitely don't have timing issues (given one
of the agents is colocated with its master).

I've completely purge all of the old Puppet installations and even
dropped the Postgres DB so I think my stack reall should work.

Does anyone have any pointers as to how I should debug this further?

Cheers,

Steph

[0] Master and agent nodes are all Ubuntu 14.04, packages are all the
official puppetlabs-release-pc1 packages, specifically
puppetdb-3.0.1-1puppetlabs1
puppetdb-termini-3.0.1-1puppetlabs1
puppetserver-2.1.1-1puppetlabs1
puppet-agent-1.2.2-1trusty

[1] https://tickets.puppetlabs.com/browse/PDB-949
[2] http://ask.puppetlabs.com/question/17184/attempt-to-assign-to-a-reserved-variable-name-trusted/

--
Steph Gosling <st...@chuci.org>

[Ken Barber]

The problem is, that if you're contacting PuppetDB in this case for
your fact information, something is wrong. The 'bug' is that if PDB is
referenced for this information it tries to clobber trusted_facts, but
you should never get this far under a compilation workflow.

This can sometimes be caused by a timing issue, due to the cache
looking like its expired (even though it was just written) so it
reaches out to the PDB terminus for its answer.

The other cause can be a misconfigured routes.yaml, or some other
workflow problem. What does your routes.yaml look like? This can be
misconfigured to always use PDB for that information, which is bad.
Facts should come from agents, not from a cache inside PDB during
compilation :-).

Also - double check your facts yaml cache, wipe it if necessary (or
back it up just in case) to ensure there is nothing stale. This is
usually not necessary and probably a bit brute force, but worth a
check if you're game.

ken.

[Kevin Corcoran]

Hi Steph,

Just to be sure, you've ensured that the clock is correct on all of your nodes, right?  I don't have a great understanding of this problem, but from what I know, the underlying issue is significant clock skew (> 30 minutes) between machines.

Failing that, there's also https://tickets.puppetlabs.com/browse/PUP-3449 - although I'm not sure if that's any help.

- Kevin

[Steph Gosling]

Ken, Kevin, good evening.

On Thu, 23 Jul 2015 17:58:20 +0100
Ken Barber <k...@puppetlabs.com> wrote:

> The problem is, that if you're contacting PuppetDB in this case for
> your fact information, something is wrong. The 'bug' is that if PDB is
> referenced for this information it tries to clobber trusted_facts, but
> you should never get this far under a compilation workflow.
>
> This can sometimes be caused by a timing issue, due to the cache
> looking like its expired (even though it was just written) so it
> reaches out to the PDB terminus for its answer.

Kevin also suggested skew could cause this but I can rule that out as
the master was failing on itself.

> The other cause can be a misconfigured routes.yaml, or some other
> workflow problem. What does your routes.yaml look like? This can be
> misconfigured to always use PDB for that information, which is bad.
> Facts should come from agents, not from a cache inside PDB during
> compilation :-).

BINGO

I had no routes.yaml (the 3.8.1 yaml was in the wrong place in the move
not only in software but to environments too). Putting that back into
$confdir and with a puppetserver restart all is well again.

Thanks both for your time and help.

Cheers,

Steph

--
Steph Gosling <st...@chuci.org>