Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: no parameter named 'quick_check'

225 views
Skip to first unread message

Peter Krawetzky

unread,
Jul 16, 2020, 1:52:06 PM7/16/20
to Puppet Users

I've reviewed sever 500 error posts in here but the answers seem to differ based on the situation.


One of our developers modified code to include a parameter available in httpfile 0.1.9 called quick_check. 

We have two installation of puppetserver one in lab domain and one in production domain.  Neither talk to the other domain.  It is completely isolated to the nodes in each domain.

What's odd is lab works but when they deploy the code to production, it doesn't work and received the 500 error below.  I've compared everything between puppetserver versions, puppet versions, httpfile module versions, etc and nothing is obvious.


This httpfile module is not installed using puppet module install but is placed in the same location as other modules created by the developers.

I've verified the code was deployed correctly to each of the 4 production puppetservers (we use a load balancer to distribute the work) into the environment defined at the node (dev).


Code:

### DOWNLOAD FROM REPO
define oracle::remote_file($remote_location=undef, $mode='0644', $owner='root', $group='root'){

httpfile { "${title}":
ensure => present,
path => "${title}",
source => "$remote_location",
quick_check => true,
# hash => 'hex form SHA2 hash OR an URL to the .sha file with that hash'
}
file{$title:
owner => $owner,
group => $group,
mode => $mode,
require => Httpfile["${title}"],
}
}


Error:

2020-07-15T08:35:15.325976-04:00 myserver puppet-agent[24036]: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: no parameter named 'quick_check' (file: /u01/puppet/dev/modules/oracle/manifests/remote_file.pp, line: 6) on Httpfile[/var/opt/BESClient/LMT/oracle/options_packs_usage_statistics.sql] (file: /u01/puppet/dev/modules/oracle/manifests/remote_file.pp, line: 6) on node myserver.mydomain.com


Any ideas what might be causing this?  Is there some cache not being refreshed on the pupperserver?

Justin Stoller

unread,
Jul 16, 2020, 7:55:51 PM7/16/20
to puppet...@googlegroups.com
It maybe because of a long environment timeout: https://puppet.com/docs/puppet/5.5/environments_creating.html#task-3930
In PE this is set to unlimited by default when using code management. The code manager will then manually evict the cache after a code deployment to ensure that new code is viewable and old code is cached for as long as possible. If you are caching code with a long environment timeout, but not using code management you can also evict the cache by using the environment-cache endpoint:

 HTH, 
Justin

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/886fd9da-c841-4d8b-80f3-d23bc2429e68o%40googlegroups.com.

Peter Krawetzky

unread,
Jul 17, 2020, 8:22:04 AM7/17/20
to Puppet Users
Does this work for Open Source or PE?  My installation is using Open Source.

Also can you provide an example of the actual curl command?  I can't seem to get the exact syntax down for it to work.  

Thanks for the advice.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet...@googlegroups.com.

Peter Krawetzky

unread,
Jul 17, 2020, 10:41:36 AM7/17/20
to Puppet Users
Ok I figured out the curl command but I get this error:

[root@mypuppetserver private_keys]# curl -v --header "Content-Type: application/json" --cert /etc/puppetlabs/puppet/ssl/certs/mypuppetserver.mydomain.com.pem 
--key /etc/puppetlabs/puppet/ssl/private_keys/mypuppetserver.mydomain.com.pem --cacert
* About to connect() to mypuppetserver.mydomain.com port 8140 (#0)
*   Trying xx.xx.xxx.xx...
* Connected to mypuppetserver.mydomain.com (xx.xx.xxx.xx) port 8140 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem
  CApath: none
* NSS: client certificate from file
*       subject: CN=mypuppetserver.mydomain.com
*       start date: Aug 14 15:32:34 2018 GMT
*       expire date: Aug 14 15:32:34 2023 GMT
*       common name: mypuppetserver.mydomain.com
*       issuer: CN=Puppet CA: mypuppetcaserver.mydomain.com
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
*       subject: CN=mypuppetserver.mydomain.com
*       start date: Aug 14 15:32:34 2018 GMT
*       expire date: Aug 14 15:32:34 2023 GMT
*       common name: mypuppetserver.mydomain.com
*       issuer: CN=Puppet CA: mypuppetcaserver.mydomain.com
> DELETE /puppet-admin-api/v1/environment-cache HTTP/1.1
> User-Agent: curl/7.29.0
> Accept: */*
> Content-Type: application/json
>
< HTTP/1.1 403 Forbidden
< Date: Fri, 17 Jul 2020 13:41:37 GMT
< Content-Length: 115
< Server: Jetty(9.4.z-SNAPSHOT)
<
* Connection #0 to host mypuppetserver.mydomain.com left intact
Forbidden request: /puppet-admin-api/v1/environment-cache (method :delete). Please see the server logs for details.[root@mypuppetserver private_keys]#

puppetserver.log entries:
2020-07-17 09:07:45,577 ERROR [qtp2067827614-66] [p.t.a.rules] Forbidden request: 0:0:0:0:0:0:0:1 access to /puppet-admin-api/v1/environment-cache (method :delete) (authenticated: false) denied by rule 'puppetlabs deny all'.
2020-07-17 09:07:45,585 ERROR [qtp2067827614-65] [p.t.a.rules] Forbidden request: 0:0:0:0:0:0:0:1 access to /puppet-admin-api/v1/environment-cache (method :delete) (authenticated: false) denied by rule 'puppetlabs deny all'.
2020-07-17 09:12:02,951 ERROR [qtp2067827614-63] [p.t.a.rules] Forbidden request: xx.xx.xxx.xx access to /puppet-admin-api/v1/environment-cache (method :delete) (authenticated: false) denied by rule 'puppetlabs deny all'.
2020-07-17 09:17:29,677 ERROR [qtp2067827614-61] [p.t.a.rules] Forbidden request: xx.xx.xxx.xx access to /puppet-admin-api/v1/environment-cache (method :delete) (authenticated: false) denied by rule 'puppetlabs deny all'.
2020-07-17 09:41:37,401 ERROR [qtp2067827614-63] [p.t.a.rules] Forbidden request: mypuppetserver.mydomain.com(xx.xx.xxx.xx) access to /puppet-admin-api/v1/environment-cache (method :delete) (authenticated: true) denied by rule 'puppetlabs deny all'.

Justin Stoller

unread,
Jul 17, 2020, 4:05:34 PM7/17/20
to puppet...@googlegroups.com
This is from our auth subsystem which is configured at /etc/puppetlabs/puppetserver/conf.d/auth.conf

It means there was no explicit auth rule for "/puppet-admin-api/v1/environment-cache" so the default "deny all" rule was applied.

You can create a rule in that auth.conf file, it will look something like: https://github.com/puppetlabs/puppetserver/blob/master/ezbake/config/conf.d/auth.conf#L110-L119
The above rule allows any GET request to any url matching "<server>:<port>/puppet/v3/environments*" by a requester presenting a certificate trusted by the Puppet CA and names that rule "puppetlabs environments" for logging purposes. Copy pasting that and substituting "/puppet-admin-api/v1/environment-cache" for the path, giving it a different name, and a method of "delete" should work. If not (or you want to do something different than allow any node to evict the cache) you can see the complete docs on auth rules here: https://github.com/puppetlabs/trapperkeeper-authorization/blob/master/doc/authorization-config.md#rules.

You also might want to confirm that you have an environment_timeout set to something troublesome too. You should be able to run `puppet config --section master --environment <foo> print environment_timeout` to see the environment_timeout for the foo environment.

HTH,
Justin

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

pkraw...@gmail.com

unread,
Aug 24, 2020, 2:06:39 PM8/24/20
to Puppet Users
Justin, I implemented the suggestion you made however after running the curl command against the 2 environments having the issue and receiving the 204 response, the puppet module is still getting the 500 error.  Do you or anyone else have any other suggestions?  Is it possible it's related to ruby and/or java?  Frankly I'm stumped.

Dirk Heinrichs

unread,
Aug 25, 2020, 2:43:03 AM8/25/20
to puppet...@googlegroups.com
Am Montag, den 24.08.2020, 11:06 -0700 schrieb pkraw...@gmail.com:

Justin, I implemented the suggestion you made however after running the curl command against the 2 environments having the issue and receiving the 204 response, the puppet module is still getting the 500 error.  Do you or anyone else have any other suggestions?  Is it possible it's related to ruby and/or java?  Frankly I'm stumped.

Didn't see this earlier, sorry.

The "no parameter named 'xxx'" error can usually be resolved by recreating the metadata for your Puppet environment(s). This can be done on the Puppet master using the following command (for the production environment):

/opt/puppetlabs/bin/puppet generate types --environment production --force

I've added this command to my environment update script after running into this problem myself a few months ago after updating some external modules from the forge.


HTH...

Dirk
--
Dirk Heinrichs
Senior Systems Engineer, Delivery Pipeline
OpenText ™ Discovery | Recommind
Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach
Vertretungsberechtigte Geschäftsführer Gordon Davies, Madhu Ranganathan, Christian Waida, Registergericht Amtsgericht Bonn, Registernummer HRB 10646
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.
signature.asc

pkraw...@gmail.com

unread,
Aug 25, 2020, 12:38:05 PM8/25/20
to Puppet Users
Dirk, why wouldn't puppet just do this automatically when a module changes?  Is there a bug somewhere?

pkraw...@gmail.com

unread,
Aug 25, 2020, 12:41:58 PM8/25/20
to Puppet Users
So a followup to the original question.

As a test we created a simple module on the node which is failing when puppet agent executes.  Running puppet apply, the parameter quick_check is found and the module completes successfully.  So why would puppet apply work and not puppet agent?

Code:

class testmod()

  {

  httpfile { "ansible-2.8.0a1.tar.gz":

    ensure      => present,

    path        => "/u01/testmod/ansible-2.8.0a1.tar.gz",

    source      => "https://mynexus.domain.com/nexus/repository/ae-raw-ansible-group/ansible/ansible-2.8.0a1.tar.gz",

    quick_check => true,

  # hash    => 'hex form SHA2 hash OR an URL to the .sha file with that hash'

           }

  }

 

Here is my run:

[root@node testmod]# puppet apply --modulepath=/home/toor --test -e "include testmod" --verbose    

Justin Stoller

unread,
Aug 25, 2020, 5:09:28 PM8/25/20
to puppet...@googlegroups.com
> why wouldn't puppet just do this automatically when a module changes?

Some background. Puppet's type and provider system modifies the running Puppet instance when they're _loaded_. This causes issues when you try to load multiple conflicting versions of a type in different environments. To work around this we have a kind of header file for your types that Puppet can read w/o actually loading the type. This way Puppet Server can load multiple versions of a type (as long as those different versions are in different environments) and check that each environment uses the type correctly for that version.

The command Dirk gave you, loads those types safely in a separate process and then serializes their parameter information into a format for Puppet to later read that doesn't corrupt its global state. It places this information in the ".resource_types" directory at the root of your environment (like "/etc/puppetlabs/code/environments/production")

Also, in order to speed up Puppet Server catalog compilation, we attempt to cache information like type parameters.

In PE, if you use our built in code management facilities, we generate this type information on every commit (if needed), distribute it to your compilers, and then evict the environment cache so that any new information will be read.

In FOSS, r10k has a config setting to generate this info when it deploys an environment [1].


Now, the error you're ultimately getting involves Puppet Server thinking that you're using the httpfile class wrong. It looks like the "quick_check" field was added in the latest version. So really the first question would be, are you using the latest version in this environment?

Assuming you're doing that you probably either have the environment cache containing an older version of the module (which should be resolved by restarting the server or evicting the environment cache) or an old .resource_types in the root of your environment that should be removed and regenerated like Dirk said. Possibly you could have an older version in a different environment that's being loaded first, but I don't think that'd cause a problem for uncached, new parameters on a type.

HTH,
Justin


--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/80022945-ecdc-43ef-b857-ca2813c37919n%40googlegroups.com.

pkraw...@gmail.com

unread,
Aug 28, 2020, 1:14:38 PM8/28/20
to Puppet Users
Great info but I think I might have found the issue.

So we don't use r10k to deploy code we use a different tool.  But what I found is on the puppet server (master) the httpfile.rb in /opt/puppetlabs/puppet/cache/lib/puppet/type is the older version.  I didn't find any ./resource_types directory in our environment directories so not sure if we are using environment isolation or not.  As part of Justin's suggestion to allow the DELETE option to be valid, I had to restart each of our 4 puppet servers so according to some of this conversation, that should have refreshed the cache right?

What else is odd is the domain where the quick_check parm is work seems to be getting refreshed somehow in /opt/puppetlabs/puppet/cache/lib/puppet subdirectories (just looking at time stamps).  The deploy process works the same in that domain along with the domain where quick_check is not working.  Since the /opt/puppetlabs/bin/puppet generate types --environment production --force operates by environment, could this possible break the environment as well?  These are production boxes I need to run this on and want to make sure I don't break anything.  Also using the environment parm will this then setup environment isolation and do i have to manually manage that each time code is deployed to that environment?

Justin Stoller

unread,
Aug 28, 2020, 2:32:05 PM8/28/20
to puppet...@googlegroups.com
On Fri, Aug 28, 2020 at 10:14 AM pkraw...@gmail.com <pkraw...@gmail.com> wrote:
Great info but I think I might have found the issue.

So we don't use r10k to deploy code we use a different tool.  But what I found is on the puppet server (master) the httpfile.rb in /opt/puppetlabs/puppet/cache/lib/puppet/type is the older version. 

I think puppet/cache is read by the agent not the server. I would expect that to cause problems on applying a catalog from the server, not result in a failed compilation. But barring a .resource_tyeps directory existing in an environment it must be an incorrect version of the httpfile.rb in the server's loadpath.
 
I didn't find any ./resource_types directory in our environment directories so not sure if we are using environment isolation or not.

Just to clarify it will be ".resource_types" with a leading dot and will be hidden by default. [1]

  As part of Justin's suggestion to allow the DELETE option to be valid, I had to restart each of our 4 puppet servers so according to some of this conversation, that should have refreshed the cache right?

Restarting or reloading will evict the in memory cache so if you have a very long environment_timeout it will work as well as doing an eviction of all your environments. It will not however remove any old files in your .resources_types directory. You will need to run the `puppet generate types` command with `--force` for that.
 

What else is odd is the domain where the quick_check parm is work seems to be getting refreshed somehow in /opt/puppetlabs/puppet/cache/lib/puppet subdirectories (just looking at time stamps).  The deploy process works the same in that domain along with the domain where quick_check is not working. 

Can you validate that the failures happen not along a "domain" but along puppet environments. Like all the nodes that use httpfile in production have this failure but those in staging don't have this issue? If you have some succeeding and some failing I would expect this to be the environment to be the condition causing the different behavior.
 
Since the /opt/puppetlabs/bin/puppet generate types --environment production --force operates by environment, could this possible break the environment as well?  These are production boxes I need to run this on and want to make sure I don't break anything.  Also using the environment parm will this then setup environment isolation and do i have to manually manage that each time code is deployed to that environment?

The environment param to `puppet generate types` specifies which environment to act on, without it the command will only act on the environment specified in the puppet.conf for the "main" section (The "main" or "user" sections are almost always unmanaged and adopt the default values which would be "production" for "environment" setting).

Running the command should be a relatively safe command, however I'm going to advocate for anyone "doing it live" on a production box. In PE we deploy this code and run this command in a staging area and then either lock the server while we copy the files over or atomically manage a symlink. If you are using environment caching as well it should be even safer because types will only be read from disk on the first compilation that uses them and then cached in memory after that.

hth,
justin



pkraw...@gmail.com

unread,
Aug 28, 2020, 2:47:26 PM8/28/20
to Puppet Users
Justin, yes it's happening in all environments which leads me to believe it's related to an old copy in /opt/puppetlabs/puppet/cache/lib/puppet/type.  Still trying to wrap my head around why one domain installation is fine and the other domain installation is not.

Here is the contents of that directory which works:
[root@myserverlab type]# pwd
/opt/puppetlabs/puppet/cache/lib/puppet/type
[root@myserverlab type]# ls -al
total 24
drwxr-xr-x 2 root root   77 Jul 16 16:04 .
drwxr-xr-x 6 root root   61 Feb  3  2020 ..
-rw-r--r-- 1 root root 1706 Jul 16 16:04 anchor.rb
-rw-r--r-- 1 root root 6921 Jul 16 16:04 file_line.rb
-rw-r--r-- 1 root root 1863 May  1  2017 httparch.rb
-rw-r--r-- 1 root root 6957 Jul 13 17:04 httpfile.rb
[root@myserverlab type]#

Here is the contents of the directory that doesn't work:
[root@myserverprod type]# pwd
/opt/puppetlabs/puppet/cache/lib/puppet/type
[root@myserverprod type]# ls -al
total 24
drwxr-xr-x 2 root root   77 Sep 30  2018 .
drwxr-xr-x 6 root root   61 Apr 24  2017 ..
-rw-r--r-- 1 root root 1752 Sep 30  2018 anchor.rb
-rw-r--r-- 1 root root 7113 Sep 30  2018 file_line.rb
-rw-r--r-- 1 root root 1863 May 15  2017 httparch.rb
-rw-r--r-- 1 root root 6357 Apr 24  2017 httpfile.rb
[root@myserverprod type]#

You can clearly see the date and size difference of httpfile.rb.  I quadruple checked the puppet module directory on the prod server and the code does have the quick_check parm.  For some reason it is just not refreshing the server cache.  Both domains have a value of 0 for environment_timeout for each environment.

pkraw...@gmail.com

unread,
Sep 15, 2020, 8:24:52 AM9/15/20
to Puppet Users
So I've done some research on the puppet generate types command.  I'm seeing many different results from not having issues to causing issues with puppet apply and puppet agent executions.  If I was to run this command and things go wrong, how do you reverse it?  Remove the .resource_types directory?

Justin Stoller

unread,
Sep 15, 2020, 1:45:42 PM9/15/20
to puppet...@googlegroups.com
On Tue, Sep 15, 2020 at 5:25 AM pkraw...@gmail.com <pkraw...@gmail.com> wrote:
So I've done some research on the puppet generate types command.  I'm seeing many different results from not having issues to causing issues with puppet apply and puppet agent executions. 
If I was to run this command and things go wrong, how do you reverse it?  Remove the .resource_types directory?

Yep!
 

Josh Cooper

unread,
Sep 15, 2020, 1:48:47 PM9/15/20
to puppet...@googlegroups.com


On Tue, Sep 15, 2020 at 5:25 AM pkraw...@gmail.com <pkraw...@gmail.com> wrote:
So I've done some research on the puppet generate types command.  I'm seeing many different results from not having issues to causing issues with puppet apply and puppet agent executions.

We fixed an issue in 6.18.0 (https://tickets.puppetlabs.com/browse/PUP-9602) that caused "puppet apply" to fail in some cases if "puppet generate types" had been run previously.

pkraw...@gmail.com

unread,
Sep 15, 2020, 1:53:53 PM9/15/20
to Puppet Users
Thanks Justin/Josh.  Josh, I did come across this during my recent searches on trying to resolve my original issue.
Reply all
Reply to author
Forward
0 new messages