Newbie: Could not retrieve catalog from remote server: Error 400 on SERVER

[Dan M]

Hi

I am setting up first sandbox environment with Puppet and facing (probably) config issue.
Any help is highly appreciated.

Setup covers two boxes:

• OS: RedHatEnterpriseServer 6.4
• Master node (ip-10-0-8-10) and Agent node (ip-10-0-8-11)
• Puppet version: 3.2.3

Two standard modules have been installed on the Master node only:

• $> sudo puppet module install puppetlabs/ntp
• $> sudo puppet module install puppetlabs/apache

puppet.conf is provided at [1]

site.pp from /biz/puppet/hieradata/devbranch/site.pp:

node ip-10-0-8-11 {
  include role::knox_portal
}

Role from /biz/puppet/modules/role/manifests/knox_portal.pp:

class role::knox_portal {
  include profile::web_server
}

Profile from /biz/puppet/modules/profile/manifests/web_server.pp

class profile::web_server {
  class { "apache": }
}

While running sudo puppet agent --test on Agent node, I am getting:

[me@ip-10-0-8-11 puppet-skeleton]$ sudo puppet agent --test
Info: Retrieving plugin
Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
Info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: syntax error on line 13, col 10: `' at /biz/puppet/modules/profile/manifests/web_server.pp:2 on node ip-10-0-8-11
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Thank you
Dan

[1] puppet.conf
[main]
    server = ip-10-0-8-10
    certname = ip-10-0-8-10
    modulepath = /etc/puppet/modules:/usr/share/puppet/modules:/biz/puppet/modules

    manifest = /biz/puppet/hieradata/$environment/site.pp
    logdir = /var/log/puppet
    rundir = /var/run/puppet
    ssldir = $vardir/ssl
 
    dns_alt_names = ip-10-0-8-10, puppet
    report = true
    reports = store, http

    reporturl = http://ip-10-0-8-10:3000/reports/upload
    pluginsync = true
 
[master]
    hiera_config = /biz/puppet/hiera/hiera.yaml

[agent]
    classfile = $vardir/classes.txt
    localconfig = $vardir/localconfig
 
    certname = ip-10-0-8-11
    dns_alt_names = ip-10-0-8-11

    report = true
    archive_files = true
    environment = devbranch

[Pablo Carranza]

What's on line 13, col 10 of /biz/puppet/modules/profile/manifests/web_server.pp?

-Pablo

[Dan M]

There is no line 13 :)
Please, see the output below:

[me@ip-10-0-8-10 ~]$ cat /biz/puppet/modules/profile/manifests/web_server.pp

[Dan M]

In a simplified form, the issue occurs when "include" statement appears in the script.
For instance, below script executes just fine without <include apache>.
It still runs on the Master node (with warnings) with <include apache> but fails on Agent node:

/biz/puppet/hieradata/devbranch/site.pp:

node default {

  include apache

  file { "/etc/sudoers":
    owner => root, group => root, mode => 660
  }
}

Above site.pp executes on Master node:

[me@ip-10-0-8-10 ~]$ sudo puppet apply --verbose /biz/puppet/hieradata/devbranch/site.pp
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/root_home.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/pe_version.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/facter_dot_d.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/puppet_vardir.rb
Info: Loading facts in /etc/puppet/modules/concat/lib/facter/concat_basedir.rb
Warning: Could not retrieve fact fqdn
Warning: Host is missing hostname and/or domain: ip-10-0-8-10
Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/root_home.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/pe_version.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/facter_dot_d.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/puppet_vardir.rb
Info: Loading facts in /etc/puppet/modules/concat/lib/facter/concat_basedir.rb
Info: Applying configuration version '1375827325'
Error: Could not start Service[httpd]: Execution of '/sbin/service httpd start' returned 1:
Error: /Stage[main]/Apache/Service[httpd]/ensure: change from stopped to running failed: Could not start Service[httpd]: Execution of '/sbin/service httpd start' returned 1:
Notice: Finished catalog run in 4.04 seconds

But fails for the Agent node:

[me@ip-10-0-8-11 puppet-skeleton]$ sudo puppet agent --test
Info: Retrieving plugin
Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
Info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: syntax error on line 13, col 10: `' at /biz/puppet/hieradata/devbranch/site.pp:3 on node ip-10-0-8-11

Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Any ideas?

[jcbollinger]

That compilation fails when your node declares class "apache" but not when it doesn't seems to pin the blame on that class.  You should run the master with debug logging enabled, and check whether its log has anything more illuminating to say.

Also, you should check whether there is more than one 'apache' module in your modulepath.  Inasmuch as the modulepath does not vary with environment I don't think that should be an issue, but all the same, I would like to rule it out.

Additionally, verify that every directory in your module path, everything in each one, and every directory in the paths to them are readable by the puppet user, and that all the directories are traversable by that user.  The master process does not normally run with elevated privilege, so you do have to pay attention to file and directory permissions.  Even if the initial setup were correct, you might conceivably cause yourself trouble later by performing module installs with privilege, as you in fact did.


John

[Rilindo Foster]

The syntax appears to be correct, from what I see. Being that this Apache module came from PuppetLabs, I think it is safe to assume that the module is robust. I would try to try enable full debugging on both the agent and the master. 

Also, on the agent, try to walk through the installation of Apache on the node. Sometimes what appears to be Puppet error is actually caused by the application. For example, I had an issue with nginx not starting up because of an error in the service code here:

class nginx::service (
  $svc = $nginx::params::svc,
) {
  
  service { $svc:
    enable => true,
    ensure => "running",
    hasrestart => true,
    hasstatus => true,
    require => Class['nginx::install'],
  }
  
}

After verifying and modifying the code, I went and tried to start up nginx manually. Turns out that default configuration was missing a semi-colon on one of the lines. That caused nginx not to start - an error that wasn't exposed easily in the system log (where puppet usually logs by default). Fixing that misconfiguration resolved the nginx start problem. :)

- Rilindo

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To post to this group, send email to puppet...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Dan M]

Hi John

Thank you for helping me.
Today I have fixed several configuration issues of Master's start-up (such as SELinux warning, missing domain names, etc), however the problem with <include apache> persist.

To be on the safe side, I am running both Master and Agent from sudo. At the same time, I have added puppet user to root group (just to eliminate possibility of inaccessible paths):

[me@ip-10-0-8-11 puppet-skeleton]$ groups puppet
puppet : puppet root

Please, find Master debug log below:

[me@ip-10-0-8-10 ~]$ sudo puppet master --no-daemonize --debug

<START_UP_MESSAGES - NO_WARNINGS>

Notice: Starting Puppet master version 3.2.3
Info: Inserting default '~ ^/catalog/([^/]+)$' (auth true) ACL
Info: Inserting default '~ ^/node/([^/]+)$' (auth true) ACL
Info: Inserting default '/file' (auth ) ACL
Info: Inserting default '/certificate_revocation_list/ca' (auth true) ACL
Info: Inserting default '~ ^/report/([^/]+)$' (auth true) ACL
Info: Inserting default '/certificate/ca' (auth any) ACL
Info: Inserting default '/certificate/' (auth any) ACL
Info: Inserting default '/certificate_request' (auth any) ACL
Info: Inserting default '/status' (auth true) ACL
Info: Caching node for ip-10-0-8-11.sampup.com
Debug: No plugins found in subpath '/etc/puppet/modules/ntp/lib' (file / directory does not exist)
Debug: No plugins found in subpath '/biz/puppet/modules/role/lib' (file / directory does not exist)
Debug: No plugins found in subpath '/biz/puppet/modules/profile/lib' (file / directory does not exist)
Debug: No plugins found in subpath '/biz/puppet/modules/README/lib' (file / directory does not exist)
Debug: No plugins found in subpath '/biz/puppet/modules/networking/lib' (file / directory does not exist)
Debug: No plugins found in subpath '/biz/puppet/modules/users/lib' (file / directory does not exist)
Info: Caching node for ip-10-0-8-11.sampup.com
Debug: importing '/biz/puppet/modules/role/manifests/init.pp' in environment devbranch
Debug: importing '/biz/puppet/modules/role/manifests/knox_portal.pp' in environment devbranch
Debug: Automatically imported role::knox_portal from role/knox_portal into devbranch
Debug: importing '/biz/puppet/modules/profile/manifests/init.pp' in environment devbranch
Debug: importing '/biz/puppet/modules/profile/manifests/base.pp' in environment devbranch
Debug: Automatically imported profile::base from profile/base into devbranch
Debug: importing '/biz/puppet/modules/networking/manifests/init.pp' in environment devbranch
Debug: Automatically imported networking from networking into devbranch
Debug: importing '/biz/puppet/modules/users/manifests/init.pp' in environment devbranch
Debug: Automatically imported users from users into devbranch
Debug: importing '/biz/puppet/modules/profile/manifests/web_server.pp' in environment devbranch
Debug: Automatically imported profile::web_server from profile/web_server into devbranch
Debug: importing '/etc/puppet/modules/apache/manifests/init.pp' in environment devbranch
Debug: Automatically imported apache from apache into devbranch
Debug: importing '/etc/puppet/modules/apache/manifests/params.pp' in environment devbranch
Debug: Automatically imported apache::params from apache/params into devbranch
Error: Puppet::Parser::AST::Resource failed with error ArgumentError: syntax error on line 13, col 10: `' at /biz/puppet/modules/profile/manifests/web_server.pp:3 on node ip-10-0-8-11.sampup.com
Error: Puppet::Parser::AST::Resource failed with error ArgumentError: syntax error on line 13, col 10: `' at /biz/puppet/modules/profile/manifests/web_server.pp:3 on node ip-10-0-8-11.sampup.com
Error: Puppet::Parser::AST::Resource failed with error ArgumentError: syntax error on line 13, col 10: `' at /biz/puppet/modules/profile/manifests/web_server.pp:3 on node ip-10-0-8-11.sampup.com
Debug: Finishing transaction 69970875958340
Debug: Received report to process from ip-10-0-8-11.sampup.com
Debug: Processing report from ip-10-0-8-11.sampup.com with processor Puppet::Reports::Store

Agent node log:

[me@ip-10-0-8-11 puppet-skeleton]$ sudo puppet agent --test

Info: Caching certificate for ip-10-0-8-11.sampup.com
Info: Caching certificate_revocation_list for ca

Info: Retrieving plugin
Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
Info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: syntax error on line 13, col 10: `' at /biz/puppet/modules/profile/manifests/web_server.pp:3 on node ip-10-0-8-11.sampup.com

Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

And the file in question:

[me@ip-10-0-8-10 puppet-skeleton]$ cat /biz/puppet/modules/profile/manifests/web_server.pp
class profile::web_server {
  # Configuration for all webservers
  class { "apache": }
  #class { "php": }
  #class { "memcache": }
}

What else can I do to understand the issue?

Dan

[Dan M]

I think I have found the fix... and it looks a little bit magical... Ruby made...

I was able to solve the problem by removing /etc/puppet/manifests folder.
It was empty at the moment of removal, but later on gets recreated by puppet.
Anyway - Agent finally works as expected.

Also I have commented out hiera_config variable.
Here is my final puppet.conf:

[main]
    server = ip-10-0-8-10.sampup.com
    certname = ip-10-0-8-10.sampup.com
   
    vardir = /var/lib/puppet
    factpath = $vardir/lib/facter
    templatedir = $confdir/templates
   
    # The Puppet log directory.
    # The default value is '$vardir/log'.
    logdir = /var/log/puppet
 
    # Where Puppet PID files are kept.
    # The default value is '$vardir/run'.
    rundir = /var/run/puppet
 
    # Where SSL certificates are kept.
    # The default value is '$confdir/ssl'.
    ssldir = $vardir/ssl
 
#    reports = store, http
#    reporturl = http://ip-10-0-8-10:3000/reports/upload
 
[master]
    dns_alt_names = ip-10-0-8-10.sampup.com, puppet

    # overriding location of the hierra.yaml
    # hiera_config = /biz/puppet/hiera/hiera.yaml

    # site.pp path
    manifest = /biz/puppet/hieradata/$environment/site.pp

    # a multi-directory modulepath:

    modulepath = /etc/puppet/modules:/usr/share/puppet/modules:/biz/puppet/modules

    # site.pp path
    manifest = /biz/puppet/hieradata/$environment/site.pp

[agent]
    pluginsync = true

    # The file in which puppetd stores a list of the classes
    # associated with the retrieved configuratiion.  Can be loaded in
    # the separate ``puppet`` executable using the ``--loadclasses``
    # option.
    # The default value is '$confdir/classes.txt'.
    classfile = $vardir/classes.txt
 
    # Where puppetd caches the local configuration.  An
    # extension indicating the cache format is added automatically.
    # The default value is '$confdir/localconfig'.
    localconfig = $vardir/localconfig
 
    certname = ip-10-0-8-11.sampup.com
    dns_alt_names = ip-10-0-8-11.sampup.com

[jcbollinger]

On Wednesday, August 7, 2013 6:03:17 PM UTC-5, Dan M wrote:

I think I have found the fix... and it looks a little bit magical... Ruby made...

I was able to solve the problem by removing /etc/puppet/manifests folder.
It was empty at the moment of removal, but later on gets recreated by puppet.
Anyway - Agent finally works as expected.

If that was the key thing that solved the problem then very likely it had something to do with the permissions or SELinux context of that directory, but see also below.

 

Also I have commented out hiera_config variable.

There is a significant possibility that this was actually the key change.  Puppet 3 performs a hiera lookup for each parameter of each parameterized class declared for your node, and despite my continuing efforts, the world seems unable to get enough of class parameterization.  I don't have to look to be confident that the classes of the puppetlabs-apache module are parameterized, especially the top-level "apache" class.  An issue related to hiera configuration makes sense as a reason for Puppet to fail when you 'include' a class, but succeed when you don't.


John

[Dan M]

John

You are correct. I faced the same problems once the Hiera was uncommented.
While fixing them I had to address following open issues:

Pre-requisitions:
puppet.conf:

hiera_config = /biz/puppet/hiera/hiera.yaml

hiera.yaml:

---
:backends:
  - json
:json:
  :datadir: /biz/puppet/hieradata/%{::environment}
:hierarchy:
  - node/%{::fqdn}
  - common
:logger:
  - puppet

List of issues I have encountered:

1. common.json was missing from /biz/puppet/hieradata/%{::environment}/
   For illustration purposes, please find the file below:
   {
      "ntp::restrict" : true,
      "ntp::autoupdate" : true,
      "ntp::enable" : true,
      "ntp::servers" : [
          "0.centos.pool.ntp.org iburst",
          "1.centos.pool.ntp.org iburst",
          "2.centos.pool.ntp.org iburst"
         ]
   }
   
2. HOST_NAME.DOMAIN_NAME.json was missing from the /biz/puppet/hieradata/%{::environment}/node
   For illustration purposes, please find the file below:
   {
     "apache::vhost::priority": "10",
     "apache::vhost::vhost_name": "lalalala.sampup.com",
     "apache::vhost::port": "80",
     "apache::vhost::docroot": "/var/www"
   }
   
3. My json files contained invalid characters.
   Make sure to use something like http://jsonformatter.curiousconcept.com/ to check JSON validity

Dan