Puppet/Passenger :: Could not retrieve catalog from remote server:Error 403 on server

[Lunixer]

Greetings,

I have a tested, working setup of Puppet and Webrick. I can add nodes, classes, etc.
Then I switched to Puppet/Passenger and get the error  below.
Puppet, Apache and Passenger are all up.

I have installed using YUM repos and GEMs. So, I have the most updated packages they have.

Puppet version: 2.7.19
Ruby version: 1.8.7 (2011-06-30 patchlevel 352 i386)
Apache: 2.2.15

The error is below.
I have found little references on the web. Has anyone come across such problem recently?

[root@puppetm01 ~]# puppet agent --test
err: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: puppetm01.example.com(xxx.xxx.xxx.xxx) access to /catalog/puppetm01.example.com [find] at line 53
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run
err: Could not send report: Error 403 on SERVER: Forbidden request: puppetm01.example.com(xxx.xxx.xxx.xxx) access to /report/puppetm01.example.com [save] at line 53

Below is the path to the catalog file to which I believe the error points.

[root@puppetm01 ]# find /var/lib/puppet | grep catalog
./client_yaml/catalog
./client_yaml/catalog/puppetm01.example.com.yaml

Thanks in advance for any pointers.
----

[Jo Rhett]

Check the owner of config.ru. The owner of this file is who passenger will run the puppetmaster daemon as. I'm guessing that it's not owned by puppet.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/xms_wXhyV2EJ.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

-- 

Jo Rhett

Net Consonance : net philanthropy to improve open source and internet projects.

[Lunixer]

Thanks for the reply.

I have checked permissions per the master puppet.conf excerpt below
.
My understanding is that Passenger does not really install anything or copies files around.
You only create a directory and copy the config.ru into it and change permissions to puppet.
The only thing that passenger does is to install a Apache module, then you configure your vhost with that info.

I don't know whether I could blame the problem on any of the other packages (I.e. ruby), because things work perfectly fine with WEBrick.

Below I added more information. Please let me know If anyone spots something out of place.

[root@puppetm01 puppet]# cat puppet.conf
[main]
    user = puppet
    group = puppet


[root@puppetm01 ]# ls -l /var/lib/puppetmaster/
-rw-r--r-- 1 puppet puppet  431 Sep 27 21:51 config.ru
drwxr-xr-x 2 puppet puppet 4096 Sep 27 21:31 public
drwxr-xr-x 2 puppet puppet 4096 Sep 27 21:31 tmp

[root@puppetm01 ~]# ps -ef | grep puppet
avahi     1989     1  0 09:34 ?   00:00:00 avahi-daemon: running [puppetm01.local]
root      2666     1  0 09:34 ?   00:00:01 /usr/bin/ruby /usr/sbin/puppetd
puppet    9734  9541  2 12:35 ?   00:00:00 master                                                                                                                                                       
puppet    9769     1  0 12:35 ?   00:00:00 Rack: /var/lib/puppetmaster                                                                                                                                  


[root@puppetm01 ]# grep puppet /etc/passwd
puppet:x:52:52:Puppet:/var/lib/puppet:/sbin/nologin
puppetdb:x:494:488:PuppetDB daemon:/usr/share/puppetdb:/sbin/nologin
puppet-dashboard:x:492:489:Puppet Dashboard:/usr/share/puppet-dashboard:/sbin/nologin

[root@puppetm01 ]# id -a puppet
uid=52(puppet) gid=52(puppet) groups=52(puppet)

[root@puppetm01 ~]# passenger-memory-stats

-------- Apache processes ---------
PID   PPID  VMSize   Private  Name
-----------------------------------
9534  1     26.8 MB  0.3 MB   /usr/sbin/httpd
9551  9534  26.7 MB  0.2 MB   /usr/sbin/httpd
9552  9534  26.8 MB  0.2 MB   /usr/sbin/httpd
9553  9534  27.0 MB  0.5 MB   /usr/sbin/httpd
9554  9534  27.0 MB  0.5 MB   /usr/sbin/httpd
9555  9534  26.8 MB  0.3 MB   /usr/sbin/httpd
9556  9534  26.8 MB  0.2 MB   /usr/sbin/httpd
9557  9534  26.9 MB  0.3 MB   /usr/sbin/httpd
9558  9534  26.8 MB  0.2 MB   /usr/sbin/httpd
9559  9534  26.8 MB  0.2 MB   /usr/sbin/httpd
### Processes: 10
### Total private dirty RSS: 3.00 MB


-------- Nginx processes --------

### Processes: 0
### Total private dirty RSS: 0.00 MB


---- Passenger processes ----
PID   VMSize   Private  Name
-----------------------------
9536  6.7 MB   0.2 MB   PassengerWatchdog
9539  17.8 MB  0.4 MB   PassengerHelperAgent
9541  18.7 MB  4.9 MB   Passenger spawn server
9544  13.2 MB  0.4 MB   PassengerLoggingAgent
9769  51.8 MB  26.0 MB  Rack: /var/lib/puppetmaster
9802  60.6 MB  36.6 MB  Passenger ApplicationSpawner: /usr/share/puppet-dashboard
9808  61.1 MB  37.2 MB  Rails: /usr/share/puppet-dashboard
### Processes: 7
### Total private dirty RSS: 105.69 MB


[root@puppetm01 ~]# passenger-status --verbose
----------- General information -----------
max      = 12
count    = 2
active   = 0
inactive = 2
Waiting on global queue: 0

----------- Application groups -----------
/usr/share/puppet-dashboard:
  App root: /usr/share/puppet-dashboard
  * PID: 9808    Sessions: 0    Processed: 2       Uptime: 58s
      URL     : http://127.0.0.1:50447
      Password: xxxxxxxxxxxxxx

/var/lib/puppetmaster:
  App root: /var/lib/puppetmaster
  * PID: 9769    Sessions: 0    Processed: 2       Uptime: 1m 56s
      URL     : http://127.0.0.1:55087
      Password: xxxxxxxxxxxxxx

[root@puppetm01 ~]# tail -f /var/log/httpd/access_log
xxx.xxx.xxx.xxx - - [28/Sep/2012:12:39:20 -0700] "POST /production/catalog/puppetm01.example.com HTTP/1.1" 403 138 "-" "-"
xxx.xxx.xxx.xxx - - [28/Sep/2012:12:39:20 -0700] "PUT /production/report/puppetm01.example.com HTTP/1.1" 500 635 "-" "-"
xxx.xxx.xxx.xxx - - [28/Sep/2012:12:39:30 -0700] "POST /production/catalog/puppetm01.example.com HTTP/1.1" 403 138 "-" "-"
xxx.xxx.xxx.xxx - - [28/Sep/2012:12:39:33 -0700] "PUT /production/report/puppetm01.example.com HTTP/1.1" 403 137 "-" "-"


[root@puppetm01 ~]# find /var/lib/puppet | grep catalog | xargs ls -l
-rw-r-----. 1 root root 13150 Sep 27 21:00 /var/lib/puppet/client_yaml/catalog/puppetm01.example.com.yaml

/var/lib/puppet/client_yaml/catalog:
total 16
-rw-r-----. 1 root root 13150 Sep 27 21:00 puppetm01.example.com.yaml



Thanks,
LL
-----

[Lunixer]

Does anyone have a hint to address this problem?

Or,

Is this destined to stump many a puppet enthusiast?
If this is a bug, where does one notify puppet labs of it?

LL
----

[Jo Rhett]

This is a trivial problem to solve, but only you can do it. tcpdump is your friend. 

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/gmqnS25CCdYJ.

To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Lunixer]

I don't think this is trivial. If it were, I would have already found the problem by looking at the obvious things.
What I have seen from several posts is that there's other error similar to the one I've seen. I even came across a bug report filed a while back with the same error I see, but I lost the link and cannot find it.

The problem is not even from a client to the master. The testing I've done is all in the master.

I'll try strace instead of tcpdump, being that this is not a TCP communication problem over the wire but rather a file or directory access problem.


LL
----

[Jo Rhett]

On Oct 1, 2012, at 5:00 PM, Lunixer wrote:

I'll try strace instead of tcpdump, being that this is not a TCP communication problem over the wire but rather a file or directory access problem.

Um, no. Puppet client talks to the server over the network, even on the same host. You really should listen to advice we provide.